Message Boards Message Boards

Experiment: Can OpenAI's GPT-3 Write Wolfram Language Code?

Posted 4 years ago

Twitter has been humming this week with people experimenting with OpenAI's GPT-3 language model. A few days ago I received an invitation to the private beta, and I think it's fair to say that I'm very impressed and surprised with how well it generalizes, even to newly provided context.

In this post I'll demonstrate the model's ability to quickly pick up on how to write Wolfram Language code.

Predicting What Comes Next

To use the language model, you provide it with a string of text (the gray text below), and its job is to predict what text should come next (the black text below). For example:

enter image description here

Specializing the Model

By providing the model with some preamble text containing a few examples that map an input to an output, it will -- surprisingly -- generalize that mapping to new inputs. For example:

enter image description here

And while that's just a toy example, it's ability to generalize really is quite impressive:

enter image description here

Converting Natural Language to Code

It didn't take long for people to try using the model to convert natural language to code. And amazingly, it works!

Let's do some experiments to see how quickly it can pick up on how to write Wolfram Language code.

Inferring Function Names

Let's see if a single example is enough to teach the model how to choose a reasonable function name:

enter image description here

Sure enough. It inferred that "Max" would be a good function name for "maximum", and it followed my lead using square brackets, passing a variable named list.

Let's see if it catch on to using "Q" as a suffix for boolean valued functions:

enter image description here

Argument Values

We're going to need to do more than choose function names -- we also need to deal with argument values properly. Let's see if the model can do that:

enter image description here

Variance in Phrasings

And of course, the model will need to be able to adapt to different phrasings:

enter image description here

Composing Functions

Let's see if it can learn how to compose functions:

enter image description here

Example 2:

enter image description here

Example 3:

enter image description here

Some Reflections and Predictions

We're just getting started here. It will be interesting to see how sophisticated and reliable GPT-3 can be at mapping natural language to code given enough examples. But these initial results are extremely promising.

One of my pet peeves as a software developer is getting stuck for 5 or 10 minutes trying to look up in the documentation or on the web how to do some simple thing. (it happened multiple times tonight) I'll usually figure it out, but it breaks my flow. For many years I've dreamed of a system that could reliably take a short natural language description and provide a high quality code snippet to demonstrate how to do something.

With GPT-3, it appears we're finally on the cusp.

https://openai.com/blog/openai-api/

Code

If you've been lucky enough to get access to the OpenAI beta, here's some of my code incase you find it useful:

$openAIBaseURL = "https://api.openai.com/v1/engines/";
$openAIPublishableKey = "TODO";
$openAISecretKey = "TODO";

(*!
    \function OpenAIURL

    \calltable
        OpenAIURL[model, endpointName] '' Produces a URL to an OpenAI API endpoint.

    Examples:

    OpenAIURL["davinci", "completions"] === "https://api.openai.com/v1/engines/davinci/completions"

    \maintainer danielb
*)
Clear[OpenAIURL];
OpenAIURL[model_, endpointName_String] :=
    Module[{},
        URLBuild[{$openAIBaseURL, model, endpointName}]
    ]

(*!
    \function OpenAICompletions

    \calltable
        OpenAICompletions[prompt] '' Given a prompt, calls the OpenAI API to produce a completion.

    Example:

    OpenAICompletions["In 10 years, some people are predicting Tesla's stock price could reach"]

    ===

    " $4,000.

    That's a lot of money."

    \maintainer danielb
*)
Clear[OpenAICompletions];
Options[OpenAICompletions] =
{
    "Model" -> "davinci",                       (*< The model name to use. *)
    "TokensToGenerate" -> 20,                   (*< The maximum number of tokens to generate. *)
    "Temperature" -> 0,                         (*< Controls randomness / variance. 0 is no variance, 1 is maximum variance. *)
    "FormatOutput" -> True,                     (*< Should the output be graphically formatted? *)
    "AutoPrune" -> True                         (*< Automatically prune what appears to be beyond the first answer? *)
};
OpenAICompletions[prompt_String, OptionsPattern[]] :=
    Module[{completion},

        completion =
            URLExecute[
                HTTPRequest[
                    OpenAIURL[OptionValue["Model"], "completions"],
                    <|
                        Method -> "POST",
                        "Body" -> ExportString[
                            <|
                                "prompt" -> prompt,
                                "max_tokens" -> OptionValue["TokensToGenerate"],
                                "temperature" -> OptionValue["Temperature"]
                            |>,
                            "JSON"
                        ],
                        "Headers" -> {
                            "Authorization" -> "Bearer " <> $openAISecretKey,
                            "Content-Type" -> "application/json"
                        }
                    |>
                ],
                "RawJSON"
            ];

        If [FailureQ[completion], Return[completion, Module]];

        completion = completion[["choices", 1, "text"]];

        $RawOpenAIOutput = completion;

        If [TrueQ[OptionValue["AutoPrune"]],
            completion = AutoPruneOpenAIOutput[completion]
        ];

        $OpenAIOutput = completion;

        If [TrueQ[OptionValue["FormatOutput"]],
            completion = FormatCompletion[prompt, completion]
            ,
            completion
        ]
    ]

(*!
    \function FormatCompletion

    \calltable
        FormatCompletion[prompt, completion] '' Formats a completion from a language model.

    Examples:

    FormatCompletion["In the beginning, God", " created the heavens and the earth."]

    ===

    Framed[
        Style[
            Row[
                {
                    Style["In the beginning, God", GrayLevel[0.5]],
                    Style[" created the heavens and the earth.", Bold]
                }
            ],
            FontSize -> 16
        ],
        FrameStyle -> GrayLevel[0.5],
        FrameMargins -> {{20, 20}, {16, 16}}
    ]

    \maintainer danielb
*)
Clear[FormatCompletion];
FormatCompletion[prompt_String, completion_String] :=
    Module[{},
        Framed[
            Style[
                Row[{Style[prompt, Gray], Style[completion, Bold]}],
                FontSize -> 16
            ],
            FrameStyle -> Gray,
            FrameMargins -> {{20, 20}, {16, 16}}
        ]
    ]

(*!
    \function OpenAICompletionUI

    \calltable
        OpenAICompletionUI[] '' A UI for producing completions using the OpenAI API.

    \maintainer danielb
*)
Clear[OpenAICompletionUI];
OpenAICompletionUI[] :=
    Module[{inputFieldVar, input, output = ""},

        Dynamic[
            Grid[
                {
                    {
                        Framed[
                            #,
                            FrameMargins -> {{0, 0}, {2, 0}},
                            FrameStyle -> None
                        ] & @
                        EventHandler[
                            Style[
                                InputField[
                                    Dynamic[inputFieldVar],
                                    String,
                                    FrameMargins -> Medium,
                                    ReturnEntersInput -> False,
                                    ImageSize -> {700, Automatic}
                                ],
                                FontSize -> 14
                            ],
                            {"MenuCommand", "HandleShiftReturn"} :>
                                (
                                    input = StringReplace[inputFieldVar, {"\[IndentingNewLine]" -> "\n"}];
                                    output = OpenAICompletions[
                                        StringTrim[input],
                                        "TokensToGenerate" -> 30
                                    ]
                                )
                        ]
                    },
                    {
                        output
                    }
                }
            ],
            TrackedSymbols :> {inputFieldVar, output}
        ]
    ]

(*!
    \function AutoPruneOpenAIOutput

    \calltable
        AutoPruneOpenAIOutput[output] '' Given some output from OpenAI's API, tries to automatically remove what is beyond the next output.

    Examples:

    AutoPruneOpenAIOutput["Just testing\n\n\"\"\"\n\nMore output"] === "Just testing"

    \maintainer danielb
*)
Clear[AutoPruneOpenAIOutput];
AutoPruneOpenAIOutput[outputIn_] :=
    Module[{matches, output = outputIn},

        matches = StringPosition[
            output,
            Repeated["\n", {1, Infinity}] ~~ "\"\"\"" ~~ Longest[___]
        ];

        If [Length[matches] > 0,
            output = StringTake[output, matches[[-1, 1]]]
        ];

        (* StringTrim right side *)
        StringReplace[
            output,
            WhitespaceCharacter.. ~~ EndOfString :> ""
        ]
    ]
POSTED BY: Daniel Bigham
9 Replies

Thanks for this. Been hacking away in Mathematica for all these reasons... it's promising for a lot of reasons and then it's also adding to the computational irreducibility.

I think its biggest value is going to be as "glue" between formal systems. At the end of the day... the entity system for Wolfram Data and the SPARQL stuff with wikidata kinda shows how much we require hybrid approaches.

Beyond that, there are some things that just going to be inefficient to express even in few shot gpt3 formulations.

But, just like Wolfram|Alpha has dramatically improved productivity this should help on some things.

Hi Syd, thanks for your interest.

Yes, your interactions with Jon are on a closely related topic -- basically, what are all of the ways a powerful language model can be used to provide compelling value to programmers and others.

This is an exciting area that I think we'll see lots of progress on in the next 10 years.

POSTED BY: Daniel Bigham

Hi Sam, you ask a good question.

That's the surprising part -- the only WL "training data" given to the model is the gray text shown in each example above. This is sometimes referred to as "few shot learning" -- when a model is able to adapt to a new task by only seeing a few examples.

That said, OpenAI trained GPT-3 on a huge corpus of text taken from the web, which undoubtedly would have contained examples of WL code, so it's possible that's it's drawing on some of that prior knowledge.

POSTED BY: Daniel Bigham
Posted 4 years ago

Wow this input from Daniel Bigham is very exciting for me ...

I had an exchange with Jon McLoone of Wolfram England last week which I trust he will not mind me sharing:

Jon McLoone This is a reminder that "Making Predictions from Financial Data" will begin in 1 Hour on: Mon, Jul 13, 2020 10:30 AM - 12:00 PM PDT

My Question to Jon ... Q: do you have a way of scraping the net for all accessible Mathematica notebooks so that I can research predictive algorithms or models for the next input while generating my own code? Could that be done on the Manipulate Examples on your website?

A: Nothing public beyond using WebSearch. We do have a project internally to build a language models for Wolfram Language and others. This is already partly exposed in Classify["ProgrammingLanguage", "WebImageSearch[\"cat\"]"] but the plan is to use it much more richly for WL by being able to detect coding errors, suggest improvements or auto-completions.

  • Q:
Wow great this is very close to what I am interested in ...Syd Geraghty

This is already partly exposed in Classify["ProgrammingLanguage", "WebImageSearch[\"cat\"]"] but the plan is to use it much more richly for WL by being able to detect coding errors, suggest improvements or auto-completions.
11:32 AM 


My Question to Jon ...Q:
Presumably that initiative extends implementation of the current predictive interface in 12.1
11:35 AM 


My Question to Jon ... Q:
So presumably this would make it easy for Patrick Schiebe to add significant functionality to the IntelliJ Idea WL plug in ... 


A:
Yes, I would hope so. More broadly there are various projects for making better developer tools, eg code scanning, unit test coverage etc. And these should all be exposed.

11:47 AM

My Question to Jon ... Q: Wonderful ...

A:
More broadly there are various projects for making better developer tools, eg code scanning, unit test coverage etc. And these should all be exposed.


11:48 AM

Access to this type of technology would make my new project called the Institute of Computational Knowledge ( https://instituteofcomputationalknowledge.com ) so much easier to manage.

While I have been a Mathematica user for 20 years at 76 I need every bit of help generating materials for the project. I have chosen to base everything I can possibly do regarding the project on the Wolfram Technology stack as a proof of concept that the Wolfram Language can support:

Stephen Wolfram's goal for WolframAlpha and the Wolfram Language

Wolfram: My goal with the Wolfram Language is to have a language in which computations can conveniently be expressed for both humans and machines—and in which we’ve integrated as much knowledge about computation and about the world as possible.  In a way, the Wolfram Language is aimed at finally achieving some of the goals Leibniz had 300 years ago.  We now know—as a result of Gödel’s theorem, computational irreducibility, etc.—that there are limits to the scientific questions that can be resolved.  And as far as moral questions are concerned: well, the Wolfram Language is going in the direction of at least being able to express things like moral principles, but it can’t invent those; they have to come from humans and human society.

Thanks so much Daniel ...

POSTED BY: Syd Geraghty
Posted 4 years ago

In each case, the final output (which is in bold) is prepended with a number of examples. GPT generalizes from those examples, which indicates which language it is working in. That "prompt," plus its training data, is enough to let it figure out the answers.

POSTED BY: David Manheim

Thank you @Daniel for sharing, very interesting. I might have missed this somehow in your post: did you have to train this neural net on some Wolfram Language data? How does it now to output things in Wolfram Language and not in some other language? Could you please explain a bit deeper this technical part.

POSTED BY: Sam Carrettie
Posted 4 years ago

Wolfram Alpha could definitely integrate GPT-3. AI Dungeon is a commercial product (game) that has done so.

https://twitter.com/nickwalton00/status/1283984454952693760?s=19 :

Yes there API is available for commercial use (their goal is for it to be a commercial product) and there aren't explicit rate limits per say, but you'd need to have a conversation before hand if you wanted to push through large amounts of traffic.

POSTED BY: Junyan Xu

Yes, definitely some good possibilities for a model like this.

POSTED BY: Daniel Bigham

I suppose this is how the free-form input feature will function eventually. Unless it already uses machine learning ?

POSTED BY: Lucien Grondin
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract