GPT-4 writes a GF grammar

[Krasimir Angelov]

Hi Everyone,

I had a meeting yesterday where we talked about large language models and GF. We asked both GPT-3 and GPT-4 to generate GF code. 

GPT-3 managed to generate a decent abstract syntax for restaurant reviews but failed on the concrete syntax. Unfortunately, this was only on the screen and I didn't save the code.

After that we tried GPT-4 and this is what we got:

abstract RestaurantReview = {

  flags startcat = Review ;

  cat

    Review ; Restaurant ; Food ; Service ; Atmosphere ;

    Rating ;

  fun

    CreateReview : Restaurant -> Food -> Service -> Atmosphere -> Rating -> Review ;

    Name : String -> Restaurant ;

    DescribeFood : String -> Food ;

    DescribeService : String -> Service ;

    DescribeAtmosphere : String -> Atmosphere ;

    Rate : Int -> Rating ;

}

concrete RestaurantReviewEng of RestaurantReview = {

  lincat

    Review = Str ;

    Restaurant = Str ;

    Food = Str ;

    Service = Str ;

    Atmosphere = Str ;

    Rating = Str ;

  lin

    CreateReview restaurant food service atmosphere rating =

      "I visited the restaurant" +++ restaurant +++ "and I must say" +++ food +++ "." +++ service +++ "." +++ atmosphere +++ "." +++ "I give it" +++ rating +++ "out of 10.";

    Name name = name ;

    DescribeFood description = "the food was" +++ description ;

    DescribeService description = "the service was" +++ description ;

    DescribeAtmosphere description = "the atmosphere was" +++ description ;

    Rate rating = show rating ;

}

It is obviously not perfect but quite good. After that we asked GPT-4 to generate an abstract syntax tree for the sentence "George bush visited Oxford university last year". The output follows. The only problem that I can spot is that you can't turn a String into PN in the way suggested.

To parse the given sentence "George bush visited Oxford university last year" into a GF abstract syntax tree (AST), we need to first create an appropriate GF grammar. Below is a simple example of a GF abstract syntax:

abstract VisitGrammar = {
  flags startcat = Sentence;
  cat
    Sentence; Person; Institution; Time;
  fun
    Visited : Person -> Institution -> Time -> Sentence;
    Name : String -> Person;
    InstitutionName : String -> Institution;
    LastYear : Time;
}

This grammar has categories for a Sentence, Person, Institution, and Time, with functions to create a visit sentence, name a person, name an institution, and represent the last year.
Then, we need a concrete syntax for the language. In this case, we'll create one for English:
concrete VisitGrammarEng of VisitGrammar = open SyntaxEng, ParadigmsEng, LexiconEng in {
  lincat
    Sentence = Utt;
    Person = NP;
    Institution = NP;
    Time = Adv;

  lin
    Visited person institution time =
      mkUtt (mkNP person (mkVP (mkV "visit") institution) time);
    Name name = mkNP (mkPN name);
    InstitutionName name = mkNP (mkCN (mkN name));
    LastYear = mkAdv "last year";
}
Now, to parse the given sentence, you can use the GF command-line tool with the following steps:
Save both the abstract and concrete syntax code in two separate files: VisitGrammar.gf and VisitGrammarEng.gf.
Load both the grammars in the GF shell using the command import VisitGrammar.gf VisitGrammarEng.gf.
Parse the sentence using the command parse "George bush visited Oxford university last year".
The output AST would look like this:
Visited (Name "George bush") (InstitutionName "Oxford university") LastYear

[Aarne Ranta]

Thanks for these results, Krasimir! They are quite impressive, and it would be great to understand how they have been achieved.

I spotted another error:

mkUtt (mkNP person (mkVP (mkV "visit") institution) time);

What is the instance of mkNP used here? I guess it should be mkCl, and another mkVP would be needed to attach the time to the VP.

  Aarne

--

---
You received this message because you are subscribed to the Google Groups "Grammatical Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gf-dev+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gf-dev/CAJv-UW7GPVvNAF05M7yxtODYAaRwnetb4ziOQW8VFx4JC6nX_A%40mail.gmail.com.

[Krasimir Angelov]

You are right. I didn't actually try to compile the code, but I think with small changes it can be made to work. If this was an exam question, the student would have passed although not with 5.

To view this discussion on the web visit https://groups.google.com/d/msgid/gf-dev/CAH881EE6EMrLsGHgZ9nCk-c5se%3DyK-eS0rRrxG9ROpHE6o_3BQ%40mail.gmail.com.

[Kaarel Kaljurand]

One could also try to experiment with including the GF spec (or FAQ,
tutorial, ...) in the prompt, and any possible compiler error messages
in follow-up prompts. This should generally improve the results.

Best,
Kaarel

> To view this discussion on the web visit https://groups.google.com/d/msgid/gf-dev/CAJv-UW7-KYsDJ7cGU2rFsW3sZcPy89Tdb_-kTbnoE-cfFz1QtQ%40mail.gmail.com.