ANTLR4 python runtime error

[JustARandomCoder]

Hello, I am new to ANTLR4 and was trying to use the Go antlr4 grammar to make a parser in Python. I downloaded this folder from Github and ran:

antlr4 -o testparser -Dlanguage=Python3 GoParser.g4 GoLexer.g4

It created - GoLexer.interp, GoLexer.tokens, GoParser.interp, GoParser.py, GoLexer.py, GoParserListener.py and GoParser.tokens

I had downloaded the python runtime using

pip install antlr4-python3-runtime

I am running on python version 3.8.10.

My code is this main.py (in the same folder, testparser):

from antlr4 import *
from GoLexer import GoLexer

code = "package main \n"
codeStream = InputStream(code)
lexer = GoLexer(codeStream)

tokens = lexer.getAllTokens()

for t in tokens:
    print(t.text, t.type)

This works, but how would I parse it?

I tried to import GoParser but it gave me an error:
ModuleNotFoundError: No module named 'GoParserBase'
So after some time, I checked the Python3 folder from the folder I downloaded from Github - this I copied the file called GoParserBase into the directory and the error seems to be gone. Now to use the parser i wrote this

from antlr4 import *
from GoLexer import GoLexer
from GoParser import GoParser
code = open('test.go', 'r').read()
codeStream = InputStream(code)
lexer = GoLexer(codeStream)
tokens = CommonTokenStream(lexer)

parser = GoParser(tokens)
mytree = parser.sourceFile()
And test.go has:

package main
func main(){
print(2)
}

I get error:

Traceback (most recent call last):

File "main.py", line 10, in <module>

mytree = parser.sourceFile()

File "/home/myuser/Downloads/somefolder/testparser/GoParser.py", line 766, in sourceFile

self.functionDecl()

File "/home/myuser/Downloads/somefolder/testparser/GoParser.py", line 1962, in functionDecl

self.block()

File "/home/myuser/Downloads/somefolder/testparser/GoParser.py", line 2299, in block

self.statementList()

File "/home/myuser/Downloads/somefolder/testparser/GoParser.py", line 2374, in statementList

la_ = self._interp.adaptivePredict(self._input,30,self._ctx)

File "/usr/local/lib/python3.8/dist-packages/antlr4/atn/ParserATNSimulator.py", line 346, in adaptivePredict

alt = self.execATN(dfa, s0, input, index, outerContext)

File "/usr/local/lib/python3.8/dist-packages/antlr4/atn/ParserATNSimulator.py", line 428, in execATN

conflictingAlts = self.evalSemanticContext(D.predicates, outerContext, True)

File "/usr/local/lib/python3.8/dist-packages/antlr4/atn/ParserATNSimulator.py", line 1077, in evalSemanticContext

predicateEvaluationResult = pair.pred.eval(self.parser, outerContext)

File "/usr/local/lib/python3.8/dist-packages/antlr4/atn/SemanticContext.py", line 110, in eval

return parser.sempred(localctx, self.ruleIndex, self.predIndex)

File "/home/myuser/Downloads/somefolder/testparser/GoParser.py", line 8090, in sempred

return pred(localctx, predIndex)

File "/home/myuser/Downloads/somefolder/testparser/GoParser.py", line 8094, in statementList_sempred

return this.closingBracket()

NameError: name 'this' is not defined

[JustARandomCoder]

Oh and I forgot to mention it works when the code is like

package main
import "fmt"
func main(){

}

but the moment i add a statement in main, it errors

[Ge Hu]

JustARandomCoder,

Check out this GitHub site: https://github.com/jszheng/py3antlr4book.  It appears to be most (all?) of the examples from "The Definitive ANTLR4 Reference" translated into Python3.  As far as how to parse a grammar I would suggest studying the book and look at how they use Listeners and Visitors to parse grammars.

Note: I do not program in Python, so I can not attest to the correctness of these example (or if they even work).

George

[Ken Domino]

(1) Always check the desc.xml to see if the grammar has been ported to the target. In this case the golang grammar has been ported to Python3.

(2) Use trgen to generate a parser for the target. See 
https://github.com/antlr/grammars-v4/wiki#using-trgen-all-targets. If you do not generate an application using trgen, you will need to do everything on your own. This means copying the target-specific support code into a directory with the .g4's, running transformGrammar.py, running the Antlr tool, writing a driver program, compiling, etc.

(3) The compilation error with "this" is caused by not applying transformGrammar.py to the .g4's prior to doing a build. Antlr grammars that contain "actions" have target-specific code suitable for Java. For other tagets, the actions are changed for the target you want using transformGrammar.py.

On Sunday, April 28, 2024 at 7:15:44 AM UTC-4 JustARandomCoder wrote:

[JustARandomCoder]

Hello Ken,

I installed trgen and ran

trgen -t Python3

Went into the folder it generated and ran

make

I get error

bash build.sh
Altering GoParser.g4
Writing ...
build.sh: line 12: antlr4: command not found
make: *** [makefile:3: build] Error 127

When i run those commands in my terminal from build.sh:

antlr4 -v $version -encoding utf-8 -Dlanguage=Python3   GoLexer.g4
antlr4 -v $version -encoding utf-8 -Dlanguage=Python3   GoParser.g4

I get:

error(2):  unknown command-line option -v

error(2):  unknown command-line option -v

[Ken Domino]

The build script has prereqs (will add checks with warnings later). Run "pip install antlr4-tools". See https://github.com/antlr/antlr4-tools.

[JustARandomCoder]

I have antlr4 installed, I fix that error by removing the -v and $version part from those two and running them in the folder but not through the script, it works i can use antlr4 now. One small question, is the Go antlr grammar extracted from the official compiler? If yes then why aren't the comments kept? The comments in Golang can be used for conditional compilation and the ast even has nodes storing them.

[Ken Domino]

I wrote the templated build script with "-v $version" because some target runtimes would be published on different dates, sometimes a month or two after the Antlr tool was released. This would cause the builds to break in grammars-v4. But, potentially a grammar could only work for certain versions of Antlr because of some feature used in the grammar or a critical bug fix in the runtime. Some desc.xml indicate a minimum required version of Antlr.

The Antlr grammar for Go is not derived from the Go compiler. It is hand-scraped from "The Go Programming Language Specification" https://go.dev/ref/spec. At some point, the scraping of the EBNF plus refactoring to bring it into Antlr syntax should be automated, i.e., with ChatGPT for scraping and the Trash toolkit for refactoring. Of course, it presumes the "Spec" is correct.

[JustARandomCoder]

Oh ok, I asked around and found out that conditional compilation is apparently a feature of the Go compiler (which is the official compiler) but isn't a part of the actual language.

Thank you for introducing me to trgen!