Python3 target generated Lexer has loads of error (Also it has some Java code inside it, why ?)

[code.r...@gmail.com]

I have attached the generated Python3Lexer.py file that I get when I do

antlr4 -Dlanguage=Python3 Python3.g4

I got Python3.g4 from antlr grammar repo.
Thanks.

[Eric Vergnaud]

Hi,

this grammar is designed for the Java runtime, and needs to be adapted for other target runtimes.

Eric

[code.r...@gmail.com]

I have tried to translate that chunk of java code to Python in Python3 grammar.
But I have not got it working properly. Can I get my code reviwed? I'll make a PR thereafter and It'd help others also.
I'm attaching the translated PythonLexer.py code.
Thanks.

[Eric Vergnaud]

Hi,

you need to do the translation in the grammar itself, not in the generated lexer.

I notice you renamed emit to emitt in order to avoid collision.

You actually need to override emit, so you must keep the same name, precisely so that you have control

on which tokens are emitted, and manage the indentation accordingly.

Good luck,

Eric

[code.r...@gmail.com]

Hi,

Thanks a lot for reviewing my code. I worked at what you said. and I have got it almost working. the only problem I'm facing is when I generate parse tree of this input

print('hello world')
#hey
dict = {'a':1,'b':2}

I get this

line 3:0 extraneous input 'dict' expecting {'if', 'in', 'or', 'and', 'not', 'is', <INVALID>, '.', '*', '(', ',', ';', '**', '=', '[', '|', '^', '&', '<<', '>>', '+', '-', '/', '%', '//', '<', '>', '==', '>=', '<=', '<>', '!=', '@', '+=', '-=', '*=', '@=', '/=', '%=', '&=', '|=', '^=', '<<=', '>>=', '**=', '//='}
line 4:0 no viable alternative at input '<EOF>'
(file_input (stmt (simple_stmt (small_stmt (expr_stmt (testlist_star_expr (test (or_test (and_test (not_test (comparison (star_expr (expr (xor_expr (and_expr (shift_expr (arith_expr (term (factor (power (atom print) (trailer ( (arglist (argument (test (or_test (and_test (not_test (comparison (star_expr (expr (xor_expr (and_expr (shift_expr (arith_expr (term (factor (power (atom (strr 'hello world')))))))))))))))))) ))))))))))))))))))))) (stmt (simple_stmt (small_stmt (expr_stmt (testlist_star_expr (test (or_test (and_test (not_test (comparison (star_expr (expr (xor_expr (and_expr (shift_expr (arith_expr (term (factor (power (atom dict)))))))))))))))) = (testlist_star_expr (test (or_test (and_test (not_test (comparison (star_expr (expr (xor_expr (and_expr (shift_expr (arith_expr (term (factor (power (atom { (dictorsetmaker (test (or_test (and_test (not_test (comparison (star_expr (expr (xor_expr (and_expr (shift_expr (arith_expr (term (factor (power (atom (strr 'a')))))))))))))))) : (test (or_test (and_test (not_test (comparison (star_expr (expr (xor_expr (and_expr (shift_expr (arith_expr (term (factor (power (atom (number (integer 1))))))))))))))))) , (test (or_test (and_test (not_test (comparison (star_expr (expr (xor_expr (and_expr (shift_expr (arith_expr (term (factor (power (atom (strr 'b')))))))))))))))) : (test (or_test (and_test (not_test (comparison (star_expr (expr (xor_expr (and_expr (shift_expr (arith_expr (term (factor (power (atom (number (integer 2)))))))))))))))))) })))))))))))))))))))) <EOF>)

However with Java runtime I get

(file_input (stmt (simple_stmt (small_stmt (expr_stmt (testlist_star_expr (test (or_test (and_test (not_test (comparison (star_expr (expr (xor_expr (and_expr (shift_expr (arith_expr (term (factor (power (atom print) (trailer ( (arglist (argument (test (or_test (and_test (not_test (comparison (star_expr (expr (xor_expr (and_expr (shift_expr (arith_expr (term (factor (power (atom (str 'hello world')))))))))))))))))) ))))))))))))))))))) \n)) (stmt (simple_stmt (small_stmt (expr_stmt (testlist_star_expr (test (or_test (and_test (not_test (comparison (star_expr (expr (xor_expr (and_expr (shift_expr (arith_expr (term (factor (power (atom dict)))))))))))))))) = (testlist_star_expr (test (or_test (and_test (not_test (comparison (star_expr (expr (xor_expr (and_expr (shift_expr (arith_expr (term (factor (power (atom { (dictorsetmaker (test (or_test (and_test (not_test (comparison (star_expr (expr (xor_expr (and_expr (shift_expr (arith_expr (term (factor (power (atom (str 'a')))))))))))))))) : (test (or_test (and_test (not_test (comparison (star_expr (expr (xor_expr (and_expr (shift_expr (arith_expr (term (factor (power (atom (number (integer 1))))))))))))))))) , (test (or_test (and_test (not_test (comparison (star_expr (expr (xor_expr (and_expr (shift_expr (arith_expr (term (factor (power (atom (str 'b')))))))))))))))) : (test (or_test (and_test (not_test (comparison (star_expr (expr (xor_expr (and_expr (shift_expr (arith_expr (term (factor (power (atom (number (integer 2)))))))))))))))))) })))))))))))))))))) \n)) <EOF>)

I see I'm not getting '\n' in the parse tree (see the last line of both parse trees). Though I have kept grammar the same except changing the java code to python. I'm attaching my Python3.g4 grammar.

Thanks.

[Eric Vergnaud]

in NEWLINE, why are you calling super().emitToken instead of self.emitToken ?

[code.r...@gmail.com]

Oh sorry, That was a silly mistake. So, I got it working finally. Tested it on few inputs. Should I make a PR ?
(@parrt had asked me to make a PR).
and Thanks a lot. :)

[code.r...@gmail.com]

PR - https://github.com/antlr/grammars-v4/pull/572