Modify Grammar/Lexer/Parser at Runtime?

[Simon Gunacker]

Hey there,

I am new to antlr and I was just wondering if it is possible to somehow modify the g4 during runtime. Something like:

1. there exists and old.g4
2. all parsers / lexers and visitors are already generated
3. the grammar gets modified
4. then ... a call like lexer.refresh() would be convenient ;-)

is that possible????

thanks, Simon

[Eric Vergnaud]

Although it almost sounds like a joke, it is actually feasible.

g4 files are parsed by antlr using an antlr grammar.

From there it is possible to alter the parse tree and regenerate lexers/parsers/listeners.

From there it is possible using facilities such as Javassist or BCEL to alter the currently loaded classes.

Obviously you would have to implement this "refresh" function yourself.

[Mike Lischke]

It's not trivial and it looks like LexerInterpreter and ParserInterpreter could help here (I haven't had the opportunity to test this myself). There are a number of tests in the Java tools test that show how to use it, e.g.:

**
* This is a regression test for antlr/antlr4#461.
* https://github.com/antlr/antlr4/issues/461
*/
@Test public void testLeftRecursiveStartRule() throws Exception {
LexerGrammar lg = new LexerGrammar(
"lexer grammar L;\n" +
"A : 'a' ;\n" +
"B : 'b' ;\n" +
"C : 'c' ;\n" +
"PLUS : '+' ;\n" +
"MULT : '*' ;\n");
Grammar g = new Grammar(
"parser grammar T;\n" +
"s : e ;\n" +
"e : e MULT e\n" +
" | e PLUS e\n" +
" | A\n" +
" ;\n",
lg);

testInterp(lg, g, "e", "a", "(e a)");
testInterp(lg, g, "e", "a+a", "(e (e a) + (e a))");
testInterp(lg, g, "e", "a*a", "(e (e a) * (e a))");
testInterp(lg, g, "e", "a+a+a", "(e (e (e a) + (e a)) + (e a))");
testInterp(lg, g, "e", "a*a+a", "(e (e (e a) * (e a)) + (e a))");
testInterp(lg, g, "e", "a+a*a", "(e (e a) + (e (e a) * (e a)))");
}

ParseTree testInterp(LexerGrammar lg, Grammar g,
String startRule, String input,
String expectedParseTree)
{
LexerInterpreter lexEngine = lg.createLexerInterpreter(new ANTLRInputStream(input));
CommonTokenStream tokens = new CommonTokenStream(lexEngine);
ParserInterpreter parser = g.createParserInterpreter(tokens);
ParseTree t = parser.parse(g.rules.get(startRule).index);
System.out.println("parse tree: "+t.toStringTree(parser));
assertEquals(expectedParseTree, t.toStringTree(parser));
return t;
}

That looks simple, but the tricky part is that LexerGrammar and Grammar classes are not part of the runtime. They are needed to generated the ATN required by the interpreters.

Mike
--
www.soft-gems.net

[Loring Craymer]

Simon--

I think that Eric and Mike have just demonstrated that the answer is "Yes, but that is because anyone in this forum is quite willing to argue about how many angels can dance on the head of a pin".  I would join in, but you did say that you were a newbie.

You can implement dynamic languages with ANTLR, but only with special effort.   ANTLR is not intended to support this sort of on-the-fly grammar modification, even if it can be done.  Read more on language theory, including Ter's books:  that should give you the appropriate mindset.

--Loring