I might be missing something, but how can one reuse parsers/lexers to repeatedly parse some input without the need to create new instances?
I'm setting up parsing this way which works fine:
MyLexer lexer = new MyLexer();
lexer.setInputStream(new ANTLRFileStream(new File("file1")));
MyParser parser = new MyParser(new CommonTokenStream(lexer));
parser.parse();
But when I try to parse another file, things are not working as expected:
lexer.setInputStream(new ANTLRFileStream(new File("file2")));
parser.parse();
The run still succeeds, no errors appear, but no tree is build. Hm.
Ok, maybe I need to manually reset the TokenStream. BufferedTokenStream has a reset() method, but it did not yield the desired effect (the input is not changed, yielding the same tree as with the first run).
Then I tried resetting the token source as well with BufferedTokenStream#setTokenSource(). This produced the following exception:
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at org.antlr.v4.runtime.BufferedTokenStream.nextTokenOnChannel(BufferedTokenStream.java:304)
at org.antlr.v4.runtime.CommonTokenStream.adjustSeekIndex(CommonTokenStream.java:65)
at org.antlr.v4.runtime.BufferedTokenStream.setup(BufferedTokenStream.java:247)
at org.antlr.v4.runtime.BufferedTokenStream.lazyInit(BufferedTokenStream.java:241)
at org.antlr.v4.runtime.CommonTokenStream.LT(CommonTokenStream.java:87)
at org.antlr.v4.runtime.Parser.enterRule(Parser.java:430)
Looking at the code, I think the "fetchedEOF" flag should be reset as well. At least that's what I've changed and things started to work, but either there's a bug or there must be a better way to reconfigure the parser, which I didn't find. Any help?
Thanks.
For now, out of habit. I haven't done any performance testing with ANTLR 4 yet.
I have an existing infrastructure originally based on ANTLR 2 where it used to be costly to create new instances. I'm simply trying to introduce an ANTLR 4 parser into the codebase and naturally started with the same tried and true approach of keeping one instance per thread.
Do you mind sharing the reasons to better create new instances? Thanks!
This sounds very promising. ANTLR 4 is really a big step forward! Thanks!