fox: 'quick' 'brown' 'fox' '\r'? '\n' ;
Then I used the generated parser with UnbufferedCharStream and UnbufferedTokenStream:
CharStream input = new UnbufferedCharStream(is);
MyLexer lex = new MyLexer(input);
lex.setTokenFactory(new CommonTokenFactory(true));
TokenStream tokens = new UnbufferedTokenStream(lex);
MyParser parser = new MyParser(tokens);
MyParser.FoxContext fox = parser.fox();
when the stream gets 'quick' - nothing happens
when 'b' comes in - entering rule 'fox'
then 'roun' - nothing (2 tokens are in the stream - none of them is known to leser yet!)
only after 'f' the listener visits the first token: 'quick'
then - nothing on 'ox'
on new line (unix): visit token 'brown'
Now the stream has all data (4 tokens), but only 2 tokens are recognized.
I found that in order to push those tokens through the system the stream can emit 2 tokens, that is any tokens known to the grammar.
It could be 2 extra new lines, or 'fox' and 'brown'.
Only then the tokens 'fox' and '\n' get visited, the parser exits rule 'fox' and parsing gets finished.
Is that a bug or a feature?
Is there a way to eliminate that lag?
Thanks!