extraneous input error does not list all possible tokens

[Martin d'Anjou]

Hello,

I am getting an extraneous input error, which is what I expect, BUT it does not list all the possible tokens, and I do not understand why. Can someone help me understand please?

Here is the code (store it in: example.groovy):

@Grapes([

  @Grab(group="org.antlr", module="antlr4", version="4.8")

])

import org.antlr.v4.runtime.ANTLRInputStream

import org.antlr.v4.runtime.Token

import org.antlr.v4.tool.Grammar

import org.antlr.v4.tool.LexerGrammar

import org.antlr.v4.runtime.CommonTokenStream

def lg = new LexerGrammar("""

lexer grammar N;

WS : [ \\t]+ -> skip ;

MACHINE: 'machine';

ACCOUNT: 'account';

ID: ~('\\r' | '\\n' | ' ' | '\\t')+;""")

def g = new Grammar("""

parser grammar N;

net: machineEntry+ EOF;

machineEntry: MACHINE ACCOUNT?; """, lg)

def ais = new ANTLRInputStream("""machine random""")

def lexEngine = lg.createLexerInterpreter(ais)

def tokens = new CommonTokenStream(lexEngine)

def parser = g.createParserInterpreter(tokens)

def t = parser.parse(g.rules.get("net").index)

for (Token token in tokens.getTokens()) {

    println("${token} , ${parser.getVocabulary().getSymbolicName(token.getType())}")

}

println(t.toStringTree(parser))

Run the code: groovy example.groovy

Result:

line 1:8 extraneous input 'random' expecting {<EOF>, 'machine'}

[@0,0:6='machine',<2>,1:0] , MACHINE

[@1,8:13='random',<4>,1:8] , ID

[@2,14:13='<EOF>',<-1>,1:14] , EOF

(net (machineEntry machine) random <EOF>)

According to the grammar, the ACCOUNT token is optional so I wonder why the error message is NOT saying that 'account' is also an expected token.

Thanks,

Martin

[Martin d'Anjou]

There is an issue on this:

https://github.com/antlr/antlr4/issues/1428