No error alternatives in Lexer

[George S Cowan]

I tried using both

{notifyErrorListeners("unclosed char literal");}

and

{false}?<fail="unclosed char literal">

in the lexer but couldn't get them to work. There were lots of surprising syntax errors and endless loops that I will spend some time documenting on GitHub, but for now I just wanted to raise the flag.

grammar CharExperiment_01;
stat : 'start' CharacterLiteral 'end' EOF;

// Lexer

CharacterLiteral
    :   '\'' SingleCharacter '\''
    |   '\'' SingleCharacter SingleCharacter+ '\'' {false}?<fail='Char literal may only have one char'>
               // fail compiles if single quotes used instead of double,
               // but does not issue message on error
    |   '\'' {notifyErrorListeners("unclosed char literal");}
               // notifyErrorListeners method is not in the lexer error listener
    ;

fragment
SingleCharacter
    :   ~['\\\r\n]
    ;

WS   : [ \r\t\n]+ -> skip ;

George

[George S Cowan]

As Sam Harwell commented on Issue 317, "ANTLR 4 lexers do not (and cannot) support customized error messages for failed predicates, because they function as DFA state machines instead of top-down parsers." Based on his suggestions, here is a working example of fudging lexer error alternatives in the parser:

grammar CharExperiment_04;
stat : 'start' charLit 'end' EOF;

charLit
    :   CharacterLiteral
    |   BadCharLiteral      {false}?<fail='Char literal may only contain one character'>
    |   EmptyCharLiteral    {false}?<fail='Empty char literal'>
    |   UnclosedCharLiteral {false}?<fail='Unclosed character literal'>
    ;

// Lexer

CharacterLiteral
    :   '\'' SingleCharacter '\''
    ;

BadCharLiteral
    :   '\'' SingleCharacter SingleCharacter+ '\''
    ;

EmptyCharLiteral
    :   '\'' '\''
    ;

UnclosedCharLiteral
    :   '\''

    ;

fragment
SingleCharacter
    :   ~['\\\r\n]
    ;

WS   : [ \r\t\n]+ -> skip ;

You may use

{notifyErrorListeners("my message");}

instead of

{false}?<fail='my message'>

but with the false/fail convention, the error messages are a little cleaner.

George

On Monday, August 12, 2013 12:36:28 PM UTC-4, George S Cowan wrote:

I tried using both

{notifyErrorListeners("unclosed char literal");}

and

{false}?<fail="unclosed char literal">

in the lexer but couldn't get them to work. There were lots of surprising syntax errors and endless loops that I will spend some time documenting on GitHub, but for now I just wanted to raise the flag.

grammar CharExperiment_01;
stat : 'start' CharacterLiteral 'end' EOF;

// Lexer

CharacterLiteral
    :   '\'' SingleCharacter '\''
    |   '\'' SingleCharacter SingleCharacter+ '\'' {false}?<fail='Char literal may only have one char'>
               // fail compiles if single quotes used instead of double,
               // but does not issue message on error

    |   '\'' ~[\r\n] {notifyErrorListeners("unclosed char literal");}

               // notifyErrorListeners method is not in the lexer error listener
    ;

fragment
SingleCharacter
    :   ~['\\\r\n]
    ;

WS   : [ \r\t\n]+ -> skip ;

George