Debugging Lexer

[Jeff Saremi]

This is really a note for @parrt

I am looking at the generated parser in C# so if  the Java version has what I'm asking for please excuse me.

Unlike the generated parser where one can clearly see methods corresponding to the rules defined the lexer and can step through them if needed the lexer has a very brief and binary oriented implementation.
I do not see any log lines either where I could increase or decrease the logging level to get to the bottom of my issues.

What I am looking for is a level to be introduced in the lexer where by one can get really fine level, detailed messages printed as the parsing progresses.

Allow me to demonstrate what I'm looking for and I think it would save thousands and thousands of hours time of developers globally:

sample grammar:

lexer grammar MyPreProcessorLexer;


SHARP:  '#' -> more, pushMode(DIRECTIVE_MODE);
TEXT: ~( '#' |  [\r\n] | '@' ) ~[\r\n]+ ;
ANY_NEW_LINE: '\r'? '\n' ->skip;
SEMI: ';' ;
WS: ' '+ -> skip;


mode DIRECTIVE_MODE;


IF: 'IF' -> pushMode(IF_MODE);
ELSE: 'ELSE' -> popMode;
ELSEIF: 'ELSEIF' -> pushMode(IF_MODE);
ENDIF: 'ENDIF' -> popMode;

Example of detailed log messages:

DEBUG ... current mode: DEFAULT_MODE
DEBUG ... matching text '#' on rule SHARP ; pushing and switching to DIRECTIVE_MODE
DEBUG ... matching text 'IF" on rule IF ; pushing and switching to IF_MODE
DEBUG ... matching text '          ' on rule WS; skipping
DEBUG ... no match for text %
DEBUG ... no match for text &

DEBUG ... mathcing text '\r\n' on rule EOL; popping mode; current mode: DIRECTIVE_MODE
...