ANLTR 4.5.1: Are modal lexers supported at all with parser grammars? (Bug or dev error?)

[George Spofford]

Modal lexers have been around at least since Lex, and are really handy to support a BNF grammar. The documentation in The Definitive ANTLR Reference is a bit sketchy, as are on-line resources. I have attached a very simple Maven project that demonstrates why I am uncertain.

There is one lexer grammar with 2 modes, and a parser grammar. The parser grammar includes the tokens from the lexer grammar. (If I import the lexer grammar from an import directory, then the non-default mode never gets created. From another thread in this discussion group, that sounds like a known issue.)

Here are the contents:

ModalLexer.g4:

lexer grammar ModalLexer;

ID : [A-Z-a-z]+ ;
INT : [0-9]+ ;
WS : [ \t\r\n]+ -> skip;

MODE_ON : '*-' -> pushMode (MyMode) ;
MODE_OFF : '-*' -> popMode ;

mode MyMode ;

MYMODE_WS : [ \t\r\n]+ -> skip;
FOO : 'foo' ;
FOO_INT : [0-9]+ ;

ModalGrammar.g4:

parser grammar ModalParser;

options {
tokenVocab=ModalLexer;
}

stmt :
ID
| INT
| MODE_ON FOO FOO_INT MODE_OFF
;

The target that is built with maven is a simple executable that takes a string to parse on the command line.

Command lines like the following succeed:

~/experiment/antlr4: mvn exec:java -Dexec.mainClass=com.foo.ParseThis -Dexec.args="foo2"

~/experiment/antlr4: mvn exec:java -Dexec.mainClass=com.foo.ParseThis -Dexec.args="22"

~/experiment/antlr4: mvn exec:java -Dexec.mainClass=com.foo.ParseThis -Dexec.args="foo"

The following fails:

~/experiment/antlr4: mvn exec:java -Dexec.mainClass=com.foo.ParseThis -Dexec.args="*- foo 32 -*"

[INFO] Scanning for projects...

[INFO]

[INFO] ------------------------------------------------------------------------

[INFO] Building test-antlr4-modal 0.1-SNAPSHOT

[INFO] ------------------------------------------------------------------------

[INFO]

[INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ test-antlr4-modal ---

line 1:2 mismatched input '<EOF>' expecting 'foo'

Particulars:

• JDK 1.8_60
  
• Mac running El Capitan
  
• Maven 3.3
  

If this test program is incorrectly formulated, what would be the recommended way to have a modal lexer work with a parser grammar?

Thanks for your assistance!

[Eric Vergnaud]

every input stream ends with an EOF, but that is not supported by the grammar.

I recommend that you buy the definitive ANTLR reference book and read it.

[George Spofford]

Hi Eric,

Thanks for responding! I have read the book a few times, and was a happy user of ANTLR 3+. What are you suggesting here? When you say 'is not supported by the grammar', are you saying I need to include it in my grammar file? 

I don't see any real examples in the book of a modal lexer grammar being used by a parser grammar, but I may have missed something. The index is really sketchy, though as a language book author myself I know how much work goes into a good index.

Thanks,

George

[Eric Vergnaud]

Yes

Also I suggest you use grun to test your grammar, it will provide much better feedback I think.

[George Spofford]

Thank you Eric, that plus some digging on grun got past the problem.

Any 2nd edition of the reference would benefit from some additional detail in this area.