BUG: including a 'lexer grammar' with modes

[Ramon Mendes]

I am using lexer modes in my grammar. As I discovered, it requires me to write the lexer rules in a separated file, a lexer grammar.

In order for my parser grammar to use this lexer grammar, I am importing it with an 'import' obviously.

I made a sample with the XML grammar found in the book:

Bug.g4

grammar Bug;

import Lex;

file: (tag | TEXT)* ;

tag : '<' ID '>'

| '<' '/' ID '>'

;

Lex.g4

lexer grammar Lex;

// Default mode rules (the SEA)

OPEN : '<' -> mode(ISLAND) ; // switch to ISLAND mode

TEXT : ~'<'+ ; // clump all text together

mode ISLAND;

CLOSE : '>' -> mode(DEFAULT_MODE) ; // back to SEA mode

SLASH : '/' ;

ID : [a-zA-Z]+ ;

Running the ANTLR tool generates the code with no error.

Problem is that the generated lexer (the class from BugLexer.java file) doesn't contains the code for handling the lexical modes. That is, when I import a lexer from my parser grammar, it seems to ignore the lexer modes.

You can note that from the content of the BugLexer class:

public static String[] modeNames = {

"DEFAULT_MODE"

};

Where is the ISLAND mode?

[Sam Harwell]

Hi Ramon,

 

You are correct that this situation is a bug. ANTLR should be reporting an error when you try to import a lexer with modes. Please report that situation on the GitHub issue tracker so we can get it fixed for to the next release:

https://github.com/antlr/antlr4/issues

 

The solution to your specific problem is different:

 

1.       Make Bug.g4 a “parser grammar” instead of just “grammar” (the way you have it now is a combined grammar with both a parser and lexer, but you only need a parser because the lexer is defined separately).

2.       Instead of using an “import” statement, you should be setting the tokenVocab option for your grammar, as follow.

 

options {

  tokenVocab=Lex;

}

 

Thank you,

Sam Harwell

 

From: antlr-di...@googlegroups.com [mailto:antlr-di...@googlegroups.com] On Behalf Of Ramon Mendes
Sent: Saturday, August 10, 2013 3:26 PM
To: antlr-di...@googlegroups.com
Subject: [antlr-discussion] BUG: including a 'lexer grammar' with modes

 

I am using lexer modes in my grammar. As I discovered, it requires me to write the lexer rules in a separated file, a lexer grammar.

In order for my parser grammar to use this lexer grammar, I am importing it with an 'import' obviously.

 

I made a sample with the XML grammar found in the book:

 

Bug.g4

grammar Bug;

 

import Lex;

 

 

file: (tag | TEXT)* ;

 

tag : '<' ID '>'

      | '<' '/' ID '>'

      ;

Lex.g4

lexer grammar Lex;

 

 

// Default mode rules (the SEA)

OPEN : '<' -> mode(ISLAND) ; // switch to ISLAND mode

TEXT : ~'<'+ ; // clump all text together

 

 

mode ISLAND;

CLOSE : '>' -> mode(DEFAULT_MODE) ; // back to SEA mode

SLASH : '/' ;

ID : [a-zA-Z]+ ;

 

Running the ANTLR tool generates the code with no error.

Problem is that the generated lexer (the class from BugLexer.java file) doesn't contains the code for handling the lexical modes. That is, when I import a lexer from my parser grammar, it seems to ignore the lexer modes.

--
You received this message because you are subscribed to the Google Groups "antlr-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to antlr-discussi...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jim Idle]

Don't include the lexer. Just import it's tokens and compile together. 

Jim