token index out of range AND rule 'NewLine' contains a lexer command with an unrecognized constant

[Right_Then]

Dear Fellows,

These above mentioned errors appear in Antlr 4.2(onwards) when i write 

• -> channel( NEWLINE_CHANNEL )
  
• -> channel( SINGLELINECOMMENT_CHANNEL )
• etc..

I have looked for solution on web and i know the solution was not to use @members { .....  } but use tokens{ ..., ..., ... } instead to define such constants, for such purposes,

but despite that the problem remains the same for me. It complains the same. I even tried channel(HIDDEN) as in the book. Now error is only about token index out of range.

I have tried to put tokens{...} in main Grammar file and then in lexer Grammar file (that is my all tokens are in a single lexer grammar and parse rules are in various other files

and import is used in main grammar file to mesh it all together) but every time error remains.

I am using Antlr 4.3 compiling with java1.7.

such problems were not there when using AntlrWorks 2 which default to Antlr 4.1.

I apologize i could not find the appropriate answer on web hence i ask or else point me to that please.

Thanks

Regards

RIGHT_THEN

[Terence Parr]

if the grammar is small, can you post it and the error?

Ter

--
You received this message because you are subscribed to the Google Groups "antlr-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to antlr-discussi...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Right_Then]

Thanks for the reply Mr. Terence

No my grammar is not small it is spread over 32 files as small snippets.

but to reproduce unrecognised token warning i cut it down to testing level. it has some symbols and channel tokens are not being recognised

i must be doing something silly!

In the Zip attachment there are three file

1. Png snapshot showing error at cmd prompt
2. ForTest.g4 which is importing Lexer Symbols 
3. Symbols.g4 Lexer grammar where -> channel ( LineComment_channel ) ; are used

Thanks

Regards

Right_Then

[Sam Harwell]

For the example you posted here, you did not define WHITESPACE_CHANNEL in the lexer grammar, which resulted in two issues:

 

1.       Warning 155, which is produced by the ANTLR Tool. This warning appeared because you used a lexer command which referenced a constant value WHITESPACE_CHANNEL which ANTLR does not recognize, and therefore does not know what integer value it represents. You defined WHITESPACE_CHANNEL as a token in the ForTest.g4 grammar, but they would need to be in Symbols.g4 in order to be recognized.

2.       The compilation error when you tried to compile the generated parser. In this case, javac also did not recognize WHITESPACE_CHANNEL because you did not define the token in your lexer grammar.

 

Rather than create a “grammar ForTest;” which imports Symbols, you should instead create a grammar which starts with the following:

 

parser grammar ForTest;

 

options {

  tokenVocab = Symbols;

}

 

You should then move the tokens{} block from ForTest.g4 to Symbols.g4.

 

Sam

--

[Sam Harwell]

Please refer to the following issue for more information about the requirement that you create separate lexer and parser grammars (at least until this issue is resolved).

 

#338: tokens section in combined grammar files can't work

 

Sam

[Right_Then]

Thanks for the reply Mr. Sam

Your observation that i initiated tokens in Common Grammar instead of Lexer Grammar is true only until one uncomments the same

in the Lexer file and comments it in the Common Grammar file. I tried it both ways. I even tried to put Lexer in the Common Grammar file 

for the purpose of this test so as to do away with the importing altogether but it failed every time. As you have briefed the same in the link you have pointed to

that in order for this to succeed import has to go away and one file being Parser Grammar and other being Lexer Grammar will have to be individually compiled.

In which Parser Grammar uses options{} to reference the Lexer tokens or perhaps!! . So I tried it and yes it succeeded until javac phase, but raises errors of java.lang.ClassCastException for ForTest in grun phase. I dont know what it is but this technique surely makes channeling complicated.

I think in order for this to work intuitively why not add signature/keyword in Antlr 4 which creates tokens/constants for channeling etc specifically. If currently allowing such thing could break other aspects of Antlr 4. because it used to work earlier and was intuitive. but again i don't know the internals of Antlr. Currently i am thinking of abandoning channeling altogether because i cannot make all my files parser grammars and one lexer grammar and go about handling both types individually. 

Thanks for the help thou.

Thanks

Regards

Right_Then

[Jim Idle]

This usually 99% of the time means that you have used tokens in the included parser that are not actually declared in the lexer. I thought this had gone away in v4 however. 

Jim

On Jul 6, 2014, at 1:17 AM, "'Right_Then' via antlr-discussion" <antlr-di...@googlegroups.com> wrote:

Thanks for the reply Mr. Terence

No my grammar is not small it is spread over 32 files as small snippets.

but to reproduce unrecognised token warning i cut it down to testing level. it has some symbols and channel tokens are not being recognised

i must be doing something silly!

In the Zip attachment there are three file

1. Png snapshot showing error at cmd prompt
2. ForTest.g4 which is importing Lexer Symbols 
3. Symbols.g4 Lexer grammar where -> channel ( LineComment_channel ) ; are used

Thanks

Regards

Right_Then

--
You received this message because you are subscribed to the Google Groups "antlr-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to antlr-discussi...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

<ErrorWithChannelTokenConstants.7z>