[ANTLR4] What difference between generated files *.tokens vs. *Lexer.tokens

[Dũng Đặng Minh]

2 generated file *.tokens and *Lexer.tokens (ex: JSON.tokens & JSONLexer.tokens for JSON grammar)

I test in several grammar (include Java) and that files are identical!

So anyone can tell me What difference of those files and Why we need both of them?

[Terence Parr]

ANTLRCreates a tokens file for each grammar that it encounters. As it splits combined grammars into both parser and lexer grammars, you will see two tokens files. The only difference will be that sometimes the parser introduces tokens not found in the lexer, though that’s typically a bug.

Ter

--
You received this message because you are subscribed to the Google Groups "antlr-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to antlr-discussi...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dũng Đặng Minh]

Thank you.
Can you give me specific example grammar file where *.tokens ≠ *Lexer.tokens :D

[Terence Parr]

any grammar with the tokens {…} command

T

On Jul 11, 2014, at 11:08 AM, Dũng Đặng Minh <dung...@gmail.com> wrote:

Thank you.
Can you give me specific example grammar file where *.tokens ≠ *Lexer.tokens :D

[Dũng Đặng Minh]

Thanks you. I got it.

However, in my opinion, it's squandered when generate 2 tokens file almost the same.
Instead, you can use some optional attribute like: <phase = PARSER | LEXER>

For example:

grammar Tok;


tokens { A, B, C }
a : X ;
X: [a-zA-Z]+;

Tokens file is:

<phase = PARSER> A=2

<phase = PARSER> B=3

<phase = PARSER> C=4

X=1

[Sam Harwell]

The file is a temporary file created during the build process. It’s not checked into source control, and it’s not distributed with the final application or library which uses ANTLR. Considering the negligible size of the file(s) in question, I don’t currently see any advantage to attempting to eliminate the creation of one or more of them during the build process.

 

If you encounter a bug in the handling of these files, that would be a separate issue we would want to address.

 

Thanks,

Sam

--