Obtaining Help
Matthew Ballance
I've been working on setting a JavaCC-based grammar for parsing
SystemVerilog (a hardware description and verification language). I
thought I'd tree FreeCC, since this seems to be a project that is more
active. FreeCC gives an error while generating the TokenManager file,
and I can't trace back to the source using the line number provided.
I'm not sure if this is the right group to post to in order to get
help or suggestions. If not, my apologies.
I'm working on this grammar for an open-source project, so I can
post full source if that would be helpful. The error message I get
looks like:
Outputting file: SVParserTokenManager.java
Exception in thread "main" org.visigoths.freecc.parser.TokenMgrError:
Lexical error at line 160, column 145. Encountered: "\n" (10),
after : "\";);"
at
org.visigoths.freecc.parser.FreeCCParserTokenManager.getNextToken(FreeCCParserTokenManager.java:
264)
at
org.visigoths.freecc.parser.FreeCCParser.jj_scan_token(FreeCCParser.java:
465)
at
org.visigoths.freecc.parser.FreeCCParser.jj_3_26(FreeCCParser.java:
8287)
at
org.visigoths.freecc.parser.FreeCCParser.jj_3R_76(FreeCCParser.java:
8671)
at
org.visigoths.freecc.parser.FreeCCParser.jj_3_24(FreeCCParser.java:
8275)
...
Line 160 looks like this in my grammar:
| < INSTANCE: "instance">
| < INT: "int">
| < INTEGER: "integer">
| < INTERFACE: "interface">
| < INTERSECT: "intersect">
| < JOIN: "join"> // << This is line 160
| < JOIN_ANY: "join_any">
| < JOIN_NONE: "join_none">
| < LARGE: "large">
| < LIBLIST: "liblist">
| < LIBRARY: "library">
Jonathan Revusky
<matt.b...@gmail.com> wrote:
>
> Hi,
> I've been working on setting a JavaCC-based grammar for parsing
> SystemVerilog (a hardware description and verification language). I
> thought I'd tree FreeCC, since this seems to be a project that is more
> active. FreeCC gives an error while generating the TokenManager file,
> and I can't trace back to the source using the line number provided.
> I'm not sure if this is the right group to post to in order to get
> help or suggestions. If not, my apologies.
Well, this list is called freecc-devel but it's the only list, so it
must be the right one. :-) At some point in the future, when there
really is a lot more discussion, I'll probably create a freecc-user
list for strictly use-related discussion, but right now volume is so
low that this is effectively a combined development/usage related
list.
So, not only is it perfectly okay to ask for help here, but in effect,
you can take advantage of the early bird special. I'm so eager to get
feedback on this work, and there are currently so few users that I
will be extra helpful.
>
> I'm working on this grammar for an open-source project, so I can
> post full source if that would be helpful.
Yes, please do that.
> The error message I get
> looks like:
> Outputting file: SVParserTokenManager.java
> Exception in thread "main" org.visigoths.freecc.parser.TokenMgrError:
> Lexical error at line 160, column 145. Encountered: "\n" (10),
> after : "\";);"
Hmm, it is hard to see much connection between the error message and
the snippet you included. It may be that the error message generated
is spurious and the problem is elsewhere. Have you tried commenting
out different sections to see if you can narrow down that way where
the real problem is? If you send me your grammar, I'll have a closer
look.
Also, please rebuild FreeCC from the latest code snapshot and try
again. You'll likely hit the same error, but at least any stack traces
will be relative to the current state of the code.
It is very easy to do this. It should be sufficient to simply type:
svn update
ant
on the command line from the root freecc directory to be running the
latest version. (Admittedly that is assuming that you have both svn
and ant installed on your machine, but I'm guessing your probably
do...)
Regards,
JR
Jonathan Revusky
It seems that the problem only occurs if you are using
DEBUG_TOKEN_MANAGER. Of course, it's not the SVParser that is failing
in this instance. It is the FreeCCParser, which makes little sense. I
discovered the problem because I tried to rebuild FreeCC itself with
DEBUG_TOKEN_MANAGER=true and hit the same problem. Also, the error
message is spurious. It gives some location info that has nothing to
do with anything AFAICS.
I tried DEBUG_TOKEN_MANAGER on a smaller grammar that I had been
working on as an example, and it worked on that, so it works in some
cases.
Well, you can definitely assume that I'm going to look into this and
fix it in the coming days. Meanwhile, it seems to work with
DEBUG_TOKEN_MANAGER=false. Or, at least, it all compiles. I don't have
any sample input to test the parser on.
Oh, and you need to do a svn update; ant again because there is a bug
where if USER_CHAR_STREAM is true, then the package declaration got
stuck in twice, and it wasn't compiling that file for that reason.
That's fixed now.
Anyway, you can expect that I'll look into this problem with
DEBUG_TOKEN_MANAGER. Actually, it's been renamed to DEBUG_LEXER, but
the older name still works.
Regards,
JR
Jonathan Revusky
> Hi,
>
> It seems that the problem only occurs if you are using
> DEBUG_TOKEN_MANAGER. Of course, it's not the SVParser that is failing
> in this instance. It is the FreeCCParser, which makes little sense. I
> discovered the problem because I tried to rebuild FreeCC itself with
> DEBUG_TOKEN_MANAGER=true and hit the same problem. Also, the error
> message is spurious. It gives some location info that has nothing to
> do with anything AFAICS.
incomplete. It wasn't hitting the LexicalException when parsing your
SVParser.jj file. It was at a later point in the process. You see, in
FreeCC, the final .java files are generated in a 2-step process. A
first .java file is generated via the appropriate freemarker template,
but then is parsed and code injections are applied. And then the final
.java file is output. So, basically, it was failing when parsing the
SVLexer.java that was generated in the first pass. This was because of
just silly typos in the template, where certain blocks that were
turned on when DEBUG_LEXER was on, under certain conditions output an
extraneous semicolon, for example.
Anyway, this is now fixed. You should be able to to build your parser
using FreeCC both with DEBUG_LEXER=true or false. Also, in similar
situations, the stack trace will now contain the file name where the
error occurred, the lack of which was the reason that the error
message was so perplexing. Once I knew where the error was coming
from, the error message was
So, basically, just do:
svn update
ant
and you should be ready to rock and roll.
Thanks a lot for reporting this. It could have taken a good while
before these issues were brought to my attention otherwise.
Regards,
JR