Get line numbers with Antlr parser

[Ulrich Hobelmann]

Hi, I'm using the Java1.5 parser + tree parser by Mike Studman.

My problem: I need to extract the line numbers from the parser.

Funny thing: in one part of the parser it works, but in the tree parser
it doesn't. In java15.g I added some printlns to print getLine() of
some nodes, and they printed the lines. The very same nodes are also
put into a tree structure to read by the java-tree parser (obviously).

Now for some reason the line information disappears while the tree is
being built. In java15.tree.g I can add printlns, but the line number
of the *same* node as in java15.g is suddenly 0, while it was, say 15,
only a second (and I assume a "return") before.

Why is this?

What can / do I have to do, to get the line numbers to the high level,
where I need it?

I also tried various ways of modifying the trees built, including
creating a custom tree (using #() instead of the ^ often used in
java15.g) with some nodes that only have line info. Every time the line
info seems to disappear.

Unfortunately Antlr documentation isn't really helpful. It only
mentions that there *is* the newline() method and that getLine() can be
used, but it doesn't explain why between files the line information
disappears.

(or is this a bug?)

Best regards,
Ulrich

[Kai Koehne]

Ulrich Hobelmann schrieb:

> Hi, I'm using the Java1.5 parser + tree parser by Mike Studman.
>
> My problem: I need to extract the line numbers from the parser.
>
> Funny thing: in one part of the parser it works, but in the tree parser
> it doesn't. In java15.g I added some printlns to print getLine() of
> some nodes, and they printed the lines. The very same nodes are also
> put into a tree structure to read by the java-tree parser (obviously).
>

> [...]

Hi,

antlr creates copies of all node objects in a tree if you have
buildAST=true in your parser/TreeParser. It seems that the information
is lost during that copying. To prevent that, create your own node class
and copy the relevant information in the initialize() methods:

public class MyNode extends CommonAST {
public void initialize(Token t)
{
super.initialize(t);
line = t.getLine();
column = t.getColumn();
}

public void initialize(AST t)
{
super.initialize(t);
line = t.getLine();
column = t.getColumn();
}
}

You have to register this class in all parsers/treewalkers that
manipulate the tree:

MyParser p;
MyTreeWalker w;
// ...
p.setASTNodeClass("MyNode");
w.setASTNodeClass("MyNode");

BTW. the antlr-interest mailing list
(http://www.antlr.org:8080/mailman/listinfo/antlr-interest) might be a
better place for such questions.

Regards,

Kai Koehne

[Ulrich Hobelmann]

Ulrich Hobelmann wrote:
> Now for some reason the line information disappears while the tree is
> being built. In java15.tree.g I can add printlns, but the line number
> of the *same* node as in java15.g is suddenly 0, while it was, say 15,
> only a second (and I assume a "return") before.

[...]

> (or is this a bug?)

I haven't found an answer to the above, but for the record let me post
my solution:
since Antlr seems to exterminate the line numbers when the nodes are
passed between parsers, the workaround was to "wrap" line numbers as
identifiers:
compoundStatement!
: lc:LCURLY!
sts:(statement)*
rc:RCURLY!
{
#compoundStatement =
#(#[SLIST,"SLIST"],
#[IDENT,Integer.toString(lc.getLine())],
#[IDENT,Integer.toString(rc.getLine())],
sts);
}
;
In the tree parser, the IDENTs are then collected and disassembled (i.e.
getText() extracts the text and Integer.parseInt() unwraps them into the
line number they once were). Ugly, but it does the job.

Until last night, that is. This morning I did some more typing, but now
for some reason I get the message: "<AST>: expecting IDENT, found
'<empty tree>'", and I forgot what exactly I changed that suddenly
produces this behavior. Everything seems to work correctly as before,
except for the annoying message.

Of course, Antlr is being true to its tradition of transparency, and
doesn't bother to mention *where* this occurs, so I have - again - no
idea what exactly is going on.

If anybody has any idea, I'd be thankful for that. (Otherwise I just
hope I'll be done with this project as soon as possible (and don't have
to touch it again).)

Regards,
Ulrich