ANTLR4 (C++) rule attribute question

[Kevin Cummings]

In ANTLR2, I did the following:

namedef: LET n:NAME EQ v=argument SEMICOLON
{ defineName(#n->getText(), v); }
;

argument returns [int v]
: v=constantValue
( PLUS v2=constantValue
{ v += v2; }
| MINUS v3=constantValue
{ v -= v3; }
)*
;

constantValue returns [int v]
: n:NAME
{ namePtr np = lookupName(#n->getText());
if (np != declaredNames.end())
v = np->second.getValue();
else
error(undefinedName, #n->getText(), #n->getLine());
}
| i:INTEGER
{ v = std::atoi(#i->getText().c_str()); }
;

In ANTLR3, it was:

namedef: LET n=NAME EQ v=argument SEMICOLON
{ pANTLR3_STRING nt = $n.text;
defineName((char *)nt->chars, v);
}
;

argument returns [int v]
: v1=constantValue
{ v = v1; }
( PLUS v2=constantValue
{ v += v2; }
| MINUS v3=constantValue
{ v -= v3; }
)*
;

constantValue returns [int v]
: n=NAME
{ pANTLR3_STRING nt = $n.text;
namePtr np = lookupName((char *)(nt->chars));
if (np != declaredNames.end())
v = np->second.getValue();
else
error(undefinedName, (char *)(nt->chars()), $n.line);
}
| i=INTEGER
{ pANTLR3_STRING it = $i.text;
v = std::atoi((char *)(it->chars));
}
;

I'm trying to find the ANTLR4 equivalent. There are examples of how to
return values in rules in the ANTLR4 documentation, but not of how to
use the returned values in other rules!

namedef: LET NAME EQ v=argument SEMICOLON
{ defineName($NAME.text, v);
// complains that v is not declared in this scope!
}
;

argument returns [int v]
: v1=constantValue
{ $v = v1; }
// complains that v1 is not declared in this scope!
( PLUS v2=constantValue
{ $v += v2; }
// complains that v2 is not declared in this scope!
| MINUS v3=constantValue
{ $v -= v3; }
// complains that v3 is not declared in this scope!
)*
;

constantValue returns [ int v ]
: NAME
{ namePtr np = lookupName($NAME.text);
if (np != declaredNames.end())
$v = np->second.getValue();
else
error(undefinedName, $NAME.text, $NAME.line);
}
|
INTEGER
{ $v = $INTEGER.int;
}
;

Should I be using some different syntactic sugar to reference the
returned values in ANTLR4???? If so, what?

If I try to use $v (instead of just v), antlr4 complains about a missing
template:

error(33): missing code generation template RulePropertyRef_ctxHeader


Yes, I realize that the C++ stuff is still under development. I'm just
trying to port a simple grammar....

I'm using ANTLR4-SNAPSHOT-4.5.4.jar from Mike Lischke, and I cloned the
ANTLR4 git around May 23rd.

Have I stumbled across a feature not yet implemented, or am I trying to
do the wrong thing?

Thanks!

--
Kevin J. Cummings
kjc...@verizon.net
cumm...@kjchome.homeip.net
cumm...@kjc386.framingham.ma.us
Registered Linux User #1232 (http://www.linuxcounter.net/)

[Jim Idle]

Basically, you should move all of the action code and so on into a parser listener or walker. That sounds like a lot of work but it really isn't, especially if you use an IDE to implement the methods. Tell v4 to generate a listener, extend that class and use the IDE to generate the overrides for the methods you need. I tend to use a code gen stack if generating code as I am listening. 

Jim

--
You received this message because you are subscribed to the Google Groups "antlr-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to antlr-discussi...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mike Lischke]

>
> I'm trying to find the ANTLR4 equivalent. There are examples of how to
> return values in rules in the ANTLR4 documentation, but not of how to
> use the returned values in other rules!
>
> namedef: LET NAME EQ v=argument SEMICOLON
> { defineName($NAME.text, v);
> // complains that v is not declared in this scope!
> }
> ;

Use the same syntax as for token references:

... defineName($NAME, $v);

>
> If I try to use $v (instead of just v), antlr4 complains about a missing
> template:
>
> error(33): missing code generation template RulePropertyRef_ctxHeader

Yes, this is a code generation template I hadn't implemented yet. This is now fixed. Re-download the jar.

Mike
--
www.soft-gems.net

[Kevin Cummings]

Thanks Mike!

On 06/04/16 06:03, 'Mike Lischke' via antlr-discussion wrote:
>>
>> I'm trying to find the ANTLR4 equivalent. There are examples of how to
>> return values in rules in the ANTLR4 documentation, but not of how to
>> use the returned values in other rules!
>>
>> namedef: LET NAME EQ v=argument SEMICOLON
>> { defineName($NAME.text, v);
>> // complains that v is not declared in this scope!
>> }
>> ;
>
> Use the same syntax as for token references:
>
> ... defineName($NAME, $v);

done

>>
>> If I try to use $v (instead of just v), antlr4 complains about a missing
>> template:
>>
>> error(33): missing code generation template RulePropertyRef_ctxHeader
>
> Yes, this is a code generation template I hadn't implemented yet. This is now fixed. Re-download the jar.

Getting a new error:

error(67): tbl.g4:540:34: missing attribute access on rule reference v in $v

> Mike

[Mike Lischke]

>> Yes, this is a code generation template I hadn't implemented yet. This is now fixed. Re-download the jar.
>
> Getting a new error:
>
> error(67): tbl.g4:540:34: missing attribute access on rule reference v in $v

Can you send me your grammar? Need to play a bit here with it.

Mike
--
www.soft-gems.net

[Kevin Cummings]

Attaching a pared down version of my grammar. (I removed most of my
codem but left in the actions causing the problem.) When I run your
antlr4 over it, I get the following messages:

> error(67): t.g4:184:34: missing attribute access on rule reference v in $v
> error(67): t.g4:224:16: missing attribute access on rule reference v1 in $v1
> error(67): t.g4:226:12: missing attribute access on rule reference v2 in $v2
> error(67): t.g4:228:12: missing attribute access on rule reference v3 in $v3

[Sam Harwell]

Hi Kevin,

My guess is you need to use $v.ctx instead of just $v to access the rule context object in an action.

Sam

---

From: Kevin Cummings <cumm...@kjchome.homeip.net>
Sent: Jun 5, 2016 1:57 PM
To: antlr-di...@googlegroups.com
Subject: Re: [antlr-discussion] ANTLR4 (C++) rule attribute question

[Kevin Cummings]

On 06/05/16 15:52, Sam Harwell wrote:
> Hi Kevin,
>
> My guess is you need to use $v.ctx instead of just $v to access the rule
> context object in an action.

Same error if I change $v to $v.ctx to try and access the return value.

> Sam

[Eric Vergnaud]

Kevin,

unless I'm missing something it seems v is declared as a java variable, not an antlr one.

Have you tried simply: v = ... ?

Eric

[Kevin Cummings]

On 06/06/16 01:20, Eric Vergnaud wrote:
> Kevin,
>
> unless I'm missing something it seems v is declared as a java variable,

Yes, you are missing something (sorry). I'm testing the latest C++
generation engine, and trying to see if this works there. Mike L. has
been pretty responsive so far. He's looking into this.

> not an antlr one.
> Have you tried simply: v = ... ?

That's where I started. B^)

>
> Eric

[Mike Lischke]

> On 06/05/16 15:52, Sam Harwell wrote:
>> Hi Kevin,
>>
>> My guess is you need to use $v.ctx instead of just $v to access the rule
>> context object in an action.
>
> Same error if I change $v to $v.ctx to try and access the return value.

But Sam is right. When I try your grammar with $v.ctx I get this code:

defineName((std::dynamic_pointer_cast<NamedefContext>(_localctx)->nameToken != nullptr ? std::dynamic_pointer_cast<NamedefContext>(_localctx)->nameToken->getText() : ""), std::dynamic_pointer_cast<NamedefContext>(_localctx)->v);

which only fails to compile because defineName is unknown. I'm a bit surprised myself that you need to use this construct which is kinda reversed to what actually happens, but that's how ANTLR expects it.

Mike
--
www.soft-gems.net

[Kevin Cummings]

But, I'm getting errors when antlr4 runs over my grammar. I'm not
getting to the compilation stage just yet (though I am compiling some of
my support code at this time).

> Mike

[Mike Lischke]

> Am 06.06.2016 um 17:47 schrieb Kevin Cummings <cumm...@kjchome.homeip.net>:
>
> On 06/06/16 09:42, 'Mike Lischke' via antlr-discussion wrote:
>>> On 06/05/16 15:52, Sam Harwell wrote:
>>>> Hi Kevin,
>>>>
>>>> My guess is you need to use $v.ctx instead of just $v to access the rule
>>>> context object in an action.
>>>
>>> Same error if I change $v to $v.ctx to try and access the return value.
>>
>>
>> But Sam is right. When I try your grammar with $v.ctx I get this code:
>>
>> defineName((std::dynamic_pointer_cast<NamedefContext>(_localctx)->nameToken != nullptr ? std::dynamic_pointer_cast<NamedefContext>(_localctx)->nameToken->getText() : ""), std::dynamic_pointer_cast<NamedefContext>(_localctx)->v);
>>
>> which only fails to compile because defineName is unknown. I'm a bit surprised myself that you need to use this construct which is kinda reversed to what actually happens, but that's how ANTLR expects it.
>
> But, I'm getting errors when antlr4 runs over my grammar. I'm not
> getting to the compilation stage just yet (though I am compiling some of
> my support code at this time).
>

Hmm, ok, maybe something I hadn't fixed yet. I updated the jar again. Please re-download and try again.

Mike
--
www.soft-gems.net

[Kevin Cummings]

Err, my bad (I have too many grammar files running around at this
point). YES, ANTLR runs without errors when I run it over the CORRECT
grammar file! (sheesh, my bad).

I'm going to back into compiltation and link mode now, and will report
back when I run into something else!

Sorry for the noise!

[Kevin Cummings]

It works!!!! (kinda!)

OK, here is what I needed to do:

0) I want the lexer to read from a file. I could open a fstream in
antlr2 and antlr3. Antlr4 says I can't do that. Instead, when I ended
up doing was reading the input file into a string using an
istreambuf_iterator (using a hack I found on SO), then having the lexer
read from the string. My code looks like:

ifile.open(infilename);
std::istreambuf_iterator<char> eos;
std::string istring(std::istreambuf_iterator<char>(ifile), eos);
ANTLRInputStream istream(istring);
tblLexer lexer(&istream);

Is there a more direct way? I was used to doing (in antlr2):

ifile.open(infilename)
TBLLexer lexer(ifile);

1) references to return values need to look like:
$rulename.ctx->returnvaluename

so, the pertinent rules from my grammar end up looking like:

> namedef
> : LET NAME EQ argument SEMICOLON
> { defineName($NAME.text, $argument.ctx->v); }
> ;
> argumentList
> : argument
> { table.push_back($argument.ctx->v); }
> ( COMMA argument
> { table.push_back($argument.ctx->v); }
> )*
> ;
> argument returns [ int v ]
> : constantValue
> { $v = $constantValue.ctx->v; }
> ( PLUS v2=constantValue
> { $v += $v2.ctx->v; }
> | MINUS v3=constantValue
> { $v -= $v3.ctx->v; }

> )*
> ;
> constantValue returns [ int v ]
> : NAME
> { namePtr np = lookupName($NAME.text);
> if (np != declaredNames.end())
> $v = np->second.getValue();
> else
> error(undefinedName, $NAME.text, $NAME.line);
> }
> |
> INTEGER
> { $v = $INTEGER.int; }
> ;

Strange how I need to know the "name" of the returned value and need to
use it in the rule that needs it (the ->v part of the references).

OK, antlr4 now runs over the .g4 file without error.

In general, I think this feature needs more documentation and some
examples on how to use it properly. I have some more complicated
grammars that use return values (and arguments) heavily. But, those
grammars used to generate ASTs. I guess I'm going to have to change
them to use visitors/listeners with Antlr4 and build my own ASTs.

2) Now I need to edit the generated Lexer.h, Parser.h, Lexer.cpp, and
Parser.cpp files, and change the following lines:

using namespace antlr4;
to
using namespace org::antlr::v4::runtime

Now all the code compiles without error (main, lexer, parser).

3) Linked by including the libantlr4.a file from the cloned git (after
building it).

4) Now I can run my program, and it generates the same output as my
antlr2 and antlr3 versions for the same input files! Success!

Thank for all the help Mike. I assume that 2) is something that changed
in the development stuff, and will eventually work without manual
intervention?

[Mike Lischke]

Hey Kevin,

> 0) I want the lexer to read from a file. I could open a fstream in
> antlr2 and antlr3. Antlr4 says I can't do that. Instead, when I ended
> up doing was reading the input file into a string using an
> istreambuf_iterator (using a hack I found on SO), then having the lexer
> read from the string. My code looks like:
>
> ifile.open(infilename);
> std::istreambuf_iterator<char> eos;
> std::string istring(std::istreambuf_iterator<char>(ifile), eos);
> ANTLRInputStream istream(istring);
> tblLexer lexer(&istream);
>
> Is there a more direct way? I was used to doing (in antlr2):
>
> ifile.open(infilename)
> TBLLexer lexer(ifile);

It's still possible that way. Here's how we do it in the runtime tests:

int main(int argc, const char* argv[]) {
std::wifstream stream;
stream.open(argv[1]);
ANTLRInputStream input(stream);
...

Additionally, there is an ANTLRFileStream class that accepts a file name and uses a wifstream to do the job. Keep in mind the content of the file must be UTF-8 encoded!

>
> 1) references to return values need to look like:
> $rulename.ctx->returnvaluename
>

This is not a C++ target specific problem, but dictated by ANTLR itself. However, I'd expect that $returnvaluename alone should work (and is then fully target language independent). However, with ANTLR4 handling such target specific stuff from actions to listeners. The latter allows to remove all language specific stuff from the grammar, allowing so to reuse the grammar for multiple languages and to have much more power at hand when acting on certain parsing events.

> In general, I think this feature needs more documentation and some
> examples on how to use it properly. I have some more complicated
> grammars that use return values (and arguments) heavily.

This is something probably only Terence can answer.

>
> 2) Now I need to edit the generated Lexer.h, Parser.h, Lexer.cpp, and
> Parser.cpp files, and change the following lines:
>
> using namespace antlr4;
> to
> using namespace org::antlr::v4::runtime
>
> Now all the code compiles without error (main, lexer, parser).
>
> 3) Linked by including the libantlr4.a file from the cloned git (after
> building it).

Hmm, there is no libantlr4.a file in the repo. Where did you get this from? There shouldn't be any precompiled libraries in the repo at all and I'm certain that what you have is not the latest one, hence the trouble with the namespace. I only recently moved from org::antlr::v4:.runtime to antlr4.

>
> 4) Now I can run my program, and it generates the same output as my
> antlr2 and antlr3 versions for the same input files! Success!

Great to hear. Can you already say anything about the performance? The ANTLR4 version of my parser is orders of magnitudes slower than the ANTLR3 one. This is partially caused by ANTLR and partially by the C++ runtime. I'm still in the process of optimizing the that.


Mike
--
www.soft-gems.net

[Kevin Cummings]

On 06/10/16 03:06, 'Mike Lischke' via antlr-discussion wrote:
> Hey Kevin,
>
>> 0) I want the lexer to read from a file. I could open a fstream in
>> antlr2 and antlr3. Antlr4 says I can't do that. Instead, when I ended
>> up doing was reading the input file into a string using an
>> istreambuf_iterator (using a hack I found on SO), then having the lexer
>> read from the string. My code looks like:
>>
>> ifile.open(infilename);
>> std::istreambuf_iterator<char> eos;
>> std::string istring(std::istreambuf_iterator<char>(ifile), eos);
>> ANTLRInputStream istream(istring);
>> tblLexer lexer(&istream);
>>
>> Is there a more direct way? I was used to doing (in antlr2):
>>
>> ifile.open(infilename)
>> TBLLexer lexer(ifile);
>
> It's still possible that way. Here's how we do it in the runtime tests:
>
> int main(int argc, const char* argv[]) {
> std::wifstream stream;

What is a "wifstream"? Stroustrup's book (C++11, Fourth Edition)
doesn't mention it.

> stream.open(argv[1]);
> ANTLRInputStream input(stream);
> ...
>
> Additionally, there is an ANTLRFileStream class that accepts a file name and uses a wifstream to do the job. Keep in mind the content of the file must be UTF-8 encoded!

Didn't see that one when I looked at the code. Good to know. I'll look
at the code again. Thanks.

>>
>> 1) references to return values need to look like:
>> $rulename.ctx->returnvaluename
>>
>
> This is not a C++ target specific problem, but dictated by ANTLR itself. However, I'd expect that $returnvaluename alone should work (and is then fully target language independent). However, with ANTLR4 handling such target specific stuff from actions to listeners. The latter allows to remove all language specific stuff from the grammar, allowing so to reuse the grammar for multiple languages and to have much more power at hand when acting on certain parsing events.

I had trouble with that form. There are also ambiguity problems when 2
rules return the same named variable (the second being assigned from the
first, essentially: $v = $v).

>> In general, I think this feature needs more documentation and some
>> examples on how to use it properly. I have some more complicated
>> grammars that use return values (and arguments) heavily.

> This is something probably only Terence can answer.
>
>>
>> 2) Now I need to edit the generated Lexer.h, Parser.h, Lexer.cpp, and
>> Parser.cpp files, and change the following lines:
>>
>> using namespace antlr4;
>> to
>> using namespace org::antlr::v4::runtime
>>
>> Now all the code compiles without error (main, lexer, parser).
>>
>> 3) Linked by including the libantlr4.a file from the cloned git (after
>> building it).
>
> Hmm, there is no libantlr4.a file in the repo. Where did you get this from? There shouldn't be any precompiled libraries in the repo at all and I'm certain that what you have is not the latest one, hence the trouble with the namespace. I only recently moved from org::antlr::v4:.runtime to antlr4.

yes, I said: "(after building it)". I went to antlr/runtime/Cpp/ and
built the runtime. It created a build/runtime folder containing:

CMakeFiles cmake_install.cmake libantlr4.a libantlr4.so
libantlr4.so.4.0.0 Makefile

I did not "install" it, but rather I'm referencing it directly from my
builds (until I do install it). (yes, I'm using the .a file, not the
.so library for now)

>>
>> 4) Now I can run my program, and it generates the same output as my
>> antlr2 and antlr3 versions for the same input files! Success!
>
> Great to hear. Can you already say anything about the performance? The ANTLR4 version of my parser is orders of magnitudes slower than the ANTLR3 one. This is partially caused by ANTLR and partially by the C++ runtime. I'm still in the process of optimizing the that.

Sorry, no, the grammar was small, and so was the example I fed to it.
But, don't let that stop you from optimizing the C++ runtime! B^)
Here is the "time" results from running my antlr2, antlr3, and antlr4
versions of the runtime with the same input file:

antlr2:
real 0m0.649s
user 0m0.003s
sys 0m0.007s

antlr3:
real 0m0.195s
user 0m0.003s
sys 0m0.003s

antlr4:
real 0m2.168s
user 0m0.020s
sys 0m0.006s

Its a noticeable difference when you look at the "user" times.

> Mike

[Mike Lischke]

>> It's still possible that way. Here's how we do it in the runtime tests:
>>
>> int main(int argc, const char* argv[]) {
>> std::wifstream stream;
>
> What is a "wifstream"? Stroustrup's book (C++11, Fourth Edition)
> doesn't mention it.

Does it have to mention all C++ STL classes that exist :-D ? Look here: http://www.cplusplus.com/reference/fstream/wifstream/. It's just a normal input file stream with a wchar_t type. The nice thing about it is that you can imbue various converters for encodings, which makes it easy to convert e.g. utf-8 file content to utf32, as used internally. We can probably provide more acceptable input formats in the future, but I thought utf-8 is enough for the moment :-)

>
>> stream.open(argv[1]);
>> ANTLRInputStream input(stream);
>> ...
>>
>> Additionally, there is an ANTLRFileStream class that accepts a file name and uses a wifstream to do the job. Keep in mind the content of the file must be UTF-8 encoded!
>
> Didn't see that one when I looked at the code. Good to know. I'll look
> at the code again. Thanks.

Keep in mind, also the file name must be utf-8 encoded.

>>
>> Hmm, there is no libantlr4.a file in the repo. Where did you get this from? There shouldn't be any precompiled libraries in the repo at all and I'm certain that what you have is not the latest one, hence the trouble with the namespace. I only recently moved from org::antlr::v4:.runtime to antlr4.
>
> yes, I said: "(after building it)". I went to antlr/runtime/Cpp/ and
> built the runtime. It created a build/runtime folder containing:

Ah, ok, missed that one.

>
> CMakeFiles cmake_install.cmake libantlr4.a libantlr4.so
> libantlr4.so.4.0.0 Makefile
>
> I did not "install" it, but rather I'm referencing it directly from my
> builds (until I do install it). (yes, I'm using the .a file, not the
> .so library for now)

Ok, so cmake works for you, fine. What platform is that?

> antlr2:
> real 0m0.649s
> user 0m0.003s
> sys 0m0.007s
>
> antlr3:
> real 0m0.195s
> user 0m0.003s
> sys 0m0.003s
>
> antlr4:
> real 0m2.168s
> user 0m0.020s
> sys 0m0.006s

10 times slower, yes I see. The parser is still fast, but the lexer has become really slow. I have a test case where the Java target gives me 2.9s for a parse run, where the parser alone takes 0.5secs, the rest is spent in the lexer. In the C++ target this becomes much much worse: 13s/0.12s. Need to dig deeper, to find the culprit.

Mike
--
www.soft-gems.net

[Kevin Cummings]

On 06/10/16 12:44, 'Mike Lischke' via antlr-discussion wrote:
>> yes, I said: "(after building it)". I went to antlr/runtime/Cpp/ and
>> built the runtime. It created a build/runtime folder containing:
>
> Ah, ok, missed that one.
>
>>
>> CMakeFiles cmake_install.cmake libantlr4.a libantlr4.so
>> libantlr4.so.4.0.0 Makefile
>>
>> I did not "install" it, but rather I'm referencing it directly from my
>> builds (until I do install it). (yes, I'm using the .a file, not the
>> .so library for now)
>
> Ok, so cmake works for you, fine. What platform is that?

Fedora 22 on my laptop (Dell Inspiron 1520).

>> antlr2:
>> real 0m0.649s
>> user 0m0.003s
>> sys 0m0.007s
>>
>> antlr3:
>> real 0m0.195s
>> user 0m0.003s
>> sys 0m0.003s
>>
>> antlr4:
>> real 0m2.168s
>> user 0m0.020s
>> sys 0m0.006s
>
> 10 times slower, yes I see. The parser is still fast, but the lexer has become really slow. I have a test case where the Java target gives me 2.9s for a parse run, where the parser alone takes 0.5secs, the rest is spent in the lexer. In the C++ target this becomes much much worse: 13s/0.12s. Need to dig deeper, to find the culprit.
>
> Mike
>

--