ANTLR 4 C++ target

[David Come]

Hello,

I would like to try the C++ 14 (experimental) target : https://github.com/antlr/antlr4-cpp but the project seems inactive.
Firstly, is it usable ? If yes, how so ?  Because I have to admit I have no clue at all how to use it (under linux).

Thanks,
David

[Mike Lischke]

Hey David,

> I would like to try the C++ 14 (experimental) target : https://github.com/antlr/antlr4-cpp but the project seems inactive.
> Firstly, is it usable ? If yes, how so ? Because I have to admit I have no clue at all how to use it (under linux).

The C++ experimental target has been stopped a while ago. Instead try the new C++11 target from: https://github.com/DanMcLaughlin/antlr4 (see the runtime/Cpp folder for instructions and code).

Mike
--
www.soft-gems.net

[David Come]

I wasn't looking specifically for a C++14 target, any ANTLR C++ target was good for me =)
But I have to say, the C++11 target NEVER EVER came up while I was googling that.

You made my day !

[Mike Lischke]

> I wasn't looking specifically for a C++14 target, any ANTLR C++ target was good for me =)
> But I have to say, the C++11 target NEVER EVER came up while I was googling that.

Hmm, that's strange. The C++14 target comes up as second hit. On the other hand you have many hits for postings in this mailing list, which discuss the C++ target.

Btw: the one I gave you is still in experimental status. We are currently trying to get the runtime tests ready which check conformity. However, I was already able to build a complex parser for MySQL which passes like 90% of our internal tests. So it's definitely mature enough for beta testing. However, expect memory problems or crashes still.

I created a new jar (http://www.soft-gems.net/files/antlr4-4.5.4-SNAPSHOT.jar) that you can use to generate your parser files. Use that e.g. to generate the files for the demo project. Note however that the Visual Studio Project needs an update yet, while cmake + XCode variants should work out of the box.

Mike
--
www.soft-gems.net

[Mike Lischke]

> I created a new jar (http://www.soft-gems.net/files/antlr4-4.5.4-SNAPSHOT.jar) that you can use to generate your parser files. Use that e.g. to generate the files for the demo project. Note however that the Visual Studio Project needs an update yet, while cmake + XCode variants should work out of the box.

The Visual Studio project is now updated and the demos works like on the other platforms. So all what's needed now is this:

1) Download the jar and place it in the demo folder.
2) Open a terminal and run generate.cmd (Windows) or generate.sh (Linux, OSX).
3) Open the demo project in XCode or Visual Studio and compile it (cmake for Linux).
4) Run and enjoy the parse tree output generated from the simple demo grammar + expression input.

As a side note: the ANTLR 4 C++ runtime is fully Unicode aware, even for identifiers. This works in the Java runtime already (because there everything beyond the Unicode BMP is converted to surrogate pairs which are lexed by ANTLR as 2 individual characters and if surrogate pairs are included in an identifier rule you can parse the full Unicode range).

Unfortunately, ANTLR does not generate proper ATN sets for the full Unicode range, so I added a little hack: if there is a character range that ends at 0xFFFF this is implicitly extended to the full Unicode range up to 0x10FFFF in the C++ runtime. This little trick will be removed once ANTLR exports proper sets.

Mike
--
www.soft-gems.net

[David Come]

Le jeudi 19 mai 2016 15:32:49 UTC+2, Mike Lischke a écrit :

> I wasn't looking specifically for a C++14 target, any ANTLR C++ target was good for me =)
> But I have to say, the C++11 target NEVER EVER came up while I was googling that.


Hmm, that's strange. The C++14 target comes up as second hit. On the other hand you have many hits for postings in this mailing list, which discuss the C++ target

I completely missed it. I did a PR on the C++14 target to change the README and make it more explicit.

Btw: the one I gave you is still in experimental status. We are currently trying to get the runtime tests ready which check conformity. However, I was already able to build a complex parser for MySQL which passes like 90% of our internal tests. So it's definitely mature enough for beta testing. However, expect memory problems or crashes still.

I'm not going to use it in production, I'm a phd student and I would like to parse a small language of my own which will ask clang/libtooling some stuff. Having a C++ target enables me to bypass Java and JNA/JNI and other foreign functions interfaces to call C++ from Java/Scala.
 

I created a new jar (http://www.soft-gems.net/files/antlr4-4.5.4-SNAPSHOT.jar) that you can use to generate your parser files. Use that e.g. to generate the files for the demo project. Note however that the Visual Studio Project needs an update yet, while cmake + XCode variants should work out of the box.

Mike
--
www.soft-gems.net

Thank you very much. I got everything up and running (\o/) but also a minor discrepancy on my test file between the Java target and C++ one on comments. Where can I report the bug ?

[Mike Lischke]

I created a new jar (http://www.soft-gems.net/files/antlr4-4.5.4-SNAPSHOT.jar) that you can use to generate your parser files. Use that e.g. to generate the files for the demo project. Note however that the Visual Studio Project needs an update yet, while cmake + XCode variants should work out of the box. 

Thank you very much. I got everything up and running (\o/) but also a minor discrepancy on my test file between the Java target and C++ one on comments. Where can I report the bug ?

You can file a bug report on Github (in the fork, not original ANTLR).

Mike

-- 
www.soft-gems.net

[Kevin Cummings]

Mike (and Dan) Great job!

On 05/19/16 11:42, 'Mike Lischke' via antlr-discussion wrote:
>
>> I created a new jar (http://www.soft-gems.net/files/antlr4-4.5.4-SNAPSHOT.jar) that you can use to generate your parser files. Use that e.g. to generate the files for the demo project. Note however that the Visual Studio Project needs an update yet, while cmake + XCode variants should work out of the box.

Thank-you MIKE! I was having lots of problems building from the cloned
GIT sources. Using this snapshot is a snap (yes, pun intended!)

I have a small (enough) grammar that I an testing out. I originally
write it for ANTLR2, ported it to ANTLR3 (and made major modifications
to be able to use the C target under C++), and now I am trying it under
your C++ ANTLR4. How great it was to be able to remove most of the
ANTLR3 stuff and return to ANTLR4 attributes! yeah!

Using your snapshot, I can generate the Lexer and Parser files. Using
your demo as a template, I have created a main.cpp to drive the program.

My problem is that I can get NONE of the files to compile. They all
fail with a similar error: Ref is an undefined type. It is referring
to all of the uses of the (supposedly) templated class Ref. I cannot
find a file anywhere in the snapshot that defines it! Where does the
C++ definition for this class live? I'm pretty sure that this is my
problem, but I've given up grepping my system for it. [I did find a
defintion online in someone else's git, but I thought that if ANTLR4 is
using it, should ANTLR4 provide its definition?

I am using your tool in a very virgin environment. I run the tool from
the command line, and I am now trying to build a small Makefile to do
the work for me. So far, none of the compilations compile to
completion. I am compiling with the following options:

> CC = g++
> CFLAGS = -fPIC -shared -g -O3 -Wall -std=c++11 $(NUMERRORS)
> LDFLAGS =
> IFLAGS =-I../antlrcpp -I. -I$(RUNTIME) -I$(RUNTIME)/atn -I$(RUNTIME)/dfa \
> -I$(RUNTIME)/misc -I$(RUNTIME)/tree -I$(RUNTIME)/tree/pattern \
> -I$(RUNTIME)/tree/xpath -I$(RUNTIME)/support
(cut and pasted (virtually) from the Makefile in the demo directory)

[This is a Fedora 22 system, and it doesn't matter if I use g++ or CLang
to do the compilation. The errors are equivalent.]

And, as a side note, I can find no directory named antlrcpp in the
snapshot either (I can find one named "antlrcpp Test").

Thanks for any insight you can provide!

--
Kevin J. Cummings
kjc...@verizon.net
cumm...@kjchome.homeip.net
cumm...@kjc386.framingham.ma.us
Registered Linux User #1232 (http://www.linuxcounter.net/)

[Mike Lischke]

Kevin,

>
> My problem is that I can get NONE of the files to compile. They all
> fail with a similar error: Ref is an undefined type. It is referring
> to all of the uses of the (supposedly) templated class Ref. I cannot
> find a file anywhere in the snapshot that defines it!

Well, the snapshot is pure Java code (to generate the parser files), no wonder you don't find the Ref<> template there. Instead include antlr4-runtime.h, which is the umbrella header for the runtime and gives you everything you need. See the demo, it uses exactly that. Does the demo compile for you?

> Where does the
> C++ definition for this class live? I'm pretty sure that this is my
> problem, but I've given up grepping my system for it. [I did find a
> defintion online in someone else's git, but I thought that if ANTLR4 is
> using it, should ANTLR4 provide its definition?

It's in runtime/src/antlr4-common.h. This header is included with the umbrella header. Just use that one and you should be done.

>
> I am using your tool in a very virgin environment. I run the tool from
> the command line, and I am now trying to build a small Makefile to do
> the work for me. So far, none of the compilations compile to
> completion.

Can't you use cmake instead? There is a cmake file in the runtime/ folder.

> I am compiling with the following options:
>
>> CC = g++
>> CFLAGS = -fPIC -shared -g -O3 -Wall -std=c++11 $(NUMERRORS)
>> LDFLAGS =
>> IFLAGS =-I../antlrcpp -I. -I$(RUNTIME) -I$(RUNTIME)/atn -I$(RUNTIME)/dfa \
>> -I$(RUNTIME)/misc -I$(RUNTIME)/tree -I$(RUNTIME)/tree/pattern \
>> -I$(RUNTIME)/tree/xpath -I$(RUNTIME)/support
> (cut and pasted (virtually) from the Makefile in the demo directory)

Are you using the latest revision? I see no make file in the demo folder, just a cmake file. Btw, it shouldn't be necessary to include all subfolders in the header search path. The headers are all included with relative paths.

>
> [This is a Fedora 22 system, and it doesn't matter if I use g++ or CLang
> to do the compilation. The errors are equivalent.]
>
> And, as a side note, I can find no directory named antlrcpp in the
> snapshot either (I can find one named "antlrcpp Test").

Indeed there is no such folder. You are looking at the wrong make file. This should be your guide: https://github.com/DanMcLaughlin/antlr4/blob/master/runtime/Cpp/CMakeLists.txt. Or use it as is, it should work out of the box.

Actually, I just realized that I had the build instructions for the runtime in the wrong readme (that of the demo). I moved that to the library readme now: https://github.com/DanMcLaughlin/antlr4/tree/master/runtime/Cpp.

Mike
--
www.soft-gems.net

[Kevin Cummings]

On 05/22/16 05:19, 'Mike Lischke' via antlr-discussion wrote:
> Kevin,
>
>>
>> My problem is that I can get NONE of the files to compile. They all
>> fail with a similar error: Ref is an undefined type. It is referring
>> to all of the uses of the (supposedly) templated class Ref. I cannot
>> find a file anywhere in the snapshot that defines it!
>
> Well, the snapshot is pure Java code (to generate the parser files), no wonder you don't find the Ref<> template there. Instead include antlr4-runtime.h, which is the umbrella header for the runtime and gives you everything you need. See the demo, it uses exactly that. Does the demo compile for you?

No, the snapshot contains compile-time code (headers) as well. But,
nothing that defines Ref.

>> Where does the
>> C++ definition for this class live? I'm pretty sure that this is my
>> problem, but I've given up grepping my system for it. [I did find a
>> defintion online in someone else's git, but I thought that if ANTLR4 is
>> using it, should ANTLR4 provide its definition?
>
> It's in runtime/src/antlr4-common.h. This header is included with the umbrella header. Just use that one and you should be done.

There is no runtime/src directory in the snapshot....

I'll go back and look in the cloned-git directory....

>>
>> I am using your tool in a very virgin environment. I run the tool from
>> the command line, and I am now trying to build a small Makefile to do
>> the work for me. So far, none of the compilations compile to
>> completion.
>
> Can't you use cmake instead? There is a cmake file in the runtime/ folder.

I'd prefer to use eclipse, but, .... I've never used cmake before. It
appears to be a system generator as opposed to a system builder.
It probably needs some configuration that I've never done before.

>> I am compiling with the following options:
>>
>>> CC = g++
>>> CFLAGS = -fPIC -shared -g -O3 -Wall -std=c++11 $(NUMERRORS)
>>> LDFLAGS =
>>> IFLAGS =-I../antlrcpp -I. -I$(RUNTIME) -I$(RUNTIME)/atn -I$(RUNTIME)/dfa \
>>> -I$(RUNTIME)/misc -I$(RUNTIME)/tree -I$(RUNTIME)/tree/pattern \
>>> -I$(RUNTIME)/tree/xpath -I$(RUNTIME)/support
>> (cut and pasted (virtually) from the Makefile in the demo directory)
>
> Are you using the latest revision? I see no make file in the demo folder, just a cmake file. Btw, it shouldn't be necessary to include all subfolders in the header search path. The headers are all included with relative paths.

Yeah, that's what I stole it from. I kept getting some files not found
until I put everything in.

>> [This is a Fedora 22 system, and it doesn't matter if I use g++ or CLang
>> to do the compilation. The errors are equivalent.]
>>
>> And, as a side note, I can find no directory named antlrcpp in the
>> snapshot either (I can find one named "antlrcpp Test").
>
> Indeed there is no such folder. You are looking at the wrong make file. This should be your guide: https://github.com/DanMcLaughlin/antlr4/blob/master/runtime/Cpp/CMakeLists.txt. Or use it as is, it should work out of the box.

Thanks, I must have missed that reference. I will read it.

> Actually, I just realized that I had the build instructions for the runtime in the wrong readme (that of the demo). I moved that to the library readme now: https://github.com/DanMcLaughlin/antlr4/tree/master/runtime/Cpp.

B^}

> Mike

[Mike Lischke]

>> Well, the snapshot is pure Java code (to generate the parser files), no wonder you don't find the Ref<> template there. Instead include antlr4-runtime.h, which is the umbrella header for the runtime and gives you everything you need. See the demo, it uses exactly that. Does the demo compile for you?
>
> No, the snapshot contains compile-time code (headers) as well. But,
> nothing that defines Ref.

Can you tell me under which path you see header files? I just looked in the jar and see only class files and some support files (dot mostly). No headers at all.

>
>>> Where does the
>>> C++ definition for this class live? I'm pretty sure that this is my
>>> problem, but I've given up grepping my system for it. [I did find a
>>> defintion online in someone else's git, but I thought that if ANTLR4 is
>>> using it, should ANTLR4 provide its definition?
>>
>> It's in runtime/src/antlr4-common.h. This header is included with the umbrella header. Just use that one and you should be done.
>
> There is no runtime/src directory in the snapshot....

Of course not! The snapshot is just the ANTLR tool. It should not have anything else but the ANTLR java code + some documentation. I was referring to the C++ runtime, which is not in the jar (the various runtimes have never been part of the ANTLR jar AFAIR).

>
> I'll go back and look in the cloned-git directory....

Yes, that's what you need to do. Clone the repo, place the jar into the demo folder and run the generate.sh script to generate the parser/lexer/visitor/listener files. Then compile everything using cmake. You can install cmake just like make from your yum or whatever respository Fedora has (I'm not very familiar with Linux tbh.). Then execute the steps as given for cmake. Should be very simple. If that doesn't compile then we may have a problem in the cmake file. But I've been told it would nicely compile the code and create the demo binary.

>
>>>
>>> I am using your tool in a very virgin environment. I run the tool from
>>> the command line, and I am now trying to build a small Makefile to do
>>> the work for me. So far, none of the compilations compile to
>>> completion.
>>
>> Can't you use cmake instead? There is a cmake file in the runtime/ folder.
>
> I'd prefer to use eclipse, but, .... I've never used cmake before. It
> appears to be a system generator as opposed to a system builder.
> It probably needs some configuration that I've never done before.

Hmm, I haven't used Eclipse for C++ development at all, but I guess it's not so much different than other IDEs. Create a new console app, add the main.cpp file to it, maybe even the full C++ runtime if you don't want to use a library and that should be it.


Mike
--
www.soft-gems.net

[Kevin Cummings]

On 05/22/16 10:48, 'Mike Lischke' via antlr-discussion wrote:
>>> Well, the snapshot is pure Java code (to generate the parser files), no wonder you don't find the Ref<> template there. Instead include antlr4-runtime.h, which is the umbrella header for the runtime and gives you everything you need. See the demo, it uses exactly that. Does the demo compile for you?
>>
>> No, the snapshot contains compile-time code (headers) as well. But,
>> nothing that defines Ref.
>
> Can you tell me under which path you see header files? I just looked in the jar and see only class files and some support files (dot mostly). No headers at all.

Sorry, I was looking at the git cloned directory structure.

antlr4/runtime/Cpp/runtime is where I see header files. There are
headers in this directory, and further sub-dirs: atn, dfa, misc,
support, and tree. None of which defines Ref.

>>>> Where does the
>>>> C++ definition for this class live? I'm pretty sure that this is my
>>>> problem, but I've given up grepping my system for it. [I did find a
>>>> defintion online in someone else's git, but I thought that if ANTLR4 is
>>>> using it, should ANTLR4 provide its definition?
>>>
>>> It's in runtime/src/antlr4-common.h. This header is included with the umbrella header. Just use that one and you should be done.
>>
>> There is no runtime/src directory in the snapshot....

I meant to say that my git-clone has no runtime/src directory that I can
find. At least not in antlr4/runtime/Cpp

>
> Of course not! The snapshot is just the ANTLR tool. It should not have anything else but the ANTLR java code + some documentation. I was referring to the C++ runtime, which is not in the jar (the various runtimes have never been part of the ANTLR jar AFAIR).
>
>>
>> I'll go back and look in the cloned-git directory....
>
> Yes, that's what you need to do. Clone the repo, place the jar into the demo folder and run the generate.sh script to generate the parser/lexer/visitor/listener files. Then compile everything using cmake. You can install cmake just like make from your yum or whatever respository Fedora has (I'm not very familiar with Linux tbh.). Then execute the steps as given for cmake. Should be very simple. If that doesn't compile then we may have a problem in the cmake file. But I've been told it would nicely compile the code and create the demo binary.

The generate script sets up a CLASSPATH, and that's about it. It looks
like it configures a build environment for running the tool in. It then
runs the tool over T.lexer.g4 and T.parser.g4, but it does not actually
compile any of the generated c++ files.

>>>> I am using your tool in a very virgin environment. I run the tool from
>>>> the command line, and I am now trying to build a small Makefile to do
>>>> the work for me. So far, none of the compilations compile to
>>>> completion.
>>>
>>> Can't you use cmake instead? There is a cmake file in the runtime/ folder.
>>
>> I'd prefer to use eclipse, but, .... I've never used cmake before. It
>> appears to be a system generator as opposed to a system builder.
>> It probably needs some configuration that I've never done before.
>
> Hmm, I haven't used Eclipse for C++ development at all, but I guess it's not so much different than other IDEs. Create a new console app, add the main.cpp file to it, maybe even the full C++ runtime if you don't want to use a library and that should be it.

I went back to the github with my web browser, and I see what you are
pointing me to, but it is not present in the cloned files on my
computers.... Did I not clone the project correctly?

I used the following to create my local directory:

$ git clone https://github.com/DanMcLaughlin/antlr4
The top level of this contains only:

antlr4-maven-plugin build CHANGES.txt contributors.txt doc
LICENSE.txt pom.xml README.md runtime runtime-testsuite tool
tool-testsuite

my runtime/Cpp/runtime does *NOT* contain a src directory

Should I remove my clone from my computer and clone it again, hoping to
get a better copy?

[Mike Lischke]

>>
>> Can you tell me under which path you see header files? I just looked in the jar and see only class files and some support files (dot mostly). No headers at all.
>
> Sorry, I was looking at the git cloned directory structure.
>
> antlr4/runtime/Cpp/runtime is where I see header files. There are
> headers in this directory, and further sub-dirs: atn, dfa, misc,
> support, and tree. None of which defines Ref.

I see, we misunderstood each other. By "snapshot" I meant the jar file (it even contains SNAPSHOT in the name, hence I assumed you would understand it the same way).

>>
>> Yes, that's what you need to do. Clone the repo, place the jar into the demo folder and run the generate.sh script to generate the parser/lexer/visitor/listener files. Then compile everything using cmake. You can install cmake just like make from your yum or whatever respository Fedora has (I'm not very familiar with Linux tbh.). Then execute the steps as given for cmake. Should be very simple. If that doesn't compile then we may have a problem in the cmake file. But I've been told it would nicely compile the code and create the demo binary.
>
> The generate script sets up a CLASSPATH, and that's about it. It looks
> like it configures a build environment for running the tool in. It then
> runs the tool over T.lexer.g4 and T.parser.g4, but it does not actually
> compile any of the generated c++ files.

Yes, that's how it is supposed to do. The generate script is just for generating the parser file from the grammar, nothing else. You still have to compile the app then. This is where the cmake file comes in. It can be executed with the parameters as given in the readme to do both: generate the parser files and build the demo.

For other platforms you would probably use the generate.cmd or generate.sh scripts in a prebuild phase in XCode or Visual Studio to let it create the parser files.

>
> I went back to the github with my web browser, and I see what you are
> pointing me to, but it is not present in the cloned files on my
> computers.... Did I not clone the project correctly?
>
> I used the following to create my local directory:
>
> $ git clone https://github.com/DanMcLaughlin/antlr4
> The top level of this contains only:
>
> antlr4-maven-plugin build CHANGES.txt contributors.txt doc
> LICENSE.txt pom.xml README.md runtime runtime-testsuite tool
> tool-testsuite
>
> my runtime/Cpp/runtime does *NOT* contain a src directory
>
> Should I remove my clone from my computer and clone it again, hoping to
> get a better copy?

I just cloned the repo with the same command and it has a runtime/Cpp/runtime/src folder. Maybe you are in a wrong branch? You should use master.

Mike
--
www.soft-gems.net

[Kevin Cummings]

On 05/23/16 03:48, 'Mike Lischke' via antlr-discussion wrote:

> I just cloned the repo with the same command and it has a runtime/Cpp/runtime/src folder. Maybe you are in a wrong branch? You should use master.

yes, I just deleted my old stuff and re-cloned (using the exact same
command I used before). It now has runtime/Cpp/runtime/src. I will try
re-compiling rsn. If I have a working tool, and a working set of
headers/libraries, I can then build executables, and I can continue to
learn antrl4 and give feedback to the development project.

Thanks again!