How to implement a new language?

[CSchulz]

Hello everyone,

I have started reading the checkstyle sources to create an implementation supporting a new language and struggled with the fact that you are using Antlr 2.

Is there any reason to use the "old" version with AST? Is the Parse Tree of Antlr 4 not good enough?

It is really hard to find some grammars for Antlr 2. I have only found some for v4.

I think the only way is to use a v4 grammar to prevent writing a new grammar. In this case I need to write a Parse Tree to AST converter.

Furthermore I need to consider how to handle the original "format" of the file, spaces, tabs and so on, which are filtered by the parser.

Does anybody have some advices how to start with such a converter?

Shall I use the ParseTreeListener or Visitor to do it?

Thanks in advance.

[Roman Ivanov]

Hi CSchulz,

> I have started reading the checkstyle sources to create an implementation supporting a new language

Please beware that you can hack on Checkcstyle as you  want to resolve your tasks, but Checkstyle will stay Java specific,. We even can not satisfy all java needs for static analysis, so we do not plan to add support for other languages.

But you can fork and create your own project.

> Is there any reason to use the "old" version with AST?

yes, the only reason is complexity of Java grammar is huge. We do not have abilities to update to ANTLR4 as I will require for one person to completely focus on that for few month - nobody can commit for that.

But we have a plan to do that - here is summary of my thoughts - https://github.com/checkstyle/checkstyle/wiki/Checkstyle-GSoC-2015-Project-Ideas#project-name-upgrade-java-grammar-from-antlr2-to-antlr4

> It is really hard to find some grammars for Antlr 2. I have only found some for v4.

forget about antlr2 , it is our legacy part that we eventually resolve.

We already use ANTLR4 for JavaDoc  parsing, please look at https://github.com/checkstyle/checkstyle/tree/master/src/main/resources/com/puppycrawl/tools/checkstyle/grammars/javadoc

> Does anybody have some advices how to start with such a converter?

> Shall I use the ParseTreeListener or Visitor to do it?

I can not give you advice without knowing all your targets, but you can review our code and see examples.

thanks,

Roman Ivanov

​

[CSchulz]

Hello Roman,

thanks for your fast reply.

> Please beware that you can hack on Checkcstyle as you  want to resolve your tasks, but Checkstyle will stay Java specific,

> We even can not satisfy all java needs for static analysis, so we do not plan to add support for other languages.

I am aware of it, but I think there is a lot of stuff which I can reuse in some way.

It could be perhaps a more hacky than a clean way.

It looks like the migration to Antlr 4 has advantages for you, thanks for sharing the information.

I was asking for Antlr 2, because checkstyle is using some classes from it and also uses an AST.

The AST support is removed in Antlr 4 (https://theantlrguy.atlassian.net/wiki/pages/viewpage.action?pageId=1900596).

This makes it necessary for me to create a converter from ParseTree to AST using your DetailAST class

to use your abstract Check class otherwise I need to rewrite a lot of stuff.

My first goal is creating a small prototype based on checkstyle to analyse XML files.

Doing simple checks like formatting and so on.

Best Regards

[Roman Ivanov]

Hi CSchulz,

> My first goal is creating a small prototype based on checkstyle to analyse XML files.

I already have a requests for XML files validation - https://groups.google.com/forum/#!topic/checkstyle/WG6KBm9i8t4. 

I recommend you to take a look at how we integrated javadoc with ANTLR4 grammar, at least in grammar you can find useful parts as javadoc is partly HTML , so we have HTML grammar there, XML grammar should be similar.

Your checks should be a child of Checker, please do not use TreeWalker as it for Java only. 

As we succeeded at stabilization of java support, I will allow contribution (new Checks) for *.properties and *.xml files to Checkstyle , as it is kind of part of java project and we already have some Checks for that. But for now no ETA.

thanks,

Roman Ivanov

​

[CSchulz]

Hello Roman,

thanks for the links, but it seems that his approach is dedicated to single files as single check.

I want a more flexible way to implement X tests.

My idea would be to have a XMLWalker extends AbstractFileCheck, which takes all files with xml extension parse them and "walk" them.

I had a look into the javadoc implementation, but it seems that it builds it own system.

To tell you, what I have done so far:

- Created my own XMLWalker extends AbstractFileCheck, which creates the DetailNode tree and walk it

- XMLTokenTypes which maps all stuff from Parser

- AbstractXMLCheck which defines some generic methods like beginXMLTree, visitXMLToken, etc.

So it is similar to the javadoc implementation and at some points c/p.

What do you think?

[Roman Ivanov]

Hi Schulz,

direction is right.

thanks,

Roman Ivanov

​