Antlr4-tools

[Agalia Valcin]

Im starting gettting my hands dirty with antlr4 and I don't unserstand the difference between Getting started the easy way using antlr4-tools and Installation decribed in the Getting started guide.

All of these can be categorized as efforts produce the most generic and cross-cutting solution possible, and they surprisingly work. Though the needs are many and endless, the users compromise some of them to ultimately accept a solution that works well enough for the majority.

But sometimes the needs are so individually specific that there simply is no common ground. Where a solution might work perfectly for some, but might be unacceptable for the rest. Parameterized rule engines like fraud detectors, loyalty programs, chat bots and campaign selection engines are perfect examples for that, because not all parameter values can be globally acceptable.

But how can we provide such endless possibilities to our users? Can we afford the cost of developing and maintaining convoluted user interfaces with full support of our feature? Do we have the backend to support that UI to begin with? Assuming we did both, will we even be able to explain those UIs to our users?

So how about we create a custom scripting/programming language instead? This way not only we would get to have a language that is completely specific to our business (and as easy as to learn and use as possible), but we would also not have to worry about arbitrary code execution and the security compromises that come with it, because we would be able to fully control which data and functions would be available to the language. No need to sandbox code, or to review user submitted code for security purposes.

Creating a DSL entails coming up with a solution to parse some arbitrary input into a set of instructions that our program can understand, and executing those instructions with another set of inputs when required.

Fortunately, there are many lexers and parser generators out there (see _of_parser_generators#Deterministic_context-free_languages), and some parser generators even have built-in lexers, reducing the number of steps required to process.

As a JVM-based tool it can generate Java code, meaning that the output can be used in just about all JVM languages from Scala to Clojure, and it has built-in support for generating lexers and parsers in C++, C#, Go, JavaScript, Python, and PHP. Furthermore, there are projects like antlr_rust to extend the runtime language support even further.

IntelliJ IDEA has an excellent ANTLR plugin that can not only syntax highlight and verify ANTLR grammars, but also produce visual ASTs and complexity reports. It also works for the community edition of IDEA so I highly recommend using it.

I find this plugin especially because as soon as you open an ANTLR grammar file the ANTLR Preview panel gets activated, which has a source input pane on the left, and a list of tabs on the right for examining the token hierarchy, which get updated automatically as you change the grammar or the source input.

Though in-editor tools are extremely useful, I feel a bit constrained by them, because it means that I have to work with inside an editor and within the scope of a project at all times, but that may not always be the actual case. Sometimes I might just want to open up a terminal, feed some input to some grammar and see its results, and this is where antlr4-tools come in to rescue.

As a JVM project, ANTLR essentially has a JAR file that contains the dependencies for the Java runtime, and can also be run using java -jar antlr4.jar ... to generate runtime codes for various languages, as well as parsing grammars and outputting debug information.

We can also inject arbitrary code into the generated lexer or parser code using @header ... or @footer ... blocks, the former of which is very helpful for adding package statements to our files.

To parse the same input in the same format in a context-free grammar, however, we would need lexer rules to capture foo or bar, and a parser rule that accepts either tokens and is terminated with an EOF.

Alright, so we can figure out the operator by looking at OP token, and the left and right numbers by looking at the NUMBERS tokens, but we can do one better and label the left and right tokens which will then help us access these tokens by their name.

While in the tag parsing mode we can ignore whitespaces, parse attributes in the = form and capture tag closing characters to leave the mode, and while not in the mode we can keep all whitespaces, accept all arbitrary input as TEXT and enter the mode when we see tag opening characters.

To use modes we need to separate our grammar into its lexer and parser components, and then we can use the pushMode(MODE_NAME) and popMode channels (e.g. TOKEN: '...' -> pushMode(MODE_NAME);) in our lexer rules to enter and exit said modes. To specify which tokens should be captured inside a mode we can putting all relevant tokens below the line mode MODE_NAME;, and everything on top will only be captured when not inside the mode.

3a8082e126