Julia Grammar and Native Lexer/Parser as General Tools for DSL Implementation

[Theodore Papamarkou]

The scope of this post is two-fold. Firstly, I was looking for info on what grammar has been used for defining Julia as a language and consequently how the lexer and parser for the Julia compiler have been built. Do you use recursive-descent LL(1) or LL(k) for k>1? Or did you employ backtracking or parser memoising as well?

The second question is whether the implementation of the Julia lexer and parser are native so that one can use this machinery for defining a new language (I don't wonder how to implement a general-purpose language via Julia but rather how to implement a DSL such as one for PGMs). Is there the native Julia machinery in place for such developmwnt or I would have to write my own lexer and parser from scratch for the intended DSL and DSL compiler?

[Jason Merrill]

Julia's parser is written in femtolisp, and I believe it is a recursive descent parser.

There's a package that implements a Julia parser in Julia: https://github.com/jakebolewski/JuliaParser.jl

For parsing a DSL, you might also want to look at PEGParser:
https://github.com/abeschneider/PEGParser

[Theodore Papamarkou]

Thanks a lot, I wasn't aware of femtolisp's existence. From a very quick look at the source, it is a recursive-descent parser which uses symbol table lookup (given that there is no documentation and that I haven't spent much time reading into the code, I may be wrong); I would have to spend more time on the source to find out if the symbol table is for monolithic scope, nested scope, data aggregates or for classes - it is helpful to know where to look, to start with.

Thanks for the links to the 2 Julia packages; it looks like PEGParser is good enough for generating parsers from inputted grammars, so it may do the job for me at the moment. It would be amazing if one of us writes a package for a Julia runtime library for ANTLR (Python has sth similar, see https://github.com/antlr/antlr4-python3), given that ANTLR is probably the state of the art parser generator nowadays.