Kotlin target to target JVM & JS at once

[Federico Tomassetti]

Hi everyone,
I am a big, big fan of ANTLR and a big, big fan of Kotlin and I would like my interests to meet.
Yes, I can today generate grammars for Java & JS and use those from Kotlin, however I would be interested in a Kotlin target because:

1) I could get more idiomatic Kotlin code
2) I could generate multi-platform code

The second point is important: recently Kotlin has added the possibility to create projects that compile for different platforms. I would target the JVM & JS right now, but Kotlin supports also native executables (on windows, linux, iOS) and WebAssembly, which I could be interested in targeting later.

Now, I could reuse the JS and Java runtimes, I would just need to define a common interface in Kotlin (it is not technically an interface: in Kotlin you need to declare expectations from the different platforms with another mechanism).

So I would just need to generate Kotlin parser & lexer based on the "common interface runtime", which basically would wrap the runtime on the different platforms.

Can someone help me getting started?

[Mike Lischke]

Hi Federico,

The second point is important: recently Kotlin has added the possibility to create projects that compile for different platforms. I would target the JVM & JS right now, but Kotlin supports also native executables (on windows, linux, iOS) and WebAssembly, which I could be interested in targeting later.

Now, I could reuse the JS and Java runtimes, I would just need to define a common interface in Kotlin (it is not technically an interface: in Kotlin you need to declare expectations from the different platforms with another mechanism).

So I would just need to generate Kotlin parser & lexer based on the "common interface runtime", which basically would wrap the runtime on the different platforms.

Can someone help me getting started?

Nobody has done a similar project like that before, from what I know. How's that organized in Kotlin? Do you use the existing runtimes and just add kinda "meta-runtime"? The generated code would be Kotlin code, right? So this is probably the best way to start from. First set up everything so that you can actually write the language template, which is used to generate the code (pick one that is closest to Kotlin, probably Java). A few more details about this setup and the next steps are written here: https://github.com/antlr/antlr4/blob/master/doc/creating-a-language-target.md. Unfortunately, this documents does go very deep. I can help you with individual questions (if I remember the details :-) ). I can also give you a script to regenerate your parser from the ANTLR4 classpath instead of the ANTLR4 jar (which is important to avoid frequent jar creation, which is necessary as the stg file is part of that jar).

Mike
-- 
www.soft-gems.net

[Federico Tomassetti]

Hi Mike, hi everyone,
I started experiment and I got some success. I think I can get this
thing done but I have now to decide which approach to take.

What I did was to create two project:
* a new target: https://github.com/ftomassetti/antlr-kotlin-target
* a new runtime: https://github.com/ftomassetti/antlr-kotlin-runtime

The target is rather easy. I am a bit confused by StringTemplate but
besides some issues with unicode and the way ATN is encoded this was
rather simple.

Now the runtime.

The way is written is by defining two modules:
- a common module which use a subset of the Kotlin standard library
that is available on all platforms (JVM & JS) and which defines some
"expectations"
- a JS module
- a JVM module

For example, the common module would contain things like:

expect abstract class Parser {
constructor(grammarFileName: String, input: TokenStream)
fun setCtxStop(token: Token)
fun getInterpreter() : ATNSimulator
}

expect class ParserATNSimulator : ATNSimulator

The JVM module would contain things like:

actual abstract class Parser : org.antlr.v4.runtime.Parser {
private val _grammarFileName: String
actual constructor(grammarFileName: String, input: TokenStream) :
super(input.wrapped) {
this._grammarFileName = grammarFileName
}

...
}

So far I got the lexer working on JVM and JS and then I spent some
time getting the parser to compile for the JVM. Now, what is the
problem? The problem is that I am trying to build a sort of wrapper
around the existing Java and JS runtime libraries and this is sort of
working because I need to extend some classes while wrapping others.
It is getting messy. The advantage would be that as the Java or JS
runtime get improved the Kotlin would also get improved, however given
I am adding some indirection the performance could be affected and I
do not like the code I am getting because by doing something more
Kotlin-esque I could get much nicer code. So I could just rewrite the
Java runtime in Kotlin.... not sure which way to go.

Anyway it is something definite doable and interesting and it would
permit to write one compiler and get it running both on the JVM and
JS.... not bad...

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "antlr-discussion" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/antlr-discussion/y1-FgPb5Q6s/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> antlr-discussi...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Software Architect & Founder at Strumenta (https://strumenta.com)
Technical blog at https://tomassetti.me
GitHub https://github.com/ftomassetti
Twitter @ftomasse
linkedin.com/in/federicotomassetti

[Mike Lischke]

Hey Federico,

> I started experiment and I got some success. I think I can get this
> thing done but I have now to decide which approach to take.

Sounds good! One more target for ANTLR4.

>
> What I did was to create two project:
> * a new target: https://github.com/ftomassetti/antlr-kotlin-target
> * a new runtime: https://github.com/ftomassetti/antlr-kotlin-runtime
>
> The target is rather easy. I am a bit confused by StringTemplate but
> besides some issues with unicode and the way ATN is encoded this was
> rather simple.

Well, you are essentially writing for a Java derivative so, it *should* be simple. It was much more of a challenge for C++.

>
> So far I got the lexer working on JVM and JS and then I spent some
> time getting the parser to compile for the JVM. Now, what is the
> problem? The problem is that I am trying to build a sort of wrapper
> around the existing Java and JS runtime libraries and this is sort of
> working because I need to extend some classes while wrapping others.
> It is getting messy. The advantage would be that as the Java or JS
> runtime get improved the Kotlin would also get improved, however given
> I am adding some indirection the performance could be affected and I
> do not like the code I am getting because by doing something more
> Kotlin-esque I could get much nicer code. So I could just rewrite the
> Java runtime in Kotlin.... not sure which way to go.

IMO you should write a full Kotlin runtime. Your users may later make more out of that (e.g. let it use other runtimes), but from a clarity and compatibility standpoint a dedicated Kotlin runtime is probably the best way. You can then integrate that like all the other targets (think of executing the full test suite, but also the release process, documentation etc.).

My 2 cents,

Mike
--
www.soft-gems.net

[Eric Vergnaud]

Hey Mike,

there is no Kotlin interpreter for Javascript, rather the Kotlin toolset lets you translate Kotlin code toJavascript, pretty much the same approach as GWT some time ago.

From there, I'm not convinced that creating a pure Kotlin runtime is necessarily the best approach, because the Kotlin tools would have translate the entire runtime to Javascript...

I haven't invested any time exploring how Kotlin libraries are made portable, so not sure what is feasible. But I think Federico's approach using a wrapper makes sense.

@Federico, my recommendation would be to pursue your effort with the wrapper. Once you get something running, we can look at reducing the indirection and code mess.

I believe there is value in a 'core' Java runtime that can be reused across Java, Kotlin, Scala etc... and a core 'JavaScript" one that can be reused across pre-ES6 (current), ES6, Typescript, JSX and Kotlin.

I can't spend time on this, but happy to support your efforts.

[Mike Lischke]

> there is no Kotlin interpreter for Javascript, rather the Kotlin toolset lets you translate Kotlin code toJavascript, pretty much the same approach as GWT some time ago.
> From there, I'm not convinced that creating a pure Kotlin runtime is necessarily the best approach, because the Kotlin tools would have translate the entire runtime to Javascript...

Hmm, if you translate Kotlin code to something else anyway (be it Java or JS) why having Kotlin support in the first place? All you would do is write your grammar, let ANTLR4 generate a Kotlin parser and then translate that to Java. Why not directly go with Java then?

>
> @Federico, my recommendation would be to pursue your effort with the wrapper. Once you get something running, we can look at reducing the indirection and code mess.
> I believe there is value in a 'core' Java runtime that can be reused across Java, Kotlin, Scala etc... and a core 'JavaScript" one that can be reused across pre-ES6 (current), ES6, Typescript, JSX and Kotlin.

There is already a typescript target (antlr4ts). I'm not sure if an automated translation can get close to that performance-wise and have the same readability.

Mike
--
www.soft-gems.net

[Federico Tomassetti]

Let me just be more specific about why I think the Kotlin target would
be useful:
1) Kotlin is an amazing language: it makes more very productive and I
think that having a proper Kotlin parser, written in idiomatic Kotlin
would be nice
2) Kotlin has now support for multi-platform project therefore having
a Kotlin target would allow having a multi-platform compiler or other
language tools that run on multiple platforms

Let's dive in the multi-platform thing.

Kotlin is a language that can be compiled:
* to JVM bytecode
* to JavaScript to be run in the browser
* to native code through LLVM (so it can run on iOS, it could be
compiled to WebAssembly, etc.)

Now, the problem is that you typically write a project in Kotlin
looking at a specific platform (e.g., JVM) and therefore you use all
the specific platform capabilities. For example, if I write a project
in Kotlin to compile it to JVM bytecode I can access the whole Java
standard library. The problem is that I cannot compile the same code
to JavaScript.

With multi-platform support this changes because you can write Kotlin
and use the multi-platform Kotlin standard library and get the same
Kotlin code to compile to different platforms. So if I get a Kotlin
parser and lexer, using just the multi-platform Kotlin standard
library I can compile that code and run it as-is on the JVM and the
browser. I can then write a whole compiler or an interpreter that will
reuse that parser and could also run on the JVM or the browser. I
think this is cool.

Currently I can write one ANTLR grammar and use different target, like
the Java and the JS target but then I get two different parsers, one
in Java and one in JS so if I want to write a compiler that run both
on Java and JS I have to write it twice.

# How the multi-platform Kotlin projects work

Now I am working on having a complete Kotlin runtime and it is already
almost done. How I did that?

1) I took the Java runtime
2) I converted it to Kotlin: there is a utility for doing that sort of
automatically but it then requires some adjustments
3) I stripped out every element that was supported only on the JVM but
not on the multi-platform Kotlin standard library
4) For those things that I needed and that were not available in the
multi-platform Kotlin standard library I defined the "expectations"
and then the "actual implementations" for both the JVM and JS. Think
of a mechanism like interface/implementation: you define the interface
in general and the implementation for the two platforms. We are
talking about a few classes, compared to the hundreds of classes of
the runtime, so the platform specific effort is pretty low

I am currently at a good point on the JVM, I still need work on the JS side.

Let's see if I find the time to get this thing finished and then write
some documentation for it.

Cheers,
Federico

[Federico Tomassetti]

Hi everyone,
basic examples now work on both the JS backend and the JVM backend.
I reorganized the code in one single project containing several
modules for the target and the runtime.
The code is available here:
https://github.com/ftomassetti/antlr-kotlin

More tests and documentation are coming.

Among other things the multi-platform support would permit to write
once library like ANTLR-C3 (https://github.com/mike-lischke/antlr4-c3)
and use it both on the JS and the JVM platforms. This is what I plan
to do by updating my port to Kotlin of that library
(https://github.com/ftomassetti/antlr4-c3-kotlin).
In the future multiple libraries could be written in Kotlin and
support automatically all platforms. It is interesting to point out
that currently multi-platform Kotlin projects can support the JVM and
JS but Kotlin itself can run on more platforms (like iOS) so in the
future multi-platform projects could be used in more and more
platforms. Which could be very useful.

Thank you for all the support, I will update this thread when I have a
more polished version to share!

Cheers,
Federico

[Mike Lischke]

Hey Federico,

I reorganized the code in one single project containing several

modules for the target and the runtime.
The code is available here:
https://github.com/ftomassetti/antlr-kotlin

I wonder why you didn't use a clone of the ANTLR4 repo and its directory layout. You will have to structure your code that way later anyway, so why not do it right from the start? I strongly recommend to write and run also the runtime tests for your runtime(s).

Regarding your runtimes: what you have in the github repo is just for demonstration, right? Atm. I only see some support classes, nothing of the main runtime classes.

Mike
-- 
www.soft-gems.net

[Federico Tomassetti]

Hi Mike,
Thank you for taking a look and providing feedback. This is the first
time I am writing an ANTLR target and I could use all the help I can
get.

On Sat, Jan 6, 2018 at 5:41 PM, 'Mike Lischke' via antlr-discussion
<antlr-di...@googlegroups.com> wrote:
> Hey Federico,
>
> I reorganized the code in one single project containing several
> modules for the target and the runtime.
> The code is available here:
> https://github.com/ftomassetti/antlr-kotlin
>
>
> I wonder why you didn't use a clone of the ANTLR4 repo and its directory
> layout.

I am not sure what you mean here. The project has a module named
antlr-kotlin-target which contains a directoy src/main/java with the
code. The code is just a class named KotlinTarget in the package
org.antlr.v4.codegen.target. There is also the template in the
resources directory named Kotlin.stg, in package
org.antlr.v4.tool.templates.codegen.Kotlin. Should the layout be
different?

> You will have to structure your code that way later anyway, so why
> not do it right from the start?

This is true only when and if this target is merged in the ANTLR main
project. That is out of my control. What I can control is to get a
project that is useful from now as a plugin of the ANTLR Tool.

> I strongly recommend to write and run also
> the runtime tests for your runtime(s).

Absolutely. I have already (very few) tests which are contained in the
antlr-kotlin-runtime-examples-js module and in the
antlr-kotlin-runtime-examples-jvm module. I should rename those
modules.

Thoses tests already run on Travis CI.

I agree I would need to get far more tests, and I will work on that as
soon as I can.

> Regarding your runtimes: what you have in the github repo is just for
> demonstration, right? Atm. I only see some support classes, nothing of the
> main runtime classes.

Well, no, I have the whole Kotlin runtime, divided in three modules:
- antlr-kotlin-runtime-common which contains the code that works as is
on both JVM and JS
- antlr-kotlin-runtime-js with the very small JS specific code
- antlr-kotlin-runtime-jvm with the very small JVM specific code

99% of the code is in antlr-kotlin-runtime-common. It contains two
main packages:
* org.antlr.v4.kotlinruntime: which is basically equivalent to the
Java runtime in org.antlr.v4.runtime
* com.strumenta.kotlinmultiplatform: which contains the code that is
needed to be implemented specifically on the two platforms. In the
antlr-kotlin-runtime-common module we get the declarations of the
"expectations" we have from the implementation for the two platforms

Cheers,
Federico

[Mike Lischke]

I wonder why you didn't use a clone of the ANTLR4 repo and its directory
layout.

I am not sure what you mean here. The project has a module named
antlr-kotlin-target which contains a directoy src/main/java with the
code. The code is just a class named KotlinTarget in the package
org.antlr.v4.codegen.target. There is also the template in the
resources directory named Kotlin.stg, in package
org.antlr.v4.tool.templates.codegen.Kotlin. Should the layout be
different?

I assumed you would want your target be integrated in ANTLR4. Will that not be the case? If you want that however, you will have to use the same folder layout as the ANTLR4 repo uses. Starting with a fork then would make sense.

I strongly recommend to write and run also
the runtime tests for your runtime(s).

Absolutely. I have already (very few) tests which are contained in the
antlr-kotlin-runtime-examples-js module and in the
antlr-kotlin-runtime-examples-jvm module. I should rename those
modules.

Thoses tests already run on Travis CI.

That's not what I mean. You should take a closer look at https://github.com/antlr/antlr4/tree/master/runtime-testsuite (the runtime test suite that runs tests for all targets). The tests run automatically on appveyor and travis and you can run them locally (e.g. via `mvn test`), but you have to integrate your runtime via another template file (see https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates) and some glue code (see https://github.com/antlr/antlr4/tree/master/runtime-testsuite/test/org/antlr/v4/test/runtime).

Regarding your runtimes: what you have in the github repo is just for
demonstration, right? Atm. I only see some support classes, nothing of the
main runtime classes.

Well, no, I have the whole Kotlin runtime, divided in three modules:
- antlr-kotlin-runtime-common which contains the code that works as is
on both JVM and JS
- antlr-kotlin-runtime-js with the very small JS specific code
- antlr-kotlin-runtime-jvm with the very small JVM specific code

99% of the code is in antlr-kotlin-runtime-common.

Ah, I missed that (probably because you used your own folder layout and I was expecting the ANTLR4 layout). How did you convert the Java code? Using a tool or just being a typing robot :-) (probably both, which is a common approach among the target authors)?

Anyway, looking at that code shows me you are pretty far with that already, great.

Mike
-- 
www.soft-gems.net

[Federico Tomassetti]

Hi,

On Sat, Jan 6, 2018 at 7:33 PM, 'Mike Lischke' via antlr-discussion
<antlr-di...@googlegroups.com> wrote:
> I wonder why you didn't use a clone of the ANTLR4 repo and its directory
> layout.
>
>
> I am not sure what you mean here. The project has a module named
> antlr-kotlin-target which contains a directoy src/main/java with the
> code. The code is just a class named KotlinTarget in the package
> org.antlr.v4.codegen.target. There is also the template in the
> resources directory named Kotlin.stg, in package
> org.antlr.v4.tool.templates.codegen.Kotlin. Should the layout be
> different?
>
>
> I assumed you would want your target be integrated in ANTLR4. Will that not
> be the case? If you want that however, you will have to use the same folder
> layout as the ANTLR4 repo uses. Starting with a fork then would make sense.

I would like the target to be integrated in ANTLR4 but I would expect
this to happen in several weeks or months. In the meantime I want to
experiment with the Kotlin target, possibly using it already in some
projects. In this case I think it is easier/cleaner to use it if it is
a plugin to the latest release of ANTLR4 instead of a fork. When I get
the green light to propose the integration it will suffice to do the
ANTLR4 fork and copy the couple of files I need for the target. I
would expect the Kotlin runtime (which represents 99.9% of the code)
to remain a separate project.

> I strongly recommend to write and run also
> the runtime tests for your runtime(s).
>
>
> Absolutely. I have already (very few) tests which are contained in the
> antlr-kotlin-runtime-examples-js module and in the
> antlr-kotlin-runtime-examples-jvm module. I should rename those
> modules.
>
> Thoses tests already run on Travis CI.
>
>
> That's not what I mean. You should take a closer look at
> https://github.com/antlr/antlr4/tree/master/runtime-testsuite (the runtime
> test suite that runs tests for all targets). The tests run automatically on
> appveyor and travis and you can run them locally (e.g. via `mvn test`), but
> you have to integrate your runtime via another template file (see
> https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates)
> and some glue code (see
> https://github.com/antlr/antlr4/tree/master/runtime-testsuite/test/org/antlr/v4/test/runtime).

I will look into that, thanks.

>
> Regarding your runtimes: what you have in the github repo is just for
> demonstration, right? Atm. I only see some support classes, nothing of the
> main runtime classes.
>
>
> Well, no, I have the whole Kotlin runtime, divided in three modules:
> - antlr-kotlin-runtime-common which contains the code that works as is
> on both JVM and JS
> - antlr-kotlin-runtime-js with the very small JS specific code
> - antlr-kotlin-runtime-jvm with the very small JVM specific code
>
>
> 99% of the code is in antlr-kotlin-runtime-common.
>
>
> Ah, I missed that (probably because you used your own folder layout and I
> was expecting the ANTLR4 layout).

Yes, I use standard Gradle/Maven layout

> How did you convert the Java code? Using a
> tool or just being a typing robot :-) (probably both, which is a common
> approach among the target authors)?

IntelliJ IDEA is amazing: you copy Java files into a Kotlin project
and they are converted automatically. Now, that sort of work but there
are things to fix of course. Also, it convert it to Kotlin code that
use the Java standard library, while I am writing a multi-platform
project and the multi-platform library is smaller than the Java
standard library. For this reason I needed to do a lot of adjustments.

> Anyway, looking at that code shows me you are pretty far with that already,
> great.

Yes, I have generated a lexer and a parser for a very simple language
and wrote some tests for the lexer and the parser, trying those both
on the JVM and the JS and they pass!
Now I will look into generating lexers and parsers for more complex languages.

While I can test the runtime by using the generated lexers and parsers
I am not sure on how to test the target. What approach do you suggest?