Re: Reflection for specified classes

[Alexey Andreev]

Hello,

Do you plan reflection support?

No

Not for all classes, but only for those, that specified in some way.

For example in pom.xml as parameter:

* Using some expression like: "com.classes.with.reflection.support.**"

* Or include reflection support for classes, fields, methods etc, that has specified annotation ("@Service, @Component..."), or implement interface etc.

* Or by writing my own logic (plugin which will implement some interface), which will decide, should class, field, method include reflection support or not: 

shouldIncludeReflectionSupport(Field field, Class owner);// for example if owner class has @Service annotation or is in package "com.classes.with.reflection.support" then return true and field will have reflection support for that class (getDeclaredField, or getDeclaredFields). Maybe my logic will decide to include some fields and some not for reflection support, in that case getDeclaredFields should not return those fields.

shouldIncludeReflectionSupport(Method method, Class owner);

shouldIncludeReflectionSupport(Class cls); // include whole reflection support for class (fields, methods)

Classes, that has reflection support should be iterable TeaVM.getClassesWithReflectionSupport() (for example to scan packeges and perform dependency injection like spring does).

There is no reflection due to AOT nature of TeaVM. In order to produce small JS, TeaVM tries to guess what methods will be actually used and compiles only them. It allows TeaVM to produce small JS, if you include huge library and use only few methods of it. An example of VM that does not follow this approach is bck2brwsr, which produces several megabytes of JS for a simple hello world app.

Instead of reflection you can write your own extensions for TeaVM, which allows some metaprogramming. Extension API, however, is not documented anywhere. I can give you some code to learn.

Reflection would be very useful to convert JSON to entity or vice versa (before/after send/receive entity to/from endpoints etc), validating, dependency injection to do some powerful framework like angular etc.

I do all this without reflection at all. Take a look here: https://github.com/konsoletyper/teavm-flavour There is templating support like in Angular, but unlike Angular, Flavour compiles templates into JavaScript, checking whether properties and methods referred from templates really exist in Java model. As for JSON, there is upcoming support.

[Carlos Ballesteros Velasco]

Hello, I have just discovered TeaVM. And it sure looks amazing! :) congratulations.

I like it doing AOT with DCE so we are not including the whole runtime just for a hello world or stuff we are just not using.

Our game company is considering using it to target html.

Currently we are using kotlin and libgdx, but we have the libgdx usage isolated so we can use other stuff, so that is not a problem, we are using it just for drawing textured triangles.

Our concerns are about reflection:

We are using reflection for two things: conversing jsons from and to typed value objects. Including generic type lists and so.

Also we are using reflection to do dependency injection.

We are using Kotlin and we are performing pretty simple constructor-based dependency injection. So we are using types in the main constructor to get singletons and prototypes. And those constructors are not called directly, so a normal DCE (without something liek a @Keep annotation or a rule to keep things) would remove it. And without reflection we couldn't instantiate it either.

So, I have read that you are not planing adding reflection to the already DCEd stuff. But also I have read that you are solving dependency injection on teavm-flavour. We have our own dependency injector, but we could wrap other to our own, in the case it supports constructor-based DI, that is what we need for kotlin. Also we need to solve JSON<-> typed VO (with generic-lists) in order to be able to use TeaVM.

Can you give me some information about how to do this and a breef about how it works?

Thank you in advance! And kudos about your awesome work! :)

[Alexey Andreev]

Our concerns are about reflection:

We are using reflection for two things: conversing jsons from and to typed value objects. Including generic type lists and so.

Also we are using reflection to do dependency injection.

We are using Kotlin and we are performing pretty simple constructor-based dependency injection. So we are using types in the main constructor to get singletons and prototypes. And those constructors are not called directly, so a normal DCE (without something liek a @Keep annotation or a rule to keep things) would remove it. And without reflection we couldn't instantiate it either.

So, I have read that you are not planing adding reflection to the already DCEd stuff. But also I have read that you are solving dependency injection on teavm-flavour. We have our own dependency injector, but we could wrap other to our own, in the case it supports constructor-based DI, that is what we need for kotlin. Also we need to solve JSON<-> typed VO (with generic-lists) in order to be able to use TeaVM.

TeaVM flavour does not have any DI yet. There is JSON <-> POJOs mapper that emulates subset of Jackson. Please, see serializer and deserializer tests if you want to know what is supported. JSON support is based on metaprogramming: Flavour sets hooks on TeaVM and generates some additional classes (in terms of TeaVM IR) that perform actual serialization/deserialization. There is DSL that helps to generate IR, see JsonSerializerEmitter for example.

Unfortunately, this does not work with latest published version (0.4.0) of TeaVM. I'm very close to release Flavour, so I'll release bugfixes for TeaVM (0.4.1) as well. If you are insterested, you can build Flavour with example application:

1. get corresponding TeaVM branch: git clone https://github.com/konsoletyper/teavm.git
2. build it: mvn clean install -DskipTests
3. get master branch of Flavour: git clone https://github.com/konsoletyper/teavm-flavour.git
4. build it: mvn clean install
5. Go to example/target subdirectory and get war file there.
6. Deploy this war in servlet container and open in the browser.
7. Add #products hash to address ending (so you get something like http://localhost:8080/teavm-flavour-example/#products ) and you should see product list which you can manipulate: add and edit products.

There are also JAX-RS subset client and templating, both are based on metaprogramming as well. Right now I am refactoring metaprogramming support to avoid lots of manual work.

[Carlos Ballesteros Velasco]

Ok. I will check this info tomorrow. We will need DI to be able to use this.

What do you mean with metaprogramming? I know about two kind of metaprogramming: being able to get an AST of program pieces at runtime, as F# or C# does, or generating code at compile-time as D, HaXe and scala do. I like the first one, but dislikes the second one because of the complexity, breaking incremental compilation and possibility generate code that is not going to be supported on IDEs. Not sure if you are refering to other kind of metaprogramming.

At any rate: are you accepting PRs? In the case we found out that something doesn't work for us, we will be able to collaborate with the project?

Are you completely agains with adding any kind of reflection even if that is just for the code after the DCE and optionally?

I have experience with all this stuff and I could help. I started a year ago (I didn't know about your project) a similar project: https://github.com/soywiz/java-aot and a few weeks ago I started https://github.com/jtransc/ though right now it is private. That aims to do an AOT. I was using ASM + SOOT to generate an intermediate feature-based AST, and with that simple AST I was targetting js and as3 and adding targets was pretty easy. We were doing also DCE and my as3 target generated an AS3 swf file of 5KB, and a kotlin hello world of 10KB, but also we were supporting reflection and there are ways in order to reduce the size footprint (like generating something similar to webassembly making it more compact but easy to generate code on the fly), also making it optional would allow people to use reflection when required, and strip it to reduce the size when not required. Even when jtransc was done using kotlin and we were pretty productive doing it, it is still a bit immature in relation to your project.

Also right now we are most interested on generating javascript so your project could fit our needs.

So I have experience with this stuff and could help if required.

I will look tomorrow at it and we will evaluate wheter it fits our needs or not and which parts are missing.

So thank you for the information and for all your awesome work!

[Alexey Andreev]

> What do you mean with metaprogramming?

Metaprogramming in general is writing a code that produces a code.

> I know about two kind of metaprogramming: being able to get an AST of
> program pieces at runtime, as F# or C# does, or generating code at
> compile-time as D, HaXe and scala do.

As I explained, it is possible to hook compiler internals and add some
behaviour there. For example, you can alter a method or generate a
class. Everything is performed in terms of TeaVM IR, not source code or
AST.

> I like the first one, but dislikes the second one because of the
> complexity,

Well, what is complexity? Reflection is complex for a person who never
used it before. You can get used to generating code and everything be
all right.
> breaking incremental compilation
Incremental compilation is broken in TeaVM. I'll fix this problem
eventually, but I'll never achieve fair incremental compilation. This
means that incrementally built JavaScript won't be equivalent to
JavaScript rebult from scratch.

> and possibility generate code that is not going to be supported on IDEs.

This code won't be even visible in IDE, since everything is made during
compilation. This is a problem, as well as debugging. I overcome this
with generating as few code as possible. My approach is: write all
complex logic in helper classes and then generate glue code between
them. This makes debugging unnecessary on most cases, and when I still
have to debug, I debug generated JavaScript.

> At any rate: are you accepting PRs? In the case we found out that
> something doesn't work for us, we will be able to collaborate with the
> project?

Yes, I do.

> Are you completely agains with adding any kind of reflection even if
> that is just for the code after the DCE and optionally?

What is DCE?

I can approve reflection, if it is disabled by default and restricted by
developer, for example, reflect methods of classes of a given package list.

[soywiz]

Understood.

I have used several kind of metaprogramming over the years and some are hurting languages more than others. My experience is the following:

I used metaprogramming in D to create a DRY (don't repeat yourself) PSP emulator. D allowed you to generate code at compile time. So I used it for several stuff:

* Create high efficient generic functions. You could generate specialized functions at compile time, like templating.

* Also you could create completely different functions and classes, so I transformed at compile-time a list of MIPS processor instruction definitions (which bits were set with which values) into a set of anidated switches that decoded those instructions and generated disassemblers, assemblers and an interpreter.

* Also I was using metaprogramming to generate functions that readed processor registers and called other functions with those registers that were just getting plain parameters.

So it was pretty cool. But after implementing lot of stuff, the compilation process was getting heavier and I ended having to wait several minutes for each time I wanted to test a very small change.

Also the core of the language was using this kind of metaprogramming. That prevented good IDEs to appear because of the complexity. Also, generating/modifying types at compilation time prevents good refactorings or autocompletion. You can for example add fields based on the constructor as kotlin does with metaprogramming with haxe, but that will prevent good tooling to appear.

After that I moved my emulator to C# and later to javascript. C# allowed some kind of good stuff without affecting the tooling. You can create functions that get instead of a result, a tree (AST) with objects describing that. For example CALL(1 + 2), you get: BinOp(Literal(1), "+", Literal(2)) so you can generate other code at runtime. For example a WHERE clause in a query. Or program in C# but end generating opengl shading language or other stuff.

That is consider metaprogramming as far as I know.

Also I was using metaprogramming to create functions on the fly at runtime. That allowed me to create high performant code without breaking the rest of the code and the tooling.

So basically that's why I like some kind of metaprogramming (obtaining information about the syntax receiving AST) and generating code at runtime dynamically, but dislikes generating code at compile time because of the problems I mentioned: breaks incremental compilation and breaks IDE refactoring (when it is bundled with the language). Thats why I like more kotlin than scala.

So with that on mind, I think that there are ways to do metaprogramming at the compile time right without abusing. You can generating factories having type information at compile time, and then somehow calling that code.

That metaprogramming is kinda a non-standard reflection. That could lead to less code (or not) depending on how it is implemented.

 

> At any rate: are you accepting PRs? In the case we found out that
> something doesn't work for us, we will be able to collaborate with the
> project?
Yes, I do.
> Are you completely agains with adding any kind of reflection even if
> that is just for the code after the DCE and optionally?
What is DCE?

Dead Code Elimination. Probably what you are doing already to stage just the required code.

 

I can approve reflection, if it is disabled by default and restricted by
developer, for example, reflect methods of classes of a given package list.

That's good to know in the case we need requiring it.

I'm going to investigate the metaprograming way and will provide feedback about wether it fits our needs or not.

Also I have a unrelated question regarding to performance:

(We need to really push the limits so we need to be sure that we don't find a bottleneck just after using a technology)

Are you using javascript typed arrays for byte[] short[] and so on?

So you are generating javascript from bytecode. How is it done? a while + switch to simulate gotos? Or are you doing some kind of relooping: converting jumps into ifs and whiles as emscripten do. Does it work with arbitrary jumps to code that could produce tooling manipulating bytecode? Or just with valid plain Java/Kotlin/Scala generated code with logic jumps that fits ifs and whiles seamlessly?

 

--
You received this message because you are subscribed to a topic in the Google Groups "TeaVM" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/teavm/bIAiqTkrT_Q/unsubscribe.
To unsubscribe from this group and all its topics, send an email to teavm+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/teavm/5640436D.3020905%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Alexey Andreev]

So with that on mind, I think that there are ways to do metaprogramming at the compile time right without abusing. You can generating factories having type information at compile time, and then somehow calling that code.

That metaprogramming is kinda a non-standard reflection. That could lead to less code (or not) depending on how it is implemented.

What TeaVM Flavour generates is only visible to compiler internals. You won't see any code in IDE.

 

What is DCE?

Dead Code Elimination. Probably what you are doing already to stage just the required code.

Oh, I see. DCE is usually referred in another context, when you talking about optimizing inside procedures. TeaVM does not eliminate dead code, nor does it eliminate anything. Instead, it uses reverse approach: it "inflates" your program. Starting from main method it looks for another methods that are missing, until it realizes that every required methods are marked for compilation.

 Also I have a unrelated question regarding to performance:

(We need to really push the limits so we need to be sure that we don't find a bottleneck just after using a technology)

Are you using javascript typed arrays for byte[] short[] and so on?

Yes, I do. See https://github.com/konsoletyper/teavm/blob/master/core/src/main/resources/org/teavm/javascript/runtime.js#L70

So you are generating javascript from bytecode. How is it done? a while + switch to simulate gotos? Or are you doing some kind of relooping: converting jumps into ifs and whiles as emscripten do. Does it work with arbitrary jumps to code that could produce tooling manipulating bytecode? Or just with valid plain Java/Kotlin/Scala generated code with logic jumps that fits ifs and whiles seamlessly?

Neither of above. Please, take a look at some of live examples in teavm.org and look through classes.js files. No, TeaVM could not be fast if it used naive approach like while+switch. It does not do relooping as well, since relooping is kind of optimization of former approach. Instead, TeaVM reconstructs structural control flow. The algorithm is quite generic and I believe it can handle any reducible CFG, so yes, tooling is accepted, unless it produces irreducible CFG. It is even possible to extend TeaVM to handle irreducible flow. See description here: https://github.com/konsoletyper/teavm/wiki/Decompilation

[Carlos Ballesteros Velasco]

On Monday, November 9, 2015 at 1:18:03 PM UTC+1, Alexey Andreev wrote:

So with that on mind, I think that there are ways to do metaprogramming at the compile time right without abusing. You can generating factories having type information at compile time, and then somehow calling that code.

That metaprogramming is kinda a non-standard reflection. That could lead to less code (or not) depending on how it is implemented.

What TeaVM Flavour generates is only visible to compiler internals. You won't see any code in IDE.

Yep. I know. Just wanted to notice which problems I had with some kind of metaprogramming. Java already has its tooling. So no problem with that!

 

 

What is DCE?

Dead Code Elimination. Probably what you are doing already to stage just the required code.

Oh, I see. DCE is usually referred in another context, when you talking about optimizing inside procedures. TeaVM does not eliminate dead code, nor does it eliminate anything. Instead, it uses reverse approach: it "inflates" your program. Starting from main method it looks for another methods that are missing, until it realizes that every required methods are marked for compilation.

You are right. I was calling it like that, but referring to that inflation too. That is what I do too.

 

 Also I have a unrelated question regarding to performance:

(We need to really push the limits so we need to be sure that we don't find a bottleneck just after using a technology)

Are you using javascript typed arrays for byte[] short[] and so on?

Yes, I do. See https://github.com/konsoletyper/teavm/blob/master/core/src/main/resources/org/teavm/javascript/runtime.js#L70

So you are generating javascript from bytecode. How is it done? a while + switch to simulate gotos? Or are you doing some kind of relooping: converting jumps into ifs and whiles as emscripten do. Does it work with arbitrary jumps to code that could produce tooling manipulating bytecode? Or just with valid plain Java/Kotlin/Scala generated code with logic jumps that fits ifs and whiles seamlessly?

Neither of above. Please, take a look at some of live examples in teavm.org and look through classes.js files. No, TeaVM could not be fast if it used naive approach like while+switch. It does not do relooping as well, since relooping is kind of optimization of former approach. Instead, TeaVM reconstructs structural control flow. The algorithm is quite generic and I believe it can handle any reducible CFG, so yes, tooling is accepted, unless it produces irreducible CFG. It is even possible to extend TeaVM to handle irreducible flow. See description here: https://github.com/konsoletyper/teavm/wiki/Decompilation

Cool. Since in Kotlin there are things that are normally statements in java, that are expressions in kotlin (if, when...) maybe the generated code is a pretty strange. Also in java after jumps the java symbolic stack is preserved so you have to store them in locals, kotlin maybe a bit stranger than java in generated code now and in the future. So I am a bit concerned about that.

With my project (jtransc) I was still in the stage of implementing some of the java runtime, and chose the while+switch approach to focus in other areas at the moment. Since I have generated code dynamically for emulators I know that some compilers generate some kind of strange jumps. In fact, in those cases emscripten do the while+switch approach, but adds plain whiles + ifs when possible.

Right now I have downloaded and compiled teavm@master and had not success compiling flavor. But was trying simpler stuff at this point.

I compiled the kotlin (we are going to focus on that) example and modified some stuff to see how output is generated.

Then I tried when + whiles with labels:

fun main(args: Array<String>) {
   mygoto@while (true) {
      while (true) {
         val a = fancycalc()
         val result = when (a) {
            is Int -> "int"
            else -> "other"
         }
         if (fancycalc() !is Int) {
            continue@mygoto;
         }
         System.out.println(result)
         Window.alert("Hello, developer! $result and value $a");
         break
      }
      if (fancycalc() is Int) break
   }
}

fun fancycalc(): Any? {
   var a = 10;
   for (n in 0 .. 3) a += n
   return a
}

Then noticed that the example was running a old version of kotlin 0.11.91.1.

Later I updated the pom.xml to point to the lastest kotlin version: 1.0-beta-1103 and after that, it didn't compiled anymore. So I couldn't continue testing kotlin stuff.

Caused by: org.apache.maven.plugin.MojoExecutionException: Unexpected error occured

at org.teavm.maven.BuildJavascriptMojo.execute(BuildJavascriptMojo.java:106)

at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)

... 20 more

Caused by: java.lang.NullPointerException

at org.teavm.classlib.java.lang.reflect.AnnotationDependencyListener.generateAnnotationValue(AnnotationDependencyListener.java:232)

at org.teavm.classlib.java.lang.reflect.AnnotationDependencyListener.generateAnnotationInstance(AnnotationDependencyListener.java:223)

[Alexey Andreev]

Right now I have downloaded and compiled teavm@master and had not success compiling flavor. But was trying simpler stuff at this point.

You can't build Flavour with master. Try release-0.4.x insdead

I compiled the kotlin (we are going to focus on that) example and modified some stuff to see how output is generated.

Then noticed that the example was running a old version of kotlin 0.11.91.1.

Later I updated the pom.xml to point to the lastest kotlin version: 1.0-beta-1103 and after that, it didn't compiled anymore. So I couldn't continue testing kotlin stuff.

Caused by: org.apache.maven.plugin.MojoExecutionException: Unexpected error occured

at org.teavm.maven.BuildJavascriptMojo.execute(BuildJavascriptMojo.java:106)

at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)

... 20 more

Caused by: java.lang.NullPointerException

at org.teavm.classlib.java.lang.reflect.AnnotationDependencyListener.generateAnnotationValue(AnnotationDependencyListener.java:232)

at org.teavm.classlib.java.lang.reflect.AnnotationDependencyListener.generateAnnotationInstance(AnnotationDependencyListener.java:223)

Strange. I tried this example and it has been built successfully. Can you send me the whole project that reproduces the problem? Maybe there is some configuration issue?

[Carlos Ballesteros Velasco]

Ok I have solved it. I changed that pom.xml with the kotlin version and it failed with that strange error. I provide the output so you can see which error was:

~/Projects/teavm/samples/kotlin   release-0.4.x ●  mvn install -e

[INFO] Error stacktraces are turned on.

[INFO] Scanning for projects...

[INFO]

[INFO] ------------------------------------------------------------------------

[INFO] Building TeaVM Kotlin web application 0.4.1-SNAPSHOT

[INFO] ------------------------------------------------------------------------

[INFO]

[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ teavm-samples-kotlin ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] skip non existing resourceDirectory /Users/soywiz/Projects/teavm/samples/kotlin/src/main/resources

[INFO]

[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ teavm-samples-kotlin ---

[INFO] Nothing to compile - all classes are up to date

[INFO]

[INFO] --- kotlin-maven-plugin:1.0.0-beta-1103:compile (compile) @ teavm-samples-kotlin ---

[INFO] Kotlin Compiler version 1.0.0-beta-1103

[INFO] Compiling Kotlin sources from [/Users/soywiz/Projects/teavm/samples/kotlin/src/main/kotlin]

[INFO] Classpath: /Users/soywiz/Projects/teavm/samples/kotlin/target/classes:/Users/soywiz/.m2/repository/org/jetbrains/kotlin/kotlin-stdlib/1.0.0-beta-1103/kotlin-stdlib-1.0.0-beta-1103.jar:/Users/soywiz/.m2/repository/org/jetbrains/kotlin/kotlin-runtime/1.0.0-beta-1103/kotlin-runtime-1.0.0-beta-1103.jar:/Users/soywiz/.m2/repository/org/teavm/teavm-classlib/0.4.1-SNAPSHOT/teavm-classlib-0.4.1-SNAPSHOT.jar:/Users/soywiz/.m2/repository/org/teavm/teavm-platform/0.4.1-SNAPSHOT/teavm-platform-0.4.1-SNAPSHOT.jar:/Users/soywiz/.m2/repository/org/teavm/teavm-core/0.4.1-SNAPSHOT/teavm-core-0.4.1-SNAPSHOT.jar:/Users/soywiz/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/Users/soywiz/.m2/repository/org/ow2/asm/asm-debug-all/5.0.4/asm-debug-all-5.0.4.jar:/Users/soywiz/.m2/repository/com/carrotsearch/hppc/0.6.1/hppc-0.6.1.jar:/Users/soywiz/.m2/repository/org/teavm/teavm-jso/0.4.1-SNAPSHOT/teavm-jso-0.4.1-SNAPSHOT.jar:/Users/soywiz/.m2/repository/org/teavm/teavm-jso-impl/0.4.1-SNAPSHOT/teavm-jso-impl-0.4.1-SNAPSHOT.jar:/Users/soywiz/.m2/repository/org/mozilla/rhino/1.7.7/rhino-1.7.7.jar:/Users/soywiz/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/Users/soywiz/.m2/repository/com/jcraft/jzlib/1.1.3/jzlib-1.1.3.jar:/Users/soywiz/.m2/repository/joda-time/joda-time/2.7/joda-time-2.7.jar:/Users/soywiz/.m2/repository/org/teavm/teavm-jso-apis/0.4.1-SNAPSHOT/teavm-jso-apis-0.4.1-SNAPSHOT.jar

[INFO] Classes directory is /Users/soywiz/Projects/teavm/samples/kotlin/target/classes

[INFO] Module name is teavm-samples-kotlin

[INFO]

[INFO] --- teavm-maven-plugin:0.4.1-SNAPSHOT:compile (web-client) @ teavm-samples-kotlin ---

[INFO] Preparing classpath for JavaScript generation

[INFO] Using the following classpath for JavaScript generation: /Users/soywiz/.m2/repository/org/jetbrains/kotlin/kotlin-stdlib/1.0.0-beta-1103/kotlin-stdlib-1.0.0-beta-1103.jar:/Users/soywiz/.m2/repository/org/jetbrains/kotlin/kotlin-runtime/1.0.0-beta-1103/kotlin-runtime-1.0.0-beta-1103.jar:/Users/soywiz/.m2/repository/org/teavm/teavm-classlib/0.4.1-SNAPSHOT/teavm-classlib-0.4.1-SNAPSHOT.jar:/Users/soywiz/.m2/repository/org/teavm/teavm-platform/0.4.1-SNAPSHOT/teavm-platform-0.4.1-SNAPSHOT.jar:/Users/soywiz/.m2/repository/org/teavm/teavm-core/0.4.1-SNAPSHOT/teavm-core-0.4.1-SNAPSHOT.jar:/Users/soywiz/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/Users/soywiz/.m2/repository/org/ow2/asm/asm-debug-all/5.0.4/asm-debug-all-5.0.4.jar:/Users/soywiz/.m2/repository/com/carrotsearch/hppc/0.6.1/hppc-0.6.1.jar:/Users/soywiz/.m2/repository/org/teavm/teavm-jso/0.4.1-SNAPSHOT/teavm-jso-0.4.1-SNAPSHOT.jar:/Users/soywiz/.m2/repository/org/teavm/teavm-jso-impl/0.4.1-SNAPSHOT/teavm-jso-impl-0.4.1-SNAPSHOT.jar:/Users/soywiz/.m2/repository/org/mozilla/rhino/1.7.7/rhino-1.7.7.jar:/Users/soywiz/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/Users/soywiz/.m2/repository/com/jcraft/jzlib/1.1.3/jzlib-1.1.3.jar:/Users/soywiz/.m2/repository/joda-time/joda-time/2.7/joda-time-2.7.jar:/Users/soywiz/.m2/repository/org/teavm/teavm-jso-apis/0.4.1-SNAPSHOT/teavm-jso-apis-0.4.1-SNAPSHOT.jar:/Users/soywiz/Projects/teavm/samples/kotlin/target/classes

[INFO] Building JavaScript file

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 4.538 s

[INFO] Finished at: 2015-11-09T15:07:12+01:00

[INFO] Final Memory: 32M/395M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.teavm:teavm-maven-plugin:0.4.1-SNAPSHOT:compile (web-client) on project teavm-samples-kotlin: Unexpected error occured: NullPointerException -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.teavm:teavm-maven-plugin:0.4.1-SNAPSHOT:compile (web-client) on project teavm-samples-kotlin: Unexpected error occured

at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)

at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)

at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)

at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)

at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)

at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)

at org.apache.maven.cli.MavenCli.execute(MavenCli.java:862)

at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:286)

at org.apache.maven.cli.MavenCli.main(MavenCli.java:197)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:483)

at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)

at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)

at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)

at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)

Caused by: org.apache.maven.plugin.MojoExecutionException: Unexpected error occured

at org.teavm.maven.BuildJavascriptMojo.execute(BuildJavascriptMojo.java:106)

at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)

... 20 more

Caused by: java.lang.NullPointerException

at org.teavm.classlib.java.lang.reflect.AnnotationDependencyListener.generateAnnotationValue(AnnotationDependencyListener.java:232)

at org.teavm.classlib.java.lang.reflect.AnnotationDependencyListener.generateAnnotationInstance(AnnotationDependencyListener.java:223)

at org.teavm.classlib.java.lang.reflect.AnnotationDependencyListener.addReader(AnnotationDependencyListener.java:201)

at org.teavm.classlib.java.lang.reflect.AnnotationDependencyListener.createAnnotationClass(AnnotationDependencyListener.java:173)

at org.teavm.classlib.java.lang.reflect.AnnotationDependencyListener.classReached(AnnotationDependencyListener.java:57)

at org.teavm.dependency.DependencyChecker.lambda$linkClass$9(DependencyChecker.java:275)

at org.teavm.dependency.DependencyChecker$$Lambda$27/497967453.run(Unknown Source)

at org.teavm.dependency.DependencyChecker.processQueue(DependencyChecker.java:505)

at org.teavm.dependency.DependencyChecker.processDependencies(DependencyChecker.java:518)

at org.teavm.vm.TeaVM.build(TeaVM.java:390)

at org.teavm.tooling.TeaVMTool.generate(TeaVMTool.java:344)

at org.teavm.maven.BuildJavascriptMojo.execute(BuildJavascriptMojo.java:101)

... 22 more

[ERROR]

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

----------------

Later I have done a "mvn clean" and the error changed. Later versions of Koltin changed how top-level functions are stored including the main, so I have changed it in the pom.xml: Instead of KotlinPackage, now the class is HelloKt because of the name of the file.

So that was fixed :)

---

BTW. Have you ever consider outputing to a binary format like webassembly is going to do? They said that the time they are converting binary format into javascript is pretty low. And would reduce the output size a lot while providing type information. It would be something like android do with DEX files: a single file just all the required dependencies obviously trimming non-used methods and classes, and instead of providing java bytecode you would provide statements and expressions as a pretty simple AST, that sould be smaller than javascript code itself, but that would have plain structures instead of goto (doing all that work at compile time). That binary format would allow you to have constant pools and providing type information that could be used to generate javascript. I think that there are some work to get source maps working on generated code (web assembly for example).

Generated code could have all that stuff.

JVM.registerClass('java.util.Objects', 'java.lang.Object', [], function (ctx) {
   ctx.registerMethod(true, 'equals', '(Ljava/lang/Object;Ljava/lang/Object;)Z', function(p0, p1) {
      throw new Error('Native or abstract :java.util.Objects: equals(Ljava/lang/Object;Ljava/lang/Object;)Z ');
   });
   ctx.registerMethod(true, 'hashCode', '(Ljava/lang/Object;)I', function(p0) {
      throw new Error('Native or abstract :java.util.Objects: hashCode(Ljava/lang/Object;)I ');
   });
});

Because how javascript works you can create high performance code even when describing classes like this.

JVM.registerClass('kotlin.io.ConsoleKt', 'java.lang.Object', [], function (ctx) {
   ctx.registerField(true, 'stdin', 'Ljava/io/BufferedReader;');
   ctx.registerMethod(true, 'println', '(I)V', function(p0) {
      var $r0 = null;
      var i0 = 0;
      i0 = p0; /* SET */
      $r0 = JVM.classes['java.lang.System']['out'] /* STATIC_FIELD_ACCESS */; /* SET */
      $r0['println(I)V'](i0); /* STM_EXPR */
      return ;
   });

And since on javascript doing this.field is the same as doing this["field"] (in terms of performance) you don't even need to mangle names.

And since you would be targetting a compact binary format all that verbosity for the generated wouldn't be a problem. But you would get reflection information to do stuff.

You could serialize/unserialize AST like this (a non-working example):

function readStatement() {

     switch (readByte()) {

             case 0: output(this.readExpression() + ';'); // expression statement

     }

}

function readExpression() {

     switch (readByte()) {

             case 0:  // binary op

                   var op = readByte();

                   val l = this.readExpression();

                   val r = this.readExpression();

                   return '' + l + op + r;

                   break;

             case 1:

                   return '' + this.readInt();

                   // integer literal

                   break;

     }

}

Probably what you expend generating javascript is less than what you gain providing reflection and reducing the size of the output. Just a small javascript with the runtime + a small code with the dynamic javascript generation.

This is just an idea. I know it would require some work. But I wanted to share it with you :)

----

Anyway. I will continue testing stuff and providing feedback. Thank you very much for your time and your help!

[Carlos Ballesteros Velasco]

Hello again Alexey.

Yesterday I was investigating a bit more. First I read your wiki, the technical stuff you wrote is awesome. The readings about graph theory was very interesting. I did some work in that direction but was wild and with my own experience, without graph theories. And this is hugely better. Congratulations :) 

I was able to run flavor, and I was trying to understand the part about mp, and how it works.

As far as I got (correct me if i'm wrong), it seems to me that there is a plugin system that you hook with maven while building, that allows you yo intercept some phases of the building.

You can listen to class and method discovery, and probably write or rewrite methods and other stuff. Is that so, or I understood it bad? You can write your own methods using java bytecode, or IR AST or both?

So with that in mind: in order to get some of the stuff we need we would create a plugin that would replace a method (using reflection) that would do the instantiation stuff we need adding to it information about types and how to construct them. I suppose that since that generate code directly instantiating classes, would get minified.

So is this the way it is supposed to be done? Maybe not. That's why I'm asking.

[Alexey Andreev]

I was able to run flavor, and I was trying to understand the part about mp, and how it works.

As far as I got (correct me if i'm wrong), it seems to me that there is a plugin system that you hook with maven while building, that allows you yo intercept some phases of the building.

You can listen to class and method discovery, and probably write or rewrite methods and other stuff. Is that so, or I understood it bad? You can write your own methods using java bytecode, or IR AST or both?

You are very close to the truth.

1. Hooking is done not int Maven, but in TeaVM level itself. You can write frontends for different build tools and plugin API still works.
2. There are several phases: class loading (when bytecode got parsed), dependency checker (that discovers which methods to compile) and rendering (producing JS). You can hook any of these phases, but it's not recommended to hook rendering phase. The first two phases work with TeaVM IR.
3. TeaVM does not allow you to operate you on Java bytecode and JavaScript AST. You can use another approaches to modify Java bytecode if you want, such as build-time instrumentation and javaagent switch. The latter option, is however, requires you to run Eclipse and Maven with javaagent, and I'm not sure it works properly.

So with that in mind: in order to get some of the stuff we need we would create a plugin that would replace a method (using reflection) that would do the instantiation stuff we need adding to it information about types and how to construct them. I suppose that since that generate code directly instantiating classes, would get minified.

Yes, you can do it manually, but recently I created metaprogramming module in Flavour project, which works as the following:

1. You create static native method and mark it with @Proxy annotation that points to class that generates a proxy
2. This method should take one argument and return value
3. Your ProxyGenerator generates code that handles every discovered parameter types.
4. Metaprogramming module automatically tracks all parameter types and dispatches execution to corresponding implementation generated by ProxyGenerator.
5. You can generate additional classes from ProxyGenerator (see ProxyGeneratorContext.submitClass).

The only place where this API is used is Templates class. I am going to improve this API and include another usecase: you pass Class to proxied method and it produces something concerning this class. This is useful when you need to proxy interfaces.

For example, consider we want method that calls "String debug()" method of an object, even if this object does not implement special interface.

@Proxy(DebugProxyGenerator.class)
public static String callDebug(Object obj);

public static class DebugProxyGenerator implements ProxyGenerator {
    private ProxyGeneratorContext context;

    @Override
    public void setContext(ProxyGeneratorContext context) {
        this.context = context;
    }

    @Override
    public void generate(String type, ProgramEmitter pe, CallLocation location) {
        ValueEmitter objVar = pe.var(1, ValueType.object(type));
        MethodReference debugReference = new MethodReference(type, "debug", ValueType.parse(String.class));
        MethodReader debugMethod = context.getClassSource().resolve(debugReference);
        if (debugMethod == null) {
            context.getDiagnostics().warning(location, "{{c0}} does not provide debug() method", type);
            pe.constant("no debug method").returnValue();
        } else {
            objVar.invokeVirtual(debugReference).returnValue();
        }
    }
}

// Use it
callDebug(5); // produces "no debug method" in runtime and warning in compile time
callDebug(new Debuggable()); // produces "it works"

class Debuggable {
    private String debug() {
        return "it works";
    }
}

Please, be careful with metaprogramming module, it's still very unstable, it does not provide good diagnostics. Also, keep in mind that for now TeaVM does not validate IR produced externally, do you'll get unpredictable errors if you produce invalid IR.

I'll include MP module directly into TeaVM in future releases, as well as IR verifier.