GraalVM's native-image incompatible with Clojure's eval?

[Khalid Jebbari]

Hi,

I was doing a small experiment with Clojure and GraalVM and ended with this minimal reproduction case of an incompatibility between Clojure's `eval` and GraalVM's native-image tool (the program that compiles a JVM program to a native executable, based on the GraalVM's SubstrateVM compiler).

Here's the Clojure program:

(ns test-cli.main

  (:gen-class))

(set! *warn-on-reflection* true)

(defn -main

  "I don't do a whole lot ... yet."

  [& args]

  (println (+ 1 1)) ;; trick to force loading clojure.lang.Numbers, not working

  (eval (read-string "(+ 1 1)")))

Using Clojure 1.9.0 and GraalVM version 1.0.0-rc9.

When I compile it with the option "--report-unsupported-elements-at-runtime" (which gives a more precised error message), here's the output when I try executing the resulting executable:

Exception in thread "main" java.lang.ClassNotFoundException: clojure.lang.Numbers, compiling:(NO_SOURCE_PATH:0:0)

        at java.lang.Throwable.<init>(Throwable.java:287)

        at java.lang.Exception.<init>(Exception.java:84)

        at java.lang.RuntimeException.<init>(RuntimeException.java:80)

        at clojure.lang.Compiler$CompilerException.<init>(Compiler.java:6804)

        at clojure.lang.Compiler.analyzeSeq(Compiler.java:7010)

        at clojure.lang.Compiler.analyze(Compiler.java:6773)

        at clojure.lang.Compiler.analyze(Compiler.java:6729)

        at clojure.lang.Compiler.analyzeSeq(Compiler.java:6998)

        at clojure.lang.Compiler.analyze(Compiler.java:6773)

        at clojure.lang.Compiler.analyze(Compiler.java:6729)

        at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:6100)

        at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5460)

        at clojure.lang.Compiler$FnExpr.parse(Compiler.java:4022)

        at clojure.lang.Compiler.analyzeSeq(Compiler.java:7001)

        at clojure.lang.Compiler.analyze(Compiler.java:6773)

        at clojure.lang.Compiler.eval(Compiler.java:7059)

        at clojure.lang.Compiler.eval(Compiler.java:7025)

        at clojure.core$eval.invokeStatic(core.clj:3206)

        at test_cli.main$_main.invokeStatic(main.clj:7)

        at test_cli.main$_main.doInvoke(main.clj:7)

        at clojure.lang.RestFn.invoke(RestFn.java:397)

        at clojure.lang.AFn.applyToHelper(AFn.java:152)

        at clojure.lang.RestFn.applyTo(RestFn.java:132)

        at test_cli.main.main(Unknown Source)

        at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:164)

Caused by: java.lang.ClassNotFoundException: clojure.lang.Numbers

        at java.lang.Throwable.<init>(Throwable.java:287)

        at java.lang.Exception.<init>(Exception.java:84)

        at java.lang.ReflectiveOperationException.<init>(ReflectiveOperationException.java:75)

        at java.lang.ClassNotFoundException.<init>(ClassNotFoundException.java:82)

        at com.oracle.svm.core.hub.ClassForNameSupport.forName(ClassForNameSupport.java:51)

        at com.oracle.svm.core.hub.DynamicHub.forName(DynamicHub.java:1036)

        at clojure.lang.RT.classForName(RT.java:2204)

        at clojure.lang.RT.classForNameNonLoading(RT.java:2217)

        at clojure.lang.Compiler$HostExpr.maybeClass(Compiler.java:1041)

        at clojure.lang.Compiler$HostExpr$Parser.parse(Compiler.java:982)

        at clojure.lang.Compiler.analyzeSeq(Compiler.java:7003)

        ... 20 more

I'm no expert in Java/JVM and would like to understand the problem. According to the SubstrateVM documentation (https://github.com/oracle/graal/blob/master/substratevm/LIMITATIONS.md) it can't compile Dynamic Class Loading/Unloading. Is Clojure's `eval` doing such dynamic loading? Or doing something else not supported by SubstrateVM as said in the documentation?

Thanks *a lot* in advance for you answers.

[Gary Trakhman]

Yes, eval will generate classes in a dynamic classloader, load them, then call methods on the newly formed class/object except for too-simple-to-be-interesting cases.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Khalid Jebbari]

Thanks a lot. So it means Clojure's `eval` is by design incompatible with SubstrateVM.

Does anyone know of others hard incompatibilities ?

[Ghadi Shayban]

Substrate makes a closed world assumption. jaotc is open world, and also based on Graal.

[Khalid Jebbari]

Thanks for pointing out jaotc. It's not what I need but it's good to know it exists. Do you know if it's possible to AOT (part of) a Clojure program with it? I'm curious.

On Mon, Nov 26, 2018, 7:06 PM Ghadi Shayban <gsha...@gmail.com wrote:

Substrate makes a closed world assumption.  jaotc is open world, and also based on Graal.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/LMbNOg67wcw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Gerard Klijs]

It's already possibly, but not really part of, that would make little sense anyway. It shines with tools you want to run quickly, where the reduction in startup time helps. There was a tweet if a formatter compiled to a native image.
I also tried to create a native image for a pretty project, but turned out there are unresolved issues with the Java Kafka Client. Another place where it can be great is really scalable micro services and/or use them as serverless functions. But you lose yit optimisations. So peak performance probably is better running on a JVM.

[Jeroen van Dijk]

Hi Khalid, 

Reflections [1] are a big problem too for Graalvm. Here is a thread of playing with Clojure and Graalvm [2]. If you follow the links, you will see some issues I ran into.

Hth,

Jeroen

[1] Ones that are and aren't caught by *warn-on-reflections* https://github.com/dundalek/closh/pull/105/files#diff-336b8a8dffae65260f854867fc8529b5R9

[2] https://github.com/dundalek/closh/issues/93

[Khalid Jebbari]

Thanks a lot for the links. Indeed it seems that parts of Clojure inevitably end up using reflections unless anotated.

[Didier]

You can give substratevm a reflection json file though. That did the trick for me.

https://github.com/oracle/graal/blob/master/substratevm/REFLECTION.md

So you just need to tell it in advance what class you will reflect on and how. Then it should work.

[Didier]

Hum, I hadn't heard of jaotc. Hopefully it will continue to be developed. A slower Clojure which starts mich faster yet is fully compatible would be a nice trade off in many cases.

I found this reading interesting: https://mjg123.github.io/2017/10/04/AppCDS-and-Clojure.html#using-cds

For using jaoct and appCDs and then clojire.main starts in 0.4 second.

[Khalid Jebbari]

Thank you Didier, I've seen this documentation about the reflection json file. However it appears that even `(set! *warn-on-reflections true)` doesn't always catch reflections. I suppose it means I have to use profilers like jvisualvm (don't remember the exact name) to watch for calls to reflection-related class/methods/functions to catch them, right ?

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/LMbNOg67wcw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Didier]

In my case, substratevm itself pointed to me which class was being reflected on. Though it didn't specify which fields or methods, but I just set it to allow all methods and fields of the reported classes to be allowed.