ClassLoader setup

[Csaba Kincses]

Hey there,

I have an issue with classloader setup when running a TeaVM build using TeaVMBuilder class.

I provided a customized classloader and looks like findClass is not properly invoked.

The classloader should be able to load Scala classes, though I get an error like that:

Class {{c0}} was not found List(scala.App)

Can you help me understand how to set this up properly to be able to create JavaScript from Scala?

[Alexey Andreev]

There are Maven and Gradle plugin out of the box, and there's also 3rd party sbt plugin. So instead of reinventing the wheel, please take a look at the existing solution. Please, take a looks at working example with Gradle: https://github.com/konsoletyper/teavm/tree/master/samples/scala

If for some reason you still need your own builder, I can't help without even knowing what's going on, what are you doing or how exactly you are building class loader.

[Csaba Kincses]

Thanks for looking into this.

I'll need to implement on-the-fly code generation - class hotloading on the backend and JS drop-ins for the frontend - the backend part is ready, now comes the frontend part. Using the plugins does not with in my workflow, it would be more cumbersome to use them in this case as this is out of what they are designed for, so "reinventing the wheel" seems a better solution. This was not problemmatic for me with the Scala compiler, I hope it won't be with TeaVM. I'll use TeaVM in favour ScalaJs as it promises a more straightforward workflow to deal with Scala 3.

Regarding the classloader implementation: 

I reuse the classloader which already worked for the Scala compiler part. This works as such:

Consists of Scala compiler's AbstractFileClassLoader and my own ContextClassLoader:

private val outputDirectory = VirtualDirectory("(memory)")
private val appliedClassLoader: AbstractFileClassLoader =
AbstractFileClassLoader(outputDirectory, ContextClassLoader(classPathContext))

Later this appliedClassLoader is passed to TeaVM:

private val classSource = ClasspathClassHolderSource(appliedClassLoader, ReferenceCache())
private val vm = TeaVMBuilder(target).setClassLoader(appliedClassLoader).setClassSource(classSource).build()

The ContextClassLoader implements a complex behavior, depending on settings it can load an exclusive list of jars, or java.class.path classes (which are set by the Scala SBT build tool and always include Scala libraries) extended with a list of jars. The parent is the platform classloader.

Regarding error output: as expected, platform classes look to be available, entry point class (the on-the-fly compiled one which resides in the AbstractFileClassLoader) is available. The surprise is that scala.App - which is a scala library class - was unavailable.

ContextClassLoader has an overridden findClass method, if I try to log from there, nothing happens:

override final def findClass(name: String): Class[?] =

println("Hello from findClass")

(javaCpLoaderOpt.flatMap(cl => Try(Class.forName(name, true, cl)).toOption) orElse
userClassLoaderOpt.flatMap(cl => Try(Class.forName(name, true, cl)).toOption))
.getOrElse(super.findClass(name))

Please let me know if any further information is needed. I need a clue to go on debugging this issue

[Alexey Andreev]

Ah, that's it. TeaVM does not need to load classes, unless they are TeaVM plugin classes or metaprogramming logic. Instead, it calls getResourceAsStream to read bytecode. In plugin code I don't use any custom class loaders with some sort of complex logic. Instead, I use URLClassLoader

[Csaba Kincses]

Thanks! A minor clarification would help: so I'll need to override getResourceAsStream as this will be invoked on the classloader I provided, or at some point, class resolution is switched to another URLClassLoader, which is unrelated? I'll check out the code, reimplement once all clear, and update as this can be useful for Scala users

[Alexey Andreev]

 

or at some point, class resolution is switched to another URLClassLoader, which is unrelated?

 

No, that would never happen. TeaVM will use class loader you provided and will call getResourceAsStream to fetch bytecode.

 

[Csaba Kincses]

Cool, thanks

[Csaba Kincses]

I fixed the original error, but issues persist. Overriding getResource fixed the previous issue. I log bytecode loading from this method now. What I find contradictory is that I get errors like:

Method {{m0}} was not found List(java.lang.invoke.MethodHandle.invoke()V)

While java.lang.invoke.MethodHandle was successfully loaded according to what is logged from my overridden getResource method.

[Alexey Andreev]

I strongly suggest you to start with Gradle or SBT plugin to make proof-of-concept and then write your own builder based on TeaVMTool.

As for these errors: they give nothing to me. Please, learn TeaVM Maven/Gradle/CLI sources to find out how to log full messages. Also, make sure that you included teavm-classlib into class loader. java.lang.invoke.MethodHandle should not *ever* be read by TeaVM, as well as any other class from java.* package

понедельник, 9 сентября 2024 г. в 17:05:13 UTC+2, kincses.cs...@gmail.com:

[Csaba Kincses]

I strongly suggest you to start with Gradle or SBT plugin to make proof-of-concept and then write your own builder based on TeaVMTool.

I already browsed related classes and kept an eye on TeaVMTool when configuring, I try to ask only what remains unclear regarding the concepts of TeaVM internals.

I fixed the logging, now its rendered using

TeaVMProblemRenderer.describeProblems(vm, log)

 instead of getProblemProvider.getProblems.asScala ... foreach(println)

I added teavm-classlib, though the issue persists.

I try to summarize the error log effectively:

There are two kinds of errors in the log, it's Method not found, and Substitutor for bootstrap method not found

Now I see a call log below the error messages, here are two examples:

ERROR: Method java.lang.invoke.MethodHandle.invoke()V was not found
    at scala.runtime.Statics.releaseFence(Statics.java:148)
    at Hello$.<clinit>((inline):2)
    at Hello.main(-1)

ERROR: Substitutor for bootstrap method java.lang.invoke.LambdaMetafactory.altMetafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;[Ljava/lang/Object;)Ljava/lang/invoke/CallSite; was not found
    at scala.App.main(App.scala:98)
    at scala.App.main$(App.scala:96)
    at Hello$.main((inline):1)
    at Hello.main(-1)

What is common is that all call logs originate from the entrypoint Hello.main, and mention that a java.lang.invoke.* method is missing.

My findings and what I do not understand conceptually are:

- What is supposed to be a substitutor for a bootstrap method? Is it a T prefixed class from teavm-classlib

- My classloader had to serve java.lang.invoke.* classes, and according to logs, it did successfully, though there were no lookups for T prefixed classes logged by the classloader

So, to wrap up, though I actively browse existing code, I think I'll need to understand the above-mentioned concepts to create an embedded on-the-fly generator.

I can't make a promise regarding that, but such working solutions can attract Scala users, I hope I can implement this.

[Alexey Andreev]

I strongly suggest you to start with Gradle or SBT plugin to make proof-of-concept and then write your own builder based on TeaVMTool.

I already browsed related classes and kept an eye on TeaVMTool when configuring, I try to ask only what remains unclear regarding the concepts of TeaVM internals.

You missed the point. I did not suggest to learn TeaVM plugin code, I suggest to start using plugins and make sure that TeaVM works for your code base. Then you may start writing your own TeaVM tool to make this dynamic code reloading or what you want.

 

There are two kinds of errors in the log, it's Method not found, and Substitutor for bootstrap method not found

Now I see a call log below the error messages, here are two examples:

ERROR: Method java.lang.invoke.MethodHandle.invoke()V was not found
    at scala.runtime.Statics.releaseFence(Statics.java:148)
    at Hello$.<clinit>((inline):2)
    at Hello.main(-1)

ERROR: Substitutor for bootstrap method java.lang.invoke.LambdaMetafactory.altMetafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;[Ljava/lang/Object;)Ljava/lang/invoke/CallSite; was not found
    at scala.App.main(App.scala:98)
    at scala.App.main$(App.scala:96)
    at Hello$.main((inline):1)
    at Hello.main(-1)

What is common is that all call logs originate from the entrypoint Hello.main, and mention that a java.lang.invoke.* method is missing.

 

Together with getResourceAsStream you may need to override getResources, since TeaVM uses it to find plugins, which are probably not loaded in your case. Anyway, I suggest you to use URLClassLoader instead of reinventing the wheel.

[Csaba Kincses]

I already have the ContextClassLoader implementation, and I want to reuse it to enforce using the same classpath where my compiler extension generates code based on a specific input.