Best way to dynamically spin Java classes?

275 views
Skip to first unread message

Steven Stewart-Gallus

unread,
Apr 26, 2020, 12:36:07 PM4/26/20
to mechanical-sympathy
I'm spinning a lot of classes dynamically for an interpreter.
I'm not sure the best way to load the classes into the VM.

Aside from Unsafe's defineAnonymousClass what's the best way to load classes into the VM?

I believe I could create a new classloader for each spun class but IIRC classloaders are expensive.

I believe as much as possible I'd want to reuse MethodHandle and LambdaMetafactory to rely on the JVMs use of defineAnonymousClass.

Dr Heinz M. Kabutz

unread,
Apr 26, 2020, 1:12:10 PM4/26/20
to mechanica...@googlegroups.com
In Java 15 we will have hidden classes, which seem like the right model for what you are trying to do. 

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/8ded7316-7675-4747-afe5-5150a7b6009e%40googlegroups.com.
--
Dr Heinz M. Kabutz (PhD CompSci)
Author of "The Java(tm) Specialists' Newsletter"
Sun/Oracle Java Champion
JavaOne Rockstar Speaker
http://www.javaspecialists.eu
Tel: +30 69 75 595 262
Skype: kabutz

Remi Forax

unread,
Apr 26, 2020, 1:31:20 PM4/26/20
to mechanical-sympathy

Steven Stewart-Gallus

unread,
May 1, 2020, 11:51:47 AM5/1/20
to mechanical-sympathy
Huh
I've often thought about doing similar.
Two nits:

1.
RT.java is clever, I know ClassValue was probably the way to do things but couldn't figure out how to do it. But it's simpler to just write to a static field to the class anyway.
One advantage of creating new classloaders over defineHiddenClass is you can just pass in the data as part of the classloader.

2.
I wouldn't even bother with a traditional constructor but just use MethodHandles to set the delegate field.

Anyway very cool what defineHiddenClass will enable.

On Sunday, April 26, 2020 at 10:31:20 AM UTC-7, Remi Forax wrote:
Shameless plug,
i've written a simple library that shows how lookup.defineHiddenClass [1] can be used to implement dynamic proxies

regards,
Rémi


De: "Dr Heinz M. Kabutz" <he...@javaspecialists.eu>
À: "mechanical-sympathy" <mechanica...@googlegroups.com>
Envoyé: Dimanche 26 Avril 2020 19:11:53
Objet: Re: Best way to dynamically spin Java classes?
In Java 15 we will have hidden classes, which seem like the right model for what you are trying to do. 
On Sun, 26 Apr 2020 at 19:36, Steven Stewart-Gallus <stevensele...@gmail.com> wrote:
I'm spinning a lot of classes dynamically for an interpreter.
I'm not sure the best way to load the classes into the VM.

Aside from Unsafe's defineAnonymousClass what's the best way to load classes into the VM?

I believe I could create a new classloader for each spun class but IIRC classloaders are expensive.

I believe as much as possible I'd want to reuse MethodHandle and LambdaMetafactory to rely on the JVMs use of defineAnonymousClass.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.
--
Dr Heinz M. Kabutz (PhD CompSci)
Author of "The Java(tm) Specialists' Newsletter"
Sun/Oracle Java Champion
JavaOne Rockstar Speaker
http://www.javaspecialists.eu
Tel: +30 69 75 595 262
Skype: kabutz

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

Remi Forax

unread,
May 1, 2020, 12:26:02 PM5/1/20
to mechanical-sympathy
De: "stevenselectronicmail" <stevensele...@gmail.com>
À: "mechanical-sympathy" <mechanica...@googlegroups.com>
Envoyé: Vendredi 1 Mai 2020 17:51:47
Objet: Re: Best way to dynamically spin Java classes?
Huh
I've often thought about doing similar.
Two nits:

1.
RT.java is clever, I know ClassValue was probably the way to do things but couldn't figure out how to do it. But it's simpler to just write to a static field to the class anyway.

it's not that easy if you want the field to be final too, because you don't control when a static block is executed, and if it is not final, users of the proxy API can mess with it because the API provide them a lookup to the proxy

One advantage of creating new classloaders over defineHiddenClass is you can just pass in the data as part of the classloader.

Yes,
using the classloader is what most of the dynamic languages JRuby or Groovy do.

If you take a look the source of MethodHandles.Lookup, internally the VM use a something called the classData that is taken as parameter of define(...) and stored in a static field by the VM (it's not a true static field because you can not access it using regular bytecode but it is stored in the same space). When it will become part of the offcial API, defineHiddenClass will becomes even easier to use.


2.
I wouldn't even bother with a traditional constructor but just use MethodHandles to set the delegate field.

I want the field to be final. Final fields of Hidden Class are truly final for the JIT, so i get better perf if a proxy is, by example, stored in a static final, so has a good citizen, i'm using a constructor :)
Also because final really means final, you can not use reflection or java.lang.invoke to set it after the fact.


Anyway very cool what defineHiddenClass will enable.

yes, all these features were available but only internally for the jdk, i think since jdk 7, finding a safe way to expose them take times.

regards,
Rémi


On Sunday, April 26, 2020 at 10:31:20 AM UTC-7, Remi Forax wrote:
Shameless plug,
i've written a simple library that shows how lookup.defineHiddenClass [1] can be used to implement dynamic proxies

regards,
Rémi


De: "Dr Heinz M. Kabutz" <he...@javaspecialists.eu>
À: "mechanical-sympathy" <mechanica...@googlegroups.com>
Envoyé: Dimanche 26 Avril 2020 19:11:53
Objet: Re: Best way to dynamically spin Java classes?
In Java 15 we will have hidden classes, which seem like the right model for what you are trying to do. 
On Sun, 26 Apr 2020 at 19:36, Steven Stewart-Gallus <stevensele...@gmail.com> wrote:
I'm spinning a lot of classes dynamically for an interpreter.
I'm not sure the best way to load the classes into the VM.

Aside from Unsafe's defineAnonymousClass what's the best way to load classes into the VM?

I believe I could create a new classloader for each spun class but IIRC classloaders are expensive.

I believe as much as possible I'd want to reuse MethodHandle and LambdaMetafactory to rely on the JVMs use of defineAnonymousClass.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
--
Dr Heinz M. Kabutz (PhD CompSci)
Author of "The Java(tm) Specialists' Newsletter"
Sun/Oracle Java Champion
JavaOne Rockstar Speaker
http://www.javaspecialists.eu
Tel: +30 69 75 595 262
Skype: kabutz
--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/b14b88e4-3a30-447d-a29b-ca7ce7fe738d%40googlegroups.com.

Steven Stewart-Gallus

unread,
May 7, 2020, 1:47:09 PM5/7/20
to mechanical-sympathy
Hi,

Just going to update this thread with a link to the repo I am asking this question for which I finally see fit to share https://github.com/sstewartgallus/jsystemf .

Mostly copypasting from a reddit post I made.

So I plan to use this as the core of a functional language similar to
the Glasgow Haskell Compiler's core but for now I don't have a higher
language made yet.

Atypically I use higher order abstract syntax for the core IR. Higher
order abstract syntax cheats for implementing things like closures by
using lambda expressions. I compile these lambdas to a point free
representation by repeatedly substituting the free variables with
pairs.

Basically Type.INT.l(x -> Type.INT.l(y -> x)) becomes rewritten to
something like curry(Type.INT.and(Type.INT).l(v -> v.second())) which
then can be reduced to simple category theory based combinators (It's
well known the lambda calculus can be compiled to closed cartesian
Unexpectedly I found that evaluation of the product of a thunk and a
function Category<<F<A, B>, A> can be implemented using uncurrying and
currying implements closures.

After I compile to a point-free closed cartesian category combinator
representation I then compile my types to a similar
representation. Just as lambda abstraction is curried so is universal
quantification over types. Interestingly existential quantification
plays the same role products play in compiling functions.

Still I haven't really implemented generics properly yet and I
struggle a bit because of Java's mediocre type system and also because
this is something I have mostly figured out myself.

Next I compile the CCC representation to MethodHandle. As a legacy of
an earlier much more complicated code base before I settled on the CCC
representation I use bytebuddy to spin custom closure types that
represent an environment + a method. I use invokedynamic heavily for
linking calls. I still haven't implemented supersaturated/partially
saturated callsites yet meaning that multiargument Invokers
(dynamically spun classes that fill interfaces for invoking function
values) don't really work.

I think I'm pretty confident I have a solid foundation now though and
just have to figure out the type representation at runtime. Currently
the type Type is more akin to a ClassDesc in Java than an actual live
Class value. I need to figure out how to instantiate/cache generic
types and methods. I hope that by flattening generic tuple types I can
recover a more JVM native argument passing convention as as of now
everything is boxed into pairs.

A lazy pure functional language on the JVM has been an itch I've been
trying to scratch for a while (even playing with GraalVM at times,
which proved not flexible enough) but I think I've figured out a solid
foundation with this even though I have not yet implemented tail calls
or lazy evaluation.

I thought some of you might be interested in this kind of bytecode spinning stuff.

Nathan Fisher

unread,
May 20, 2020, 1:38:52 AM5/20/20
to mechanica...@googlegroups.com
There’s also bytebuddy which has commercial support if that’s important to you.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
--
Nathan Fisher

Steven Stewart-Gallus

unread,
May 21, 2020, 2:08:50 PM5/21/20
to mechanical-sympathy
Hi Nathan,

Thanks. ByteBuddy is extremely overpowered for my use case. I am just spinning extremely light stub classes.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages