Java Interop on steroids?

[eglue]

Don't get me wrong, I'm as much against types as the next R̶i̶c̶h̶ ̶H̶i̶c̶k̶e̶y̶  guy.

However -- there are many popular Java frameworks that love to reflect on their annotations and their generic type signatures.

To name a heavyweight: Spring. But also, of late: big data frameworks, many written in Java, love reflecting on generic type signatures. My org is looking at Beam and Flink, for example.

These frameworks use types not for the static checking really but as parameters governing their own dynamic behavior. For example, Spring will use types at runtime to simply match objects to where they should be dynamically injected. Beam will look at your type signatures and do runtime validations to ensure it can process things appropriately. Of course this is unfortunate, using types this way, when it is all really just data. Clojure does -- or would do -- it better, simpler, directer, and all of that.

Yet we would like to leverage these frameworks. Or rather, we must for various pragmatic and business reasons.

And any time we need to "communicate" to these frameworks "through" their desired fashion of generic types and annotations, we can, of course, create the appropriate .java files to represent what is needed (and then do the invocation back to Clojure via IFn.invoke or Compiler.eval, etc). Yes, this works.

However this is quite tedious because in these frameworks I mentioned you end up having to create these Java files quite a bit. For example, when expressing a streaming data pipeline to Beam, you may specify multiple transforms, each a function with its own type signature.

A little searching and it seems Clojure has shied away from generating generic type information in places where it could offer this capability. 

For example, in `proxy` ... or I suppose also in `gen-class`, `reify`, and other dynamic bytecode generation features of Clojure.

However it seems to me that `proxy` (and these others) could allow one to pass in a representation of desired type arguments, annotations, etc. and then we could remain in Clojure to interop with these popular frameworks.

I respect Clojure's efforts to keep its core small and wait for worthy features to prove themselves.

So my question is not when is Clojure going to do this, but rather:

Are there any precedents in the community for someone building out the kind of richer Java interop that I'm nodding toward here?

For example, does anyone know of an attempt out there to build a `proxy` plus-plus, that would allow one to extend a generic class with provided type parameters and have this metadata properly rendered in the bytecode that proxy produces?

If not, as a practical and hopefully quick and workable solution, I was thinking it'd be possible to take the bytecode emitted by proxy and re-run it through ASM to create a *new* class with simply the proxy-produced class bytes filled-in with the desired, provided type parameters. I bet this could be sufficient and fast, with the slight overhead of the extra class.

To do this, I think I'd need access to these proxy-made bytes... either by having proxy answer them somehow, or offering a hook to contribute to the defined bytecode before it is committed to the classloader, or by having DynamicClassLoader have these bytes on hand for inquiring parties, or something else along these lines. This would likely be something that Clojure core would have to expose .. correct me if I'm wrong.

Would love to hear any other immediate thoughts on this.

I think once you realize that this generic type information is not even being used for "static typing" by these frameworks but rather as an (albeit poor) means to receive semantic information from their clients (as parameters to govern their own dynamic behavior), then the need/value of being able to remain in Clojure and communicate to these libraries through generic params and annotations perhaps becomes more understandable..

[Sean Corfield]

You might be interested in how we provide type-based annotations on Clojure functions so that tooling (in our case New Relic) sees those annotations:

 

https://corfield.org/blog/2013/05/01/instrumenting-clojure-for-new-relic-monitoring/

 

I agree that this could be a lot easier.

 

Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

 

---

From: clo...@googlegroups.com <clo...@googlegroups.com> on behalf of eglue <atd...@gmail.com>
Sent: Thursday, June 20, 2019 9:03:45 PM
To: Clojure
Subject: Java Interop on steroids?

 

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/128dd732-b79e-4c35-999f-691cdc42512b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Nuernberger]

Sean,

That is an interesting blog post.  Sorry if I am not following everything but why not use the annotation support in gen-class for those types of things?

https://github.com/clojure/clojure/blob/8af7e9a92570eb28c58b15481ae9c271d891c028/test/clojure/test_clojure/genclass/examples.clj#L34

clo...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clo...@googlegroups.com.

[Nathan Fisher]

Storm recently moved away from Clojure in its core.

https://storm.apache.org/2019/05/30/storm200-released.html

I wonder how much of the legacy Clojure core could be optimised or if they reached an upper limit imposed by the runtime/architecture. That being said I suspect for 90% of orgs they'll never hit that boundary.

clojure+u...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/beca47df-4227-4d08-a71f-0e98e34a7acf%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Nathan Fisher

 w: http://junctionbox.ca/

[Sean Corfield]

Two reasons:

1. When I searched for ‘annotations in Clojure’ I pretty much came up empty – so I didn’t know you could do that sort of stuff with gen-class 😊
2. I was very focused on adding annotations to specific functions that I wanted New Relic to trace so I didn’t think much about class level stuff
3. We’ve since dropped the New Relic tracing stuff from our code (because we got the data we needed and moved on)

 

Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

 

---

From: clo...@googlegroups.com <clo...@googlegroups.com> on behalf of Chris Nuernberger <ch...@techascent.com>
Sent: Friday, June 21, 2019 12:40:28 PM
To: Clojure
Subject: Re: Java Interop on steroids?

 

clojure+u...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/beca47df-4227-4d08-a71f-0e98e34a7acf%40googlegroups.com.

[Sean Corfield]

I got the impression this was the primary reason for Storm’s rewrite:

 

While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute.

 

Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

 

---

From: clo...@googlegroups.com <clo...@googlegroups.com> on behalf of Nathan Fisher <nfi...@junctionbox.ca>
Sent: Friday, June 21, 2019 4:17:43 PM
To: clo...@googlegroups.com

Subject: Re: Java Interop on steroids?

To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/CA%2Bc2dWdFKexJEjAtmt2qt2JD%3DUo5FvxeM1kZATBSwU5K6n9b1Q%40mail.gmail.com.

[Matching Socks]

By "generic type information", you mean the X in List<X> ?

[atdixon]

Here is my problem, distilled. This code should tell the full story:

static class Apple<K, V> {}

Apple<String, Integer> a = new Apple<String, Integer>() {};

Type[] x = ((ParameterizedType)a.getClass().getGenericSuperclass())

.getActualTypeArguments();

// x is a Type array containing String, Integer

HOWEVER, via Clojure `proxy`, I don't have a way to tell it which type arguments to use--

(proxy [Apple [String Integer]] []) ;; something like this is not supported

Certain frameworks, however, consult these type arguments to govern their own dynamic behavior.

Having Clojure's proxy support specifying type arguments would allow for this not uncommon Java interop need.

[Nathan Fisher]

They did cite a significant performance boost as a side effect.

To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/MWHPR2201MB1134DA8985E3FEBE75B43B63F4E60%40MWHPR2201MB1134.namprd22.prod.outlook.com.

For more options, visit https://groups.google.com/d/optout.

[Aaron Dixon]

That is likely more to do with doing a full rewrite with 5+ years of learned experience under their belt than any insurmountable performance issue by Clojure. 

(Storm was mostly Clojure from day one but always had Java code in the mix, as well; in other words, they've always had Java if they had needed it... so crediting their perf gains to the PL switch is suspect imo.)

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/Yi922qYALg8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/CA%2Bc2dWeiUnbb23R1fwaEE8PCx7R4gVYdb_6r0SwRTUcCpVhPsg%40mail.gmail.com.

[Chris Nuernberger]

Nathan,

For the subset of problems for which clojure has insufficient power to optimize well, I think a good percentage of those would be best off solved by numeric languages like numpy or a numeric compiler like TVM as this gives you a pathway to much more scalability and performance in the end.  In short, they aren't best solved by Java either but best solved via translation to, execution in, and then translation back from a numeric form.  Aided by 4 years of understanding the problem this sort of transformation should definitely be possible.

​

As for being able to contribute without knowing clojure I think this will probably overall lead to an ossification of design over time but the larger set of involved people is definitely a great thing in and of itself.  Potentially the audience for Storm is mainly other jvm languages.

The best person I found who learned clojure fast was already very good at python and used Koans. I think it should be reasonable to stay within some of the extent of the language described in things like the Koans for the majority of the codebase and still provide a pathway for new people.

That's a lot of thinking for a weekend :-).  Thanks for the post it was interesting!

Chris

​

To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/beca47df-4227-4d08-a71f-0e98e34a7acf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Didier]

> They did cite a significant performance boost as a side effect.

I think it isn't very clear from the wording. They didn't just rewrite it in Java, they also changed the architecture:

> Storm 2.0.0 introduces a new core featuring a leaner threading model, a blazing fast messaging subsystem and a lightweight back pressure model

So it is hard I think to do an apples to apples comparison. In my opinion, a system like Storm will fundamentally be more limited by its architecture in performance than the language it uses.

That said, even if I think the performance improvements are probably mostly due to architectural changes they also made. It shouldn't come as a surprise that Java would be faster than Clojure in most cases. I don't want to make false pretenses. Clojure and Java are not equal in semantics, and Java's mutable, eager and object grouped methods semantics are almost always going to be more performant. Clojure makes a trade offs of performance and memory for simplicity.

When people say Clojure can match Java in performance, it almost always implies using escape hatches and changing the semantics back to imperative code. The good news though, when you use Clojure's semantics, you benefit in simplicity and the performance impact is marginal, so it is still fast enough for almost all use cases.

Now, back to type annotations. I really don't think Storm 2.0 performance improvements were due to usage of reflection that hadn't been addressed from Clojure. But who knows.

Do the framework you're talking about do static analysis of the types? Because generic types are erased at runtime, so there wouldn't ever be a way for proxy to set them in.

[atdixon]

> Do the framework you're talking about do static analysis of the types? Because generic types are erased at runtime, so there wouldn't ever be a way for proxy to set them in.

They aren't entirely erased. They're erased from the code, but Java compilers are obligated to emit generic type signatures (as metadata) in the compiled class file.

The frameworks I mentioned reflect on these generic type parameters *at runtime* to drive their dynamic behavior. 

More information here:

https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html#getGenericSuperclass--

https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-4.html#jvms-4.7.9.1

Searching back through this group, I found that in 2012 someone proposed a patch to Clojure supporting this in `proxy` (link: https://groups.google.com/d/msg/clojure/Xv1pKATfP0c/od_uwpHlNhMJ) but it looks like it never made it in.

[Didier]

Oh, I didn't know about that, pretty funky stuff.

[Chris Nuernberger]

eglue,

1.  I think this is a great idea if it is really necessary.  I would be in favor of a reify++ alone to simplify things.  I find reify amazing at code compression and heavily use it via type specific macros to implement interfaces that for instance support a particular primitive type.

2.  Is a possible workaround to define java interfaces that implement the type specific generic interfaces and then reify those explicitly or is the set of possible interface specialization types unknown a-priori?

3.  The case where something is unbounded or unknown a-priori I would think would often end up with a java class as on of the specializations.  In this case, regardless of the cause, one answer might be an upgraded reify pathway. 

4.  Are these perhaps cases where you can create just a little bit of java as a generator somehow to generate the interface you need to reify?

I would personally find reify a much nicer pathway than calling clojure vars from java. 

I also think there could be low hanging fruit (or just good unknown libraries) in the pathway for calling clojure vars from java.

Interesting problem (at least to me), thanks!

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+u...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

[Chris Nuernberger]

5.  If you need a concrete class definition that then implements a set of type specific interfaces this would seem to fall into a category of gen-class assuming you could specify the interfaces with type specifications.  I can't immediately place a way to do this with anything mentioned above.  It may be then important to be able to override Object methods but maybe you already can with gen-class.

[atdixon]

I'm glad someone else is thinking on this too!

#2 - For my case at the moment (Apache Beam), I believe we will always know the types in advance so using a Java class is workable but of course a (proxy++) would be ideal. Beam asks for us to extend abstract generic class so we must use (proxy). It also asks for our instances to be Serializable (again, proxy explicitly refuses to help here but I believe this, too, should be surmountable, without the security implications of that link). In any case, what we do looks like this:

// pseudo-ish code
public class MyFn extends BeamFn<Integer, String> implements Serializable {
   public MyFn(Var fn) { this.fn = fn; }
   @Override public String invoke(Integer input) { return (String) fn.invoke(input); }
}

On the Clojure side:

;; pseudo-ish code

(defn my-fn [^Integer val] ...return a string...)

...
   (register-beam-step (MyFn. #'my-fn))
...

Note how we pass the Clojure function as a Var; this is b/c Beam wants to send the function over the wire. IFn is not serializable; Var is however and can be resolved back to Clojure function on the other end during deserialization.

Now of course this whole dance could be eliminated with a specialized library that included a proxy++ function that included the ability for clients to specify generic type parameter values and serialization support (the mechanics of which would need ironing out but I think should be possible.)

And to your point about types not known at runtime... this proxy would support that use case on-the-fly, which opens a bunch of possible interesting options, as well.

clo...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clo...@googlegroups.com.

[Chris Nuernberger]

OK that makes more sense, BeamFn is not an interface. I imagine they argued about that decision at some point.

In your example, MyFn looks to be a general class as it can work with any clojure var. Do you have a package with many of these type of stub classes defined? I mean, you could name it BeamFnIntegerString thus encoding the specializations of the baseclass into the type. One of my questions was about the range of types that end up specializing the BeamFn<>.

Great link about the proxy attack, that is very interesting.  I want to dive into that one a bit more.

clojure+u...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/6c1a93e7-b25b-4199-a41a-9308b231ffc1%40googlegroups.com.