Clojure run time isolation with the ability to share data structures

[Giri Anantharaman]

Hello

I am newbie to Clojure. Been using for a couple of months now. With that disclaimer out of the way, we have use-cases where we would like multiple clojure run times with in the same JVM  and have these run times share clojure data structures among themselves. We attempted to solve this problem with some help from OSGI (since our use-case was mostly OSGI bundles running clojure). This post to share our approach and solicit feedback / comments from others.

- OSGI provides individual bundle class loaders for each OSGI bundle. So given this running multiple clojure run times inside a single JVM is straight forward. However in order to share clojure data structures across these run times, several data structure interfaces from clojure.lang (IMap etc) must be loaded by a common class loader. However clojure.lang package also includes other classes, implementations and not to mention the clojure runtime (RT.java) and compiler (Compiler.java)

- Given the fact that clojure uses Thread Context Class Loader (TCCL) to load its classes, plus given that OSGI bundles can export packages, we were able to create a class loader that loaded all clojure.lang interfaces using a common class loader and things like RT / Compiler etc from individual bundle class loaders. Thus we have run time isolation with data structure sharing.

However we ran into some issues with sharing clojure.lang.IKeywordLookup. This interface references "clojure.lang.Keyword". If one were to load "clojure.lang.IKeywordLookup" at some point it would also load "clojure.lang.Keyword" using the same class loader. "clojure.lang.Keyword" is also referenced else where in clojure (clojure.core, other implementation classes in clojure.lang). Therefore "clojure.lang.Keyword" would be loaded by different class loaders and this causes a java class loader linkage error, since while linking say a class A, that uses Keyword by itself and IKeywordLookup, JVM does not know which version of Keyword class should be used to resolve the class fully.

We worked around it by NOT sharing IKeywordLookup across run times. We can still share maps, vectors, lists etc. So this gets us most of what we want. However i was wondering if the issue we faced with IKeywordLookup / Keyword fixable in some other way. In the absence of a IKeyword interface we could not think of any.

Would be happy to hear what others think.

-Giri

[Arnout Roemers]

Hi Giri,

At my company, we also desired multiple, isolated Clojure runtimes in one JVM instance, sharing Clojure data structures/types. So indeed, we also created special ClassLoaders in order to do this. We had the same problem with some data types being tied to the Clojure runtime somehow. Even more, there is also the problem with package protected fields in some of those data structure/type classes, causing issues when an isolated clojure.lang.RT tries to access a commonly loaded data structure/type. 

Our solution: patch the Clojure runtime. For us, the result is that we can share almost anything, including keywords, functions and vars. Our patched Clojure can be found at http://github.com/touch/clojure. It is not a tidy patch (as the diff shows many whitespace/indent changes), but maybe it is of some use to you.

In the long run, we plan to make a tidy patch, and also release our Clojure runtime container software as an open source project.

-Arnout