Does OSGi make sense for Clojure?

[Todor Boev]

Hi,
I am an OSGi enthusiast. Lately I have been looking at scheme and
clojure. I can't help but wonder if there are any genuine benefits
clojure can get from being a full OSGi citizen. It seems to me that
OSGi is to statically compiled Java what the REPL is for Clojure.
Except the REPL is more powerful. Currently OSGi seems like a nice
interoperation environment for clojure and the other JVM based
languages. This alone is a sufficient motivation to make clojure OSGi
compatible, which it currently is not because it's internal class
loading does not play nice with OSGi's own dynamic class loading.

The devilish question however is: if all our code was in clojure would
running withing OSGi bring any genuine benefits? Does OSGi as a
dynamic runtime for the JVM provide anything that the REPL does not
already have?

Cheers,
Todor

[Stuart Sierra]

It's possible to modify Clojure to run under OSGi (search the list
archives) but fundamentally they don't fit. OSGi assumes that it has
sole control of class loading. But Clojure needs its own
classloader. To make them cooperate, I think you would need to
integrate Clojure with the OSGi container itself.

-SS

[Jeff Heon]

There is this project going on:
http://wiki.github.com/romanroe/ogee

[Mark Derricutt]

I do have a novel idea to work around this but I've yet to put my theory to code and publish it.

The problems I had when loading clojure based bundles into OSGi was that after unloading, the classes remained loaded by the bundle providing RT.

The basic idea was to add a bundle listener to the OSGi bundle, that when uninstalled, unloads the namespace from RT (unfortunately I can't seem to spot that API anywhere anymore).

mark

[Roman Roelofsen]

@Mark: I doubt that this will work. Unloading a namespace would mean
to remove a class from a classloader and this does not work in Java.
OSGi handles this by removing the bundle's classloader completely.

@All: The huge added value that OSGi still has is multi-version
support. Lets say I want to use clojure.contrib version 1.1 and and
one of my other libraries wants to use clojure.contrib version 1.0.
This clearly does not work in Clojure currently since all
Clojure-based classes/namespaces live in one classloader. So currently
Clojure suffers from the same JAR-hell issues that are already solved
by OSGi :-/ Also, OSGi's service layer could be a nice way to decouple
some parts of the application. Additionally, several projects exist
that provide "infrastructure features" (e.g. remoting, web server) and
they all use the service registry for this.

I tried several integration approaches in Ogee but non of them really
worked out well:

* Full Isolation*

We can fully isolate different Clojure Modules (== logical group of
jar file(s)) by giving each of them their own classloader that
contains the clojure runtime. The problem is that Clojure's startup
time is approx. 1 sec so if we install 50 modules it would take way
too long to start the application.

* Isolation with fine-grained sharing *

I tried to extract common not-runtime specific Clojure classes into a
shared classloader to allow the modules to interact with each other.
E.g. classes like Var, IFn, ... However, there is a circular
dependency between Var -> something -> RT -> something -> Var and the
RT clearly has to stay isolated. Trying to split this cycle gave me a
LinkageError or something like that. Also, Clojure seems to rely on
the ContextClassLoader and I worry that one module could accidentally
pollute a "foreign" Clojure runtime.


I am currently not sure if OSGi's module layer makes sense at all in a
dynamic language. In a statically-typed language like Java, the use of
classes is baked into the bytecode. Therefore, when class A uses class
B, the resolver has to link A's classloader to B's classloader even
before A is loaded. That makes it impossible for A to use B in
different version. However, this constraint does not apply for a
dynamic language like Clojure at all!

I think a more idiomatic way for Clojure would be something like this:

A namespace could optionally declare a version number:

(ns package.fancylib :version 2)

This would generate a class like "fancylib__init__$version2.class" or
so. When other namespaces require/use the fancylib namespace _without_
a version a number, the Clojure runtime could easily find the latest
version:

1) find all "package/fancylib__init__$version*.class" files
2) order them
3) return first entry

When other namespaces require/use the fancylib _with_ a version number
(require 'package.fancylib :version 2), the Clojure runtime could
easily infer the correct classname.

I guess it is important that each module has its own class loader and
that there is one chaining parent clojure classloader. The module
classloader could transparently do the namespace mangling.

Ogee is currently based on OSGi but all Clojure modules live in the
same classloader. Whenever the Clojure setup changes, Ogee
automatically restarts the whole Clojure runtime so at least you get a
bit more dynamic feeling. The focus is currently on good idiomatic
service layer integration and support for servlets, remoting,
configuration management, etc. Maybe Ogee can later implement
something like what I described above but for performance reason it's
obviously preferable that the Clojure runtime itself provides
something in the future.

Cheers,

Roman

2009/10/6 Mark Derricutt <ma...@talios.com>:

[Todor Boev]

There is rarely need to embed a runtime/framework into the OSGi
container itself. All that is needed is to refactor the Clojure
runtime to let OSGi do the class/resource loading. The goal would be
to get a central Clojure bundle and deploy application code into
separate bundles. For statically compiled code this is not a very hard
conversion. You just call Bundle.loadClass() on the application bundle
to get OSGi do the heavy lifting for you. Clojure however compiles
code at runtime. Currently the OSGi class load sequence does not have
a slot for pluggable compilers. If it did than the bundle ClassLoader
would call out to the clojure compiler giving it a chance to convert
resources found in the bundle into byte codes. Since such support is
not available the next best thing is to compile the clojure code
*before* it is reaches OSGi. This technique is not yet widely used but
I expect it do become ever more popular. The clojure runtime must
register a "clojure:" URL handler with the OSGi continer. It can than
intercept bundles installed from locations like "clojure:http://
acme.com/device.jar". The clojure runtime will compile the source code
found inside and literally generate an OSGi bundle on the fly.
Magic! :)

I am not sure if the clojure runtime can hotswap individual functions.
In case it can this OSGi conversion will increase the granularity of
dynamic deployment/hotswap from a function to a bundle. The REPL
itself can also be deployed as a bundle. It can even plug into one of
the generic text console manager bundles. The Felix TUI bundle for
example.

In summary - I am relatively confident Clojure can be converted into a
set of bundles that run against the generic OSGi API. If this will be
useful is a different question :)