Neo4j and OSGi -- let's make peace :-)

[Hendy Irawan]

Dear Neo4j developers,

I'm aware this has been discussed a few times before so I'm hoping
this issue can be resolved. :)

I've been struggling with neo4j in OSGi for over 12 hours now, and
have been doing OSGi since 2007, and I'm in love with Neo4j (see
http://scala-enterprise.blogspot.com/2012/02/scala-source-code-and-density.html
), so I'm quite motivated to get this sorted.

For a start, I think the core issue isn't with classpath per se but
with dependency injection. So is it possible to get an API like this:

graphDb.setIndexProvider( new LuceneIndexProvider() );

Regardless of OSGi, this would bring the most flexibility and just
sidestep the whole ServiceLoader problem.
This would be usable in Spring, Guice, CDI, JBoss Modules, etc.

With OSGi it's actually safe to assume that Blueprint is around. Since
a Blueprint config is just an XML file it will give the most
flexibility while freeing Neo4j from any org.osgi dependencies.

Neo4j can have a Blueprint config like this inside: (not tested)

<service auto-export="interfaces">
<bean class="org.neo4j.kernel.GraphDatabaseFactory">
<property name="indexProvider"><reference
interface="org.neo4j.kernel.graphdb.index.IndexProvider"/></property>
</bean>
</service>

Now an OSGi developer can just get the factory using :

<reference id="graphDbFactory"
interface="org.neo4j.kernel.GraphDatabaseFactory"/>

and never worry about any classpath stuff.

Another option is to use SPI-Fly but unless we really want to use
ServiceLoader I really think it's much better to just rely on
dependency injection.

Let me know what you think.

Hendy

[Toni Menzel]

Hi Hendy,

+1 on the SPI part.

Regarding some class loader assumptions neo4j makes, you also checked the osgi- friendly branch in the neo4j codebase ?

What it ultimately needs ( think) is a proper, not so simplistic, server re-implementation using proper class loader isolation and a pluggable service api.

 - using osgi sounds almost too natural to the few of us ;)

But i can think of making the server actually runtime agnostic, so one can chose OSGi but also could live in a classic variant using a monotholic class loader.

With core Neo4j on OSGi i am quite confident. Don't want to mention the trouble with SDN right here. Its a different beast. Would love this whole thing implemented without even mentioning the word Spring..

/Toni

--
Toni Menzel Source

[Hendy Irawan]

Hi,

Thank you Toni.

I did check out the bundle-friendly branch, then got to sleep while waiting for it to build ;-) I'll use it as base.

Can you clarify what do you mean by server re-implementation ? I thought it was just a classloading problem when neo4j tries to find the IndexProvider ? So is there bigger / other problematic code points in neo4j ? I'm hoping you can point what they are.

To continue the graphDb factory implementation I outlined before, I'll also try to add @Inject in some places so that it can be used inside CDI or Spring (can someone familiar with Spring confirm this?) can simply do something like:

@Inject GraphDatabaseFactory graphDbFactory;

...

GraphDatabaseService graphDb = graphDbFactory.createEmbedded("/some/path");

GraphDatabaseService graphDb2 = graphDbFactory.createEmbeddedReadOnly("/some/path");

and have the right dependencies provided automatically, since Neo4j's dependencies are also managed by CDI :

class GraphDatabaseFactory {

  @Inject ExtensionLoader extensionLoader;

This has the added benefit of not having to call shutdown(), I think we can just mark it as @PreDestroy. Same goes for OSGi Blueprint, we can mark it as destroy-method. I'm not sure why OSGi Blueprint doesn't support @PostConstruct/PreDestroy annotations, maybe I should ask someone. ;)

Indeed this means javax.inject interfaces has to be on the classpath, but it doesn't require an actual CDI implementation.

That would be less memory-leaky.

For those not having the luxury of DI frameworks nor OSGi, for example inside unit tests, they can do it the POJO way:

factory = new GraphDbFactory( new Jdk6ExtensionLoader() );

db = factory.createEmbeddedDatabase(.....);

or even do it the hard way:

db = new EmbeddedReadOnlyGraphDatabase(....);

db.setIndexProvider( new LuceneIndexProvider() );

This way we can scrap all the different "loaders" inside neo4j (for the currently disabled OSGiLoader, can delete it) , neo4j provides only default loader (JDK 6 ServiceLoader) and for the rest, either use the "native" DI via @Inject or Blueprint, or let them write their own ExtensionLoader implementation.

What do you guys think. Is this a good idea?

Hendy

[Peter Neubauer]

Guys,
There has been a refactoring of all the dependency injection by Mattias and Rickard very recently. This unifies the kernel modules and other extensions and as I see it opens the way to do something like DI friendly from the outside even for index providers. Before doing anything, check out current master...

Rickard, git any details?

Send from a device with crappy keyboard and autocorrection.

/peter

[Hendy Irawan]

I tried to compile master but doesn't work :(

[ERROR] Specifications.java:[54,15] cannot find symbol

symbol  : method and(java.lang.Iterable<java.lang.Object>)

location: class org.neo4j.helpers.Specifications

[ERROR] Specifications.java:[64,15] cannot find symbol

symbol  : method or(java.lang.Iterable<java.lang.Object>)

location: class org.neo4j.helpers.Specifications

[ERROR] Specifications.java:[74,17] incompatible types

found   : org.neo4j.helpers.Specification<java.lang.Object>

required: org.neo4j.helpers.Specification<T>

[ERROR] Specifications.java:[150,33] cannot find symbol

symbol  : method or(java.lang.Iterable<java.lang.Object>)

location: class org.neo4j.helpers.Specifications

[ERROR] Specifications.java:[178,33] cannot find symbol

symbol  : method and(java.lang.Iterable<java.lang.Object>)

location: class org.neo4j.helpers.Specifications

[ERROR] Iterables.java:[342,63] <FROM,TO>map(org.neo4j.helpers.Function<? super FROM,? extends TO>,java.lang.Iterable<FROM>) in org.neo4j.helpers.collection.Iterables cannot be applied to (<anonymous org.neo4j.helpers.Function<java.lang.Iterable<T>,java.util.Iterator<T>>>,java.lang.Iterable<java.lang.Object>)

[INFO] 6 errors 

[Peter Neubauer]

Are you on Java7? Try JDK 1.6

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Neo4j 1.6 released                 - dzone.com/6S4K
The Neo4j Heroku Challenge   - http://neo4j-challenge.herokuapp.com/

[Hendy Irawan]

I am on JDK 6. Here's what Maven says:

ceefour@annafi:~/vendor_dev/neo4j/kernel$ mvn compile -DskipTests -e -X

Apache Maven 3.0.3 (r1075438; 2011-03-01 00:31:09+0700)

Maven home: /opt/apache-maven-3.0.3

Java version: 1.6.0_23, vendor: Sun Microsystems Inc.

Java home: /usr/lib/jvm/java-6-openjdk/jre

Default locale: en_US, platform encoding: UTF-8

OS name: "linux", version: "3.2.1-030201-i7", arch: "amd64", family: "unix"

I tried making these changes but no effect :

        <artifactId>maven-compiler-plugin</artifactId>

        <executions>

          <execution>

            <id>default-compile</id>

            <configuration>

              <compilerArgument>-proc:none</compilerArgument>

              <includes>

                <!--<include>org/neo4j/kernel/impl/annotations/**</include>-->

              </includes>

            </configuration>

          </execution>

...

        <artifactId>build-helper-maven-plugin</artifactId>

        <executions>

          <execution>

            <phase>generate-sources</phase>

            <goals><goal>add-source</goal></goals>

            <configuration>

              <sources>

                <source>target/generated-sources/version</source>

<source>org/neo4j/kernel/impl/annotations/**</source>

              </sources>

            </configuration>

          </execution>

        </executions>

[Peter Neubauer]

Could you try with Oracle JDK instead of OpenJDK please?

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Neo4j 1.6 released                 - dzone.com/6S4K
The Neo4j Heroku Challenge   - http://neo4j-challenge.herokuapp.com/

[Hendy Irawan]

Found the problem. It's the Java heap.

giving this:

export JAVA_OPTS='-Xms384M -Xmx512M -XX:MaxPermSize=256M'

makes the build happy.

Why the OpenJDK compiler doesn't just report an OOM is truly beyond me. Should I report this to OpenJDK devs? (Although I'm expecting the reply "just use OpenJDK 7")

[Hendy Irawan]

I think there should be a note in the README to:

- make sure to use JDK 6

- make sure to enable sufficient memory for Maven + javac

If you want, I can make that change in my fork and make a pull request.

FYI, OpenJDK 7 also cannot compile neo4j-kernel, with or without the memory tweak.

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile (default-compile) on project neo4j-kernel: Compilation failure

Failure executing javac, but could not parse the error:

An exception has occurred in the compiler (1.7.0_147). Please file a bug at the Java Developer Connection (http://java.sun.com/webapps/bugreport)  after checking the Bug Parade for duplicates. Include your program and the following diagnostic in your report.  Thank you.

java.lang.AssertionError: Missing type variable in where clause T

        at com.sun.tools.javac.util.RichDiagnosticFormatter.unique(RichDiagnosticFormatter.java:234)

        at com.sun.tools.javac.util.RichDiagnosticFormatter.access$100(RichDiagnosticFormatter.java:67)

        at com.sun.tools.javac.util.RichDiagnosticFormatter$RichPrinter.visitTypeVar(RichDiagnosticFormatter.java:384)

        at com.sun.tools.javac.util.RichDiagnosticFormatter$RichPrinter.visitTypeVar(RichDiagnosticFormatter.java:326)

I guess I should report it to OpenJDK.. since it's easy to reproduce (just get this particular commit -> 0cd0b459f13dcd4ca18017baf4aa3008e2acd7f9 )

[Peter Neubauer]

Yes,
please do Hendy!

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Neo4j 1.6 released                 - dzone.com/6S4K
The Neo4j Heroku Challenge   - http://neo4j-challenge.herokuapp.com/

[Hendy Irawan]

Done :-)

https://github.com/neo4j/community/pull/287

[Peter Neubauer]

Merged.
Thanks Hendy for the swift docs!

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Neo4j 1.6 released                 - dzone.com/6S4K
The Neo4j Heroku Challenge   - http://neo4j-challenge.herokuapp.com/

[Hendy Irawan]

You're welcome Peter.

I have more GOOD news.

After several hours of understanding neo4j internals, I am able to make the latest trunk work out-of-the-box with OSGi. Look ma, no superbundles! ;-)

I did little refactoring, so for neo4j end-users it should work just fine like before. I have lots more ideas for the refactoring, but for now I keep it conservative because my mission is just to make neo4j work in OSGi so I can continue my job, not to provoke a rehab. :)

To use Neo4j in OSGi using Blueprint is very easy :

<reference id="graphDbFactory" interface="org.neo4j.graphdb.GraphDatabaseFactory"/>

then :

val graphDb = graphDbFactory.createEmbeddedReadOnly(dbDir.getPath)

That's it! No more magic tricks. :)

I think this could still be improved because all DI frameworks including Blueprint has support for init and destroy methods. shutdown() is a natural target for destroy method.

Please review my changes at this branch: https://github.com/soluvas/neo4j/tree/soluvas-osgi


Hendy

[Peter Neubauer]

Man,
That is fantastic news! I think the rest of the team will be very happy to review a pull request with some of the superbundle tests (especially bare bones and Springs Data Neo4j) added.

I promise a T-Shirt.

Send from a device with crappy keyboard and autocorrection.

/peter

[Hendy Irawan]

Thank you very much Peter.

I hope the tests pass.

Anyway, this is how it looks like in Karaf :

karaf@root> list
...

[ 140] [Active     ] [            ] [   60] blueprints-core (1.1)

[ 141] [Active     ] [            ] [   60] blueprints-neo4j-graph (1.1)

[ 142] [Active     ] [            ] [   60] neo4j-jmx (1.7.0.SNAPSHOT)

[ 143] [Active     ] [            ] [   60] lucene-core (3.5.0)

[ 145] [Active     ] [            ] [   60] neo4j-graph-matching (1.7.0.SNAPSHOT)

[ 146] [Active     ] [            ] [   60] neo4j-graph-algo (1.7.0.SNAPSHOT)

[ 147] [Active     ] [            ] [   60] neo4j-cypher (1.7.0.SNAPSHOT)

[ 149] [Active     ] [            ] [   60] neo4j-udc (1.7.0.SNAPSHOT)

[ 154] [Active     ] [Created     ] [   60] neo4j-kernel (1.7.0.SNAPSHOT)

[ 158] [Active     ] [Created     ] [   60] neo4j-lucene-index (1.7.0.SNAPSHOT)

[ 159] [Active     ] [Created     ] [   60] satukancinta-dump_2.9.1 (1.0.0.SNAPSHOT)

karaf@root> ls

...
neo4j-kernel (154) provides:

----------------------------

org.neo4j.graphdb.GraphDatabaseFactory

org.osgi.service.blueprint.container.BlueprintContainer

neo4j-lucene-index (158) provides:

----------------------------------

org.neo4j.graphdb.index.IndexProvider

org.osgi.service.blueprint.container.BlueprintContainer

satukancinta-dump_2.9.1 (159) provides:

---------------------------------------

org.osgi.service.blueprint.container.BlueprintContainer

karaf@root> ls -u

neo4j-kernel (154) uses:

------------------------

org.neo4j.graphdb.index.IndexProvider

satukancinta-dump_2.9.1 (159) uses:

-----------------------------------

org.neo4j.graphdb.GraphDatabaseFactory

karaf@root> ls 154

neo4j-kernel (154) provides:

----------------------------

objectClass = org.neo4j.graphdb.GraphDatabaseFactory

osgi.service.blueprint.compname = graphDatabaseFactory

service.id = 319

----

objectClass = org.osgi.service.blueprint.container.BlueprintContainer

osgi.blueprint.container.symbolicname = neo4j-kernel

osgi.blueprint.container.version = 1.7.0.SNAPSHOT

service.id = 321

karaf@root> ls 158

neo4j-lucene-index (158) provides:

----------------------------------

name = lucene

objectClass = org.neo4j.graphdb.index.IndexProvider

osgi.service.blueprint.compname = luceneIndexProvider

service.id = 318

----

objectClass = org.osgi.service.blueprint.container.BlueprintContainer

osgi.blueprint.container.symbolicname = neo4j-lucene-index

osgi.blueprint.container.version = 1.7.0.SNAPSHOT

service.id = 320

karaf@root> ls -u 159

satukancinta-dump_2.9.1 (159) uses:

-----------------------------------

objectClass = org.neo4j.graphdb.GraphDatabaseFactory

osgi.service.blueprint.compname = graphDatabaseFactory

service.id = 319

As you can see, inside OSGi runtime, Neo4j behaves just like a proper OSGi citizen, exposing and consuming services, even between its own bundles (neo4j-kernel consumes the service of neo4j-lucene-index).

However, in the sources, there are no dependencies to OSGi at all. In fact, all the OSGi* classes can probably be removed. If this refactoring is continued, most classes/interfaces will reduce fat.

This is one abstract class that I've turned into an interface (to make it proxyable by DI frameworks, and another reason is there's no reason it should be an abstract class in the first place), now lost a lot of weight: (it's more POJO-ish as well)

public interface IndexProvider {

    IndexImplementation load( DependencyResolver dependencyResolver) throws Exception;

    public String identifier();

}

If the current code is good enough, the next step would be to re-enable the OSGi manifest headers. Then there would be no need of bundle wrappers. :)

Hendy

[Rickard Öberg]

Hey Hendy,

I was responsible for the code refactoring that now allows for
dependency injection. I'm now looking at the stuff you did, and will
clean that up and extend with a factory+builder pattern, with a
fallback on being able to do "new EmbeddedGraphDatabase" for legacy
purposes.

A key goal is definitely to be able to specify clearly what the SPI
is, and the refactoring was one step in that direction. Merging in
your stuff, with an additional builder step, will finish that, I hope.

/Rickard

[Hendy Irawan]

Hi Rickard,

Thank you.

I completely agree with the builder pattern. Supporting standard POJO programming model :

   factory -> set properties -> init -> destroy

 lifecycle, for the graph database (hopefully other parts too like index) will make it usable in any DI container. And reducing non-core functionality like the various helper utils e.g. Service class (IMHO it should be gone).

"new EmbeddedGraphDatabase" was not a good approach (especially since it immediately calls run() ), but I agree about supporting it for backward compatibility.

Hendy

[Rickard Öberg]

On Wed, Feb 29, 2012 at 11:46 AM, Hendy Irawan <ceefo...@gmail.com> wrote:
> I completely agree with the builder pattern. Supporting standard POJO
> programming model :
>
>    factory -> set properties -> init -> destroy
>
>  lifecycle, for the graph database (hopefully other parts too like index)
> will make it usable in any DI container. And reducing non-core functionality
> like the various helper utils e.g. Service class (IMHO it should be gone).

Agree. I did something similar to this already in my Reloaded experiment:
https://github.com/rickardoberg/neo4j-reloaded/blob/master/src/test/java/org/neo4j/graph/ImdbTest.java

So a fluent type builder pattern, with additional support for
supplying SPI level services, is what I'm looking at doing.

> "new EmbeddedGraphDatabase" was not a good approach (especially since it
> immediately calls run() ), but I agree about supporting it for backward
> compatibility.

Yes and yes.

/Rickard

[Hendy Irawan]

Hi all,

I just remembered one trick that I've just tested to work successfully on Kettle / PDI 4.2.1 :

If only one JAR needs the service (e.g. kettle-core), a quick hack is to put Fragment-Host on the JARs providing the service (e.g. kettle-engine and kettle-db).

I haven't tried this on neo4j, but I think it might work for "supporting" OSGi quickly in neo4j 1.6 (by fragment-hosting neo4j-lucene-index to neo4j-kernel).

Hendy

[Peter Neubauer]

Hendy,
Let's wait for Rickards additional refactoring coming through, and then set up the different scenarios in the neo4j-osgi repo. Maybe you could start with this already if you want?

Send from a device with crappy keyboard and autocorrection.

/peter

[Hendy Irawan]

Hi,

I was thinking for this for neo4j 1.6 maintenance version only, no major refacotirng changes.

I'll let you know how this goes.

In 1.7 because of we're refactoring anyway it'd be better to make it right, agree with Rickard here. :-)

An upside of having neo4j OSGi-friendly is usable in Eclipse IDE or Eclipse RAP.

Hendy