Lightsocket: a lightweight ZeroMQ socket server with a Blueprints module

[James Thornton]

Hello All -

For the last few weeks I have been working with WordNet (http://wordnet.princeton.edu/) and playing around with ZeroMQ (http://www.zeromq.org/), a high-speed message passing framework.

WordNet is a large database of words, and it includes lexical and semantic relationships between the lemmas (words) and synsets (synonym sets). I loaded it into Neo4j, along with all of the semantic and lexical relationships.

Originally I was going to load an RDF (http://semanticweb.cs.vu.nl/lod/wn30/) through SailGraph, but I decided to write a program to load it directly out of NLTK (http://www.nltk.org/) into Neo4j so that I could easily modify it if needed. This process also helped me understand WordNet's structure.

Loading over REST through the normal transitional API was loading at a rate of about 50 entries per second. This was taking too long, and I wanted to be able to reload it if we needed to change something without it taking hours. 

To speed up the process, I wanted to use the new Neo4j Batch Loader that Marko just created, but it's written in Java and NLTK is a CPython library that won't work with Jython. I decided to use this as an opportunity to experiment with ZeroMQ. And so I wrote Lightocket. 

Lightsocket is a lightweight socket server that Bulbs uses to plug into the JVM. But it doesn't require that you use use Bulbs or even an Python client -- you can use any language that has a ZeroMQ binding (http://www.zeromq.org/bindings:c). 

It's written in Jython (for now), and it was inserting ~3,000 entries per second from a single Python client on my dev server, and going up to 10K/sec with multiple clients. 

I modeled WordNet in Bulbs (http://bulbflow.com/) and created a library for interacting with it through Rexster. That library is called WordGraph, and I'll post it in the next few days. 

Lightsocket is on Github now (https://github.com/espeed/lightsocket).

I also used this as an opportunity to experiment with binary serialization and RPC protocols, primarily MessagePack. JSON serialization is one of the biggest performance killers, and so I wanted to see how much we could improve upon that by using MessagePack. 

Unfortunately the documentation is somewhat lacking and it's not a simple drop in with Jython. So right now Lightsocket is still using JSON, but I am continuing to experiment with it.

Soon I'll put up more detailed docs at (http://lightsocket.bulbflow.com/), but if you want to play around with it, look at the README on Github along with the comments in the example client and example resource.

- James

[Daniel Quest]

James,

Thats pretty darned cool! I was also looking at NLTK in python. Nice
to see some work going towards integrating that on the Graph Stack!

-Daniel

[Andreas H.]

Hey James,

that sounds great! I just started to build a Neo4j REST-Client in C++
but I doubted the performance.
I think, I will drop the idea and instead go the ZeroMQ way using the C
++-bindings for it.

Sooner or later I will definitely drop you a message about success or
problems I run into =)

-Andy

[James Thornton]

> I just started to build a Neo4j REST-Client in C++ 
> but I doubted the performance. 
> I think, I will drop the idea and instead go the ZeroMQ way using the C 
> ++-bindings for it.

Note that Lightsocket is not a production server -- it's very much an experimental utility at this point.

I went looking around to see if there was a ZeroMQ-based Java server container (like a Jetty or Tomcat) or if anyone had modified one to run as a ZeroMQ server, but I couldn't find much on it. 

From what I have gathered, it seems the way to add ZeroMQ support in Rexster would be to create a JCA adapter (http://en.wikipedia.org/wiki/Java_EE_Connector_Architecture). 

See http://stackoverflow.com/questions/2154490/an-ear-java-ee-application-which-listen-to-a-socket-request

Does anyone have any insight on this?

- James

[TheSweetlink]

That is really cool James. I'm happy to see bulbs making great
progress like this. Keep up the slick work.

-David

[Andreas H.]

Hi!

I took James' code and after struggling a bit with Java, I had it running.

From there, I built my own C++-Client that now talks to James'  Jython Server via ZeroMQ.
So I finally have a way to interact with a Neo4J database from my C++-programs without using HTTP.

If anyone is interested, the sources can be found in a branch of a fork of James' Google Code repository:
https://github.com/Squelsh/lightsocket/tree/squelsh
Don't hesitate to contact me with questions / enhancements / ideas...

I still have to create a more useful makefile and fix a small ZMQ bug, but it runs (:

Hope that helps someone. I still can't believe, that I should be the only one using C++ AND Neo4j

- Andy

[Arun Koshy]

_ awesome _

[SDen]

Your developments are very useful.
I had one project in our laboratory this year which deal with data
well described by graphs.
But we had to use embedded variants of relational db engines.

Soon I will return to that problem again and your solution will be
very useful.

So C++ still alive ;)

Cheers.

Denys.