Advice: .Net, REST, batching, locking, performance.....

[baldric]

hi all

I wanted some advice on a system we're developing, using Neo4j. We are
developing in C#.net, and making good use so far of Neo4jClient
(http://hg.readify.net/neo4jclient/wiki/Home) - so we're basically
using the REST Api.

Our system, in a nutshell, has a particular components that I need
advice on:
The system will periodically read information from the graph, do a
complex set of calculations within our code (i.e. external to the
graph), and then persist the result sets back to the graph.
Reading the data out of the graph looks like it will be quite quick
(thanks to some nifty Gremlin, a little chunking of the results, and
some parallelisation), but it is the writing it back that concerns me
- the data would likely have doubled or tripled in size (i.e. if we
read 20000 nodes, we might commonly write back about 45000...)

Because we are working in .Net, we are fairly limited in the
approaches we can take - as in, as far as I can see, we can only do
things over the REST Api.
Up till now, I've been running simple hacky tests to try out the
batching api over REST, with reasonable success.
However, I came across a problem while trying to parallelise my
batches, which I'm pretty sure was because of deadlocks (struggling to
get my logs working - but thats a topic for another day). Firstly, is
this a normal problem, or am I not reading the signs correctly? (For
info, the nodes I insert would probably each have 4 or 5 edges, and at
least 2 or 3 of those edges are to nodes which in turn will have a LOT
of edges... which means the graph is quite dense. In other words, the
likelihood of creating 2 edges on different threads, simultaneously,
where those edges both connect to the same node, is pretty high)

I ended up writing some truly horrible code to batch sets of data that
are completely disconnected, to prevent deadlocks from occurring when
running in parallel, which worked quite well (although my algorithm
itself now takes time to run locally)

I guess this is a fairly broad, open-ended question right now: does
anything I've said jump out as "wrong! wrong! wrong!"? Am I missing
something about batching over REST, and deadlocks, and
parallelisation? Have I completely missed an option available to me?
Any other .net guys out there who have ideas, specifically in the area
of how to get a pretty heavy, transactional system to perform
well...over REST?

[Peter Neubauer]

Kevin,
I think what you are seeing is not a special case of REST but the
effect of trying to modify supernodes from different threads. That is
of course a consistency problem and might result in waits and locks.

For the .NET case, there are decent drivers out there, see
http://docs.neo4j.org/chunked/snapshot/tutorials-rest.html for the
different options.

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Neo4j 1.6 released                 - dzone.com/6S4K
The Neo4j Heroku Challenge   - http://neo4j-challenge.herokuapp.com/

[Mohammad Habbab]

I have the same concern as baldric. My Application uses Neo4jClient library, which doesn't support neither batching nor manual-locking (aquiring locks). Could someone please explain a scenario where a dead-lock could occur due to REST API usage. and how can i prevent it ? Isn't neo4j supposed to assure that REST Calls don't cause dead-lock by sacrificing response-spead and enforcing (aquiring) write-locks to all write-operations ? I couldn't find a detailed discussion about this .. so it would be very helpful If explained so i can avoid risky-design decisions !
 
Thanx.

[Tatham Oddie]

Habbab,

> My Application uses Neo4jClient library, which doesn't support neither batching

We use batching internally, but just haven't exposed a nice generic API for it. You're welcome to do so. ;) If you use overloads like the one that lets you create a node and add relationships in one method call, it will be sent as one REST call.

> nor manual-locking (aquiring locks)

This isn't really something that our client library lacks, considering it's not exposed by the REST API to my knowledge: http://docs.neo4j.org/chunked/stable/rest-api.html



-- Tatham

[Neil Barnwell]

When you say "we use batching internally" do you mean in neo4jclient? I might look at that and see what it would take. :)

[Tatham Oddie]

> When you say "we use batching internally" do you mean in neo4jclient?

Yes. If you call IGraphClient.Create(node, relationships, indexEntries), we run all of that in a single batch.

http://hg.readify.net/neo4jclient/src/fe7ad58cfb70/Neo4jClient/GraphClient.cs#cl-111

The code is pretty easy, but the first step is working out a nice evolution of the API to facilitate exposing this. I'd generally like to move to a TPL-style approach (System.Theading.Tasks) so that we can support async, await and batching via a commonly recognised API model.

If you're interested in helping, I'd recommend the first step be creating a wiki page at http://hg.readify.net/neo4jclient/wiki/Home with some ideas about what the API might look like. That way we won't waste any coding efforts down the wrong path.


-- Tatham