Batch weirdness with current py2neo

[Alan Robertson]

Hi,

Twice I've tried to convert to py2neo 2.x. Both times I ran into this problem. The first time I had to revert the 2.0 changes and put out the release anyway. This time I'm sticking with it until I get past it somehow...

The symptom is that I get this result from a transaction:

BatchError: Batch job 2 failed with NotFoundException
Node 34950 not found

I can assure you that the number 34950 does not appear anywhere in my code (I put in lots of debug). The transaction consists of three things:
    Create a new node
    Relate the newly created node to an old node (which has node id 35031).
    Insert the newly-created node into a legacy index.

This all works with the 1.6 series of code.

Here's a little of the debug from the transaction:

==== Performing batch.create(0:({dateofbirth:"unknown",domain:"global", firstname:"Annika", lastname:"Hansen", nodetype:"Person",time_create_iso8601:"2015-08-17 17:23:50", time_create_ms:1439832230360})) - for new node

==== RELATIONSHIP(0 -[IS_A]-> (n35031 {domain:"metadata",name:"Person",nodetype:"CMAclass"}))

==== add_to_index_or_fail: node 0; index Index(Node, u'http://localhost:7474/db/data/index/node/Person')("Hansen","Annika")

Here's the explanation of the debug output:
The 0 in "batch.create(0:(..." means that this newly-created node has index 0 in the transaction (as it should).
The "RELATIONSHIP(0 -[IS_A]-> (n35031 {..." means that we created a relationship between a node 0 and a Py2neo Node whose id is 35031.
The "add_to_index_or_fail" line means we called the legacy.add_to_index_or_fail function on node 0, putting it into the Person legacy index with the values "Hansen" and "Annika" as the index values.

This is all exactly the same code which worked before. I mainly added "legacy." to a bunch of calls, changed a few imports and added more debug when it didn't work.

This looks a bit like a bug in py2neo to me.  Suggestions on how to proceed?

--

Alan Robertson / CTO
Al...@AssimilationSystems.com / +1 303.947.7999

Assimilation Systems Limited
http://AssimilationSystems.com

[David Fauth]

Alan,

I'll send a note to Nigel Small and have him take a look.

Dave

[Alan Robertson]

I turned on Nigel's debugging. This is the error this time (same but different node number mentioned)

BatchError: Batch job 2 failed with NotFoundException

Node 35832 not found
Here's what Nigel's debugging says:

> Sending batch request with 3 jobs
> {0} POST node {"domain":"global","nodetype":"Person","firstname":"Annika","lastname":"Hansen","time_create_iso8601":"2015-08-18
20:28:46","time_create_ms":1439929726819,"dateofbirth":"unknown"}
> {1} POST {0}/relationships {"to":"node/35910","type":"IS_A"}
> {2} POST index/node/Person?uniqueness=create_or_fail {"value":"Annika","key":"Hansen","uri":"{0}"}
< Received batch response for 3 jobs
< {0} 201 <Node graph=u'http://localhost:7474/db/data/' ref=u'node/35917' labels=set([]) properties={u'domain': u'global', u'node
type': u'Person', u'firstname': u'Annika', u'lastname': u'Hansen', u'time_create_iso8601': u'2015-08-18 20:28:46', u'time_create_
ms': 1439929726819, u'dateofbirth': u'unknown'}>
< {1} 201 <Relationship graph=u'http://localhost:7474/db/data/' ref=u'relationship/60444' start=u'node/35917' end=u'node/35910' t
ype=u'IS_A' properties={}>
It points to the problem as being with the index/node/Person update.

> Twitter Linkedin skype
> --
> You received this message because you are subscribed to the Google Groups "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--

Alan Robertson / CTO
Al...@AssimilationSystems.com / +1 303.947.7999

Assimilation Systems Limited
http://AssimilationSystems.com

Twitter Linkedin skype

[Nigel Small]

Hi Alan

So IIRC, there are outstanding faults with the uniqueness modes that underpin the add_to_index_or_fail function. I forget the exact nature of these but I put a warning in the py2neo docs about using them:
http://py2neo.org/2.0/legacy.html#py2neo.legacy.LegacyWriteBatch.add_to_index_or_fail

If memory serves (which it often doesn't) the issue was related to this:
https://github.com/neo4j/neo4j/issues/906

Going forward, I'd advise moving to labels and schema indexes rather than using legacy indexes. Additionally, Cypher transactions are generally superior to the old HTTP batches which have some functional limitations. Both of these features (schema indexes and Cypher transactions) are now mature and still in active development so will often provide better results.

Cheers

Nigel

[Alan Robertson]

Thanks again for your response.

I'm making good progress in getting going with your newer APIs. Unfortunately, I had to go on and do a bunch of work while waiting for the response, and now after I get all this working, I'll have a substantial merge job ahead of me because  I had to touch a good bit of the same code for the new features I added to my code :-(.

As soon as that's done and working, then I can switch over to Cypher transactions.

stream(statement, parameters=None, **kwparameters)

Execute the query and return a result iterator.

Parameters:	statement – A Cypher statement to execute. parameters – A dictionary of parameters.
Return type:	py2neo.cypher.RecordStream

OK. I see it returns a RecordStream. Unfortunately it doesn't document what's in the RecordStream :-(

What's fields are in the Records in the RecordStream?

Will I get one Record per transaction element?

What do those Records look like for various kinds of Cypher queries?

When I add a node where do I find the newly-created Node object?

When I add a relationship, where do I find the newly-created Relationship object?


    Thanks!

    -- Alan

[Alan Robertson]

Merge all complete. Next release of my code will require the new version of py2neo.