Re: py2neo 1.6.4 doesn't work with Neo4j 2.1.2 - "Cannot find node..."

[Alan Robertson]

Hi Nigel,

Many thanks for your quick reply!

I can reproduce it with what I think is a smallish environment.

If you download my code from here:  http://hg.linux-ha.org/assimilation/archive/tip.tar.gz

Untar it, and go into the cma directory.  Run the python script store.py

It will fail.  Each time you run it, it will give a higher node in the graph that it cannot find.

If it doesn't fail, then we can try the docker thing to reproduce the environment more exactly.

To try the docker version, you just need to install docker and say "docker build ." (or docker.io) with my Dockerfile in your directory.  It will do the rest - grab the OS, libraries, things we need, build my code and run my tests.  It just takes longer.  For this environment, it helps to understand docker when you want to debug it.

It's pretty awesome.  But hopefully you won't need to do that.

I also have a more complex set of tests that only require python that will likely fail if there's a Neo4j/Py2neo issue that affects me.  Maybe I should create a separate test environment for that - that you could easily run.




On 07/15/2014 04:42 PM, Nigel Small wrote:

Familiar with the concept but have never actually used it.

On 15 July 2014 23:41, Alan Robertson <al...@unix.sh> wrote:

Yes.  There are lots of ways to do that.  After all, it's all open source :-D.

Are you familiar with docker?

On 07/15/2014 04:19 PM, Nigel Small wrote:

Can you share some code that I can run to recreate the issue?

Cheers
Nigel

On 15 July 2014 23:09, Alan Robertson <al...@unix.sh> wrote:

Neo4j package: neo4j-2.1.2-1.noarch

# pip list | grep py2neo
py2neo (1.6.4)

The context is this:
I just did a commit of a transaction which added nodes to the database, and am now trying to retrieve these same nodes by the ids for them which were returned by the transaction.


tests.cma_test TestCMABasic.test_several_startups ... error: tests.cma_test TestCMABasic.test_several_startups
Traceback (most recent call last):
  File "./tests/cma_test.py", line 506, in test_several_startups
    CMAinit(io, cleanoutdb=True, debug=DEBUG)
  File "./cmainit.py", line 75, in __init__
    CMAdb.store.commit()
  File "./store.py", line 912, in commit
    for attr in newnode.get_properties():
  File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/py2neo/neo4j.py", line 1398, in get_properties
    self._properties = assembled(self._properties_resource._get()) or {}
  File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/py2neo/neo4j.py", line 367, in _get
    raise ClientError(e)
ClientError: Cannot find node with id [0] in database.

ERROR in 2.49s
tests.cma_test TestCMABasic.test_startup ... error: tests.cma_test TestCMABasic.test_startup
Traceback (most recent call last):
  File "./tests/cma_test.py", line 433, in test_startup
    CMAinit(io, cleanoutdb=True, debug=DEBUG)
  File "./cmainit.py", line 75, in __init__
    CMAdb.store.commit()
  File "./store.py", line 912, in commit
    for attr in newnode.get_properties():
  File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/py2neo/neo4j.py", line 1398, in get_properties
    self._properties = assembled(self._properties_resource._get()) or {}
  File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/py2neo/neo4j.py", line 367, in _get
    raise ClientError(e)
ClientError: Cannot find node with id [1] in database.

--
    Alan Robertson <al...@unix.sh> - @OSSAlanR

"Openness is the foundation and preservative of friendship...  Let me claim from you at all times your undisguised opinions." - William Wilberforce

-- 
    Alan Robertson <al...@unix.sh> - @OSSAlanR

"Openness is the foundation and preservative of friendship...  Let me claim from you at all times your undisguised opinions." - William Wilberforce

-- 
    Alan Robertson <al...@unix.sh> - @OSSAlanR

"Openness is the foundation and preservative of friendship...  Let me claim from you at all times your undisguised opinions." - William Wilberforce

[Alan Robertson]

And for good measure, if you use the Dockerfile quoted below (and attached), this is a clean-room reproduction of the problem in CentOS6 (with Python 2.7).

###############################################
FROM centos:latest
# Would prefer headless, but not available in CentOS6
RUN yum -y install wget java-1.7.0-openjdk scl-utils redhat-lsb-core
RUN wget -qO- http://dev.centos.org/centos/6/SCL/scl.repo >> /etc/yum.repos.d/centos.scl.repo
RUN yum -y install python27-python python27-python-setuptools
###  Neo4j RPM
RUN wget http://debian.neo4j.org/neotechnology.gpg.key -O /tmp/neo4j.key  && rpm --import /tmp/neo4j.key && rm -f /tmp/neo4j.key
RUN echo '[neo4j]' > /etc/yum.repos.d/neo4j.repo && echo 'name=Neo4j Yum Repo' >> /etc/yum.repos.d/neo4j.repo && echo 'baseurl=http://yum.neo4j.org' >> /etc/yum.repos.d/neo4j.repo && echo 'enabled=1' >> /etc/yum.repos.d/neo4j.repo && echo 'gpgcheck=1' >> /etc/yum.repos.d/neo4j.repo && yum -y install neo4j
RUN scl enable python27 'easy_install pip'
RUN scl enable python27 'pip install py2neo'
RUN echo "Pulling Assimilation project source"
RUN mkdir /tmp/neotest && cd /tmp/neotest && wget -q http://hg.linux-ha.org/assimilation/raw-file/tip/cma/store.py  && wget -q http://hg.linux-ha.org/assimilation/raw-file/tip/cma/assimevent.py
RUN NEO=neo4j; cd /tmp/neotest && /etc/init.d/${NEO} start && sleep 15 && scl enable python27 '/usr/bin/env python --version; python store.py'
##################################################

Do this:
    make a directory
    cd into the directory you just made
    copy the above text into a file in that directory - named Dockerfile
    docker.io build .

This is a completely clean-room demonstration of the problem in CentOS6.

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nigel Small]

Cool, thanks. I should get a chance to look into it over the next couple of weeks.

Cheers
Nigel

[Alan Robertson]

A little more detail about the failure mode:
    I added a single node to the database, related it to itself three ways, added it to the index.
    committed the transaction - returned results from the commit looked correct
    Tried a get_properties() on the object returned from the transaction commit
        Just before the get_properties() failed, I printed that object, and got this result:
                {"a":52,"c":3.14159,"b":2,"name":"Drone121"}
                [I also printed the node _id -- which was 0]

I updated the test in source control to add debug statements, and run the test with debug on.  If you run the Dockerfile, or just wget using the URLs I supplied before, you'll have the newest version of the test.  Keep in mind that docker caches results, so if you've cached the results of the Dockerfile, you'll have to tweak it somehow (change the echo or something) to make sure it grabs the latest source.

As a reminder, this works just fine with 2.0.x (where x >= 1) versions of Neo4j.   It only fails with 2.1.x versions of Neo4j.

The output with the current debug turned on looks like this:

Python 2.7.5
('COMMITTING THIS THING:', "{\n\tdb: Resource(u'http://localhost:7474/db/data/'),\n\tclasses: {<class '__main__.Drone'>: True},\n\tuniqueindexmap: {'Drone': True},\n\tclasskeymap: {'Drone': {'index': 'Drone', 'vattr': 'name', 'key': 'Drone'}}\n\tbatchindex: None,\n\t   clients: \n[<__main__.Drone object at 0x27211d0>,\n\t   newrels: \n[{'to': <__main__.Drone object at 0x27211d0>, 'props': None, 'from': <__main__.Drone object at 0x27211d0>, 'type': 'ISA'}, {'to': <__main__.Drone object at 0x27211d0>, 'props': None, 'from': <__main__.Drone object at 0x27211d0>, 'type': 'WASA'}, {'to': <__main__.Drone object at 0x27211d0>, 'props': None, 'from': <__main__.Drone object at 0x27211d0>, 'type': 'WILLBEA'},\n\t deletions: \n\n\tweaknoderefs: {}\n\tstats: {'totaltime': datetime.timedelta(0), 'index': 0, 'nodecreate': 0, 'separate': 0, 'relate': 0, 'lastcommit': None, 'nodedelete': 0, 'attrupdate': 0}\n\tbatch: None\n}")
Performing batch.create(0: ({"a":52,"c":3.14159,"b":2,"name":"Drone121"})) - for new node
Performing batch.create(0-[:ISA]->0): node relationships
ADDING rel 0-[:ISA]->0
Performing batch.create(0-[:WASA]->0): node relationships
ADDING rel 0-[:WASA]->0
Performing batch.create(0-[:WILLBEA]->0): node relationships
ADDING rel 0-[:WILLBEA]->0
add_to_index_or_fail: node 0; index Index(Node, u'http://localhost:7474/db/data/index/node/Drone')("Drone","Drone121")
('Batch Updates constructed: Committing THIS THING:', "{\n\tdb: Resource(u'http://localhost:7474/db/data/'),\n\tclasses: {<class '__main__.Drone'>: True},\n\tuniqueindexmap: {'Drone': True},\n\tclasskeymap: {'Drone': {'index': 'Drone', 'vattr': 'name', 'key': 'Drone'}}\n\tbatchindex: 4,\n\t   clients: \n[<__main__.Drone object at 0x27211d0>,\n\t   newrels: \n[{'from': <__main__.Drone object at 0x27211d0>, 'seqno': 1, 'abstract': rel(0, 'ISA', 0), 'to': <__main__.Drone object at 0x27211d0>, 'props': None, 'type': 'ISA'}, {'from': <__main__.Drone object at 0x27211d0>, 'seqno': 2, 'abstract': rel(0, 'WASA', 0), 'to': <__main__.Drone object at 0x27211d0>, 'props': None, 'type': 'WASA'}, {'from': <__main__.Drone object at 0x27211d0>, 'seqno': 3, 'abstract': rel(0, 'WILLBEA', 0), 'to': <__main__.Drone object at 0x27211d0>, 'props': None, 'type': 'WILLBEA'},\n\t deletions: \n\n\tweaknoderefs: {}\n\tstats: {'totaltime': datetime.timedelta(0), 'index': 1, 'nodecreate': 1, 'separate': 0, 'relate': 3, 'lastcommit': None, 'nodedelete': 0, 'attrupdate': 0}\n\tbatch: <py2neo.neo4j.WriteBatch object at 0x2721450>\n}")
SUBMITRESULT: (0 {"a":52,"c":3.14159,"b":2,"name":"Drone121"})
LOOKING at new node with batch index 0
NEW NODE looks like (0 {"a":52,"c":3.14159,"b":2,"name":"Drone121"})
SUBJ (our copy) looks like <__main__.Drone object at 0x27211d0>

Traceback (most recent call last):

  File "store.py", line 1099, in <module>
    testme()
  File "store.py", line 1014, in testme
    store.commit()  # The updates have been captured...
  File "store.py", line 923, in commit
    %       (newnode._id, str(newnode.get_properties())))
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/py2neo/neo4j.py", line 1398, in get_properties
    self._properties = assembled(self._properties_resource._get()) or {}
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/py2neo/neo4j.py", line 367, in _get
    raise ClientError(e)
py2neo.exceptions.ClientError: Cannot find node with id [0] in database.

[Alan Robertson]

Since the result said "cannot find node" the question is: Was that a correct result?   So, I used neo4j-shell to look at the database after the failure - and it looked like the updates were not done.  There is no node zero (as indicated by the REST return), nor (for good measure) a node 1 or 2.  So the result was correct for the current database.  The database was wrong, but that result was correct (given the incorrect database).

We appear to have gotten a successful return from the REST update transaction, but in spite of that, the node I created wasn't there.

I will look further into what we got back from the REST transaction -- but right now, this is looking like a Neo4j 2.1 bug.  I use Nigel's latest version quite successfully with 2.0.1.  I know you guys are at OSCON.  If you read this and can tell me what you want to see to diagnose it, that would be awesome.

On 07/19/2014 12:09 PM, Alan Robertson wrote:

A little more detail about the failure mode:
    I added a single node to the database, related it to itself three ways, added it to the index.
    committed the transaction - returned results from the commit looked correct
    Tried a get_properties() on the object returned from the transaction commit

        Just before the get_properties() failed, I printed that object (Nigel's Neo4j node object), and got this result:

                {"a":52,"c":3.14159,"b":2,"name":"Drone121"}

                [I also printed the node _id value -- which was 0]

[Alan Robertson]

Our project bug report for this is here: https://trello.com/c/B41LtH8K  The github issue is: https://github.com/neo4j/neo4j/issues/2771.  You can reproduce this in a clean-room environment by using this Dockerfile:  http://hg.linux-ha.org/assimilation/file/tip/docker/testNeo/Dockerfile

the DEB package of 2.1.0 works fine on all versions of Ubuntu, but the RPM of 2.1.3 fails under CentOS6.  Slightly different versions of Python, same version of Py2neo.  Installing 2.0.1 from tar ball on CentOS6 also works fine.

This looks more like a Neo4j problem than a py2neo problem...

Also: Why is the .DEB unstable package so far behind the .RPM package?



More details below:

This combination doesn't work with CentOS6 under Docker:

Python 2.7.5
USING PYTHON VERSION 2.7.5 (default, Oct 29 2013, 22:46:38)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)]
USING NEO4J VERSION (2, 1, 3, u'')
USING py2neo VERSION 1.6.4

3 minutes ago - Edit - Delete

Alan Robertson

This combination of versions works fine with Ubuntu under Docker: USING PYTHON VERSION 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2]
USING NEO4J VERSION (2, 1, 0, u'RC2')
USING py2neo VERSION 1.6.4

[Alan Robertson]

OK.  This is an RPM problem *only*.  It works fine if I install 2.1.3 from tar ball - but not from the RPM.  I'll update the github issue.

[Alan Robertson]

Julian:

Have you had a chance to look at this bug yet?  Any prognosis on when?

[Aru Sahni]

I'm seeing similar errors w/ Neo4j 2.1.4, py2neo 1.7 and Ubuntu packages. No issues with the homebrew version.

~A

[Nigel Small]

There is no py2neo 1.7.

[Aru Sahni]

Whoops. Rather, I tried it with 1.6.4 and then whatever code is in the 1.7 branch :-)