neo4j upgrade, loss of data!

[shai katzir]

Just now, I have upgraded my neo4j server from 2.0.3 to 2.1.2 by explicit update (allow_store_update=true).

After upgrading, half of my test cases didn't work!

Going through the data, I found out a lot of it is missing.

I would add any data or log to allow investigating the problem but there is none...

All I have is the console.log showing everything is fine:

2014-06-19 08:01:42.094+0000 INFO  [API] Setting startup timeout to: 120000ms based on -1
Starting upgrade of database store files
2014-06-19 08:01:42.529+0000 INFO  [org.neo4j]: Starting upgrade of database store files
# dense nodes: 2590, which is 22 %
..........2014-06-19 08:01:44.558+0000 INFO  [org.neo4j]: Store upgrade 10% complete
 10%
..........2014-06-19 08:01:45.560+0000 INFO  [org.neo4j]: Store upgrade 20% complete
 20%
..........2014-06-19 08:01:45.560+0000 INFO  [org.neo4j]: Store upgrade 30% complete
 30%
..........2014-06-19 08:01:47.728+0000 INFO  [org.neo4j]: Store upgrade 40% complete
 40%
..........2014-06-19 08:01:48.729+0000 INFO  [org.neo4j]: Store upgrade 50% complete
 50%
..........2014-06-19 08:01:48.923+0000 INFO  [org.neo4j]: Store upgrade 60% complete
 60%
..........2014-06-19 08:01:48.923+0000 INFO  [org.neo4j]: Store upgrade 70% complete
 70%
..........2014-06-19 08:01:48.923+0000 INFO  [org.neo4j]: Store upgrade 80% complete
 80%
..........2014-06-19 08:01:48.924+0000 INFO  [org.neo4j]: Store upgrade 90% complete
 90%
..........2014-06-19 08:01:48.924+0000 INFO  [org.neo4j]: Store upgrade 100% complete
 100%
2014-06-19 08:01:48.924+0000 INFO  [org.neo4j.unsafe.impl.batchimport.ParallellBatchImporter] Import completed [TODO import stats]
2014-06-19 08:01:48.924+0000 INFO  [org.neo4j.unsafe.impl.batchimport.ParallellBatchImporter] Import completed [TODO import stats]
Finished upgrade of database store files
2014-06-19 08:01:49.578+0000 INFO  [org.neo4j]: Finished upgrade of database store files
2014-06-19 08:01:49.578+0000 INFO  [org.neo4j.unsafe.impl.batchimport.ParallellBatchImporter] Shutting down
2014-06-19 08:01:49.578+0000 INFO  [org.neo4j.unsafe.impl.batchimport.ParallellBatchImporter] Shutting down
2014-06-19 08:01:50.020+0000 INFO  [org.neo4j]: Migration completed in 7 s


11:01:54.959 [main] WARN  o.e.j.server.handler.ContextHandler - o.e.j.s.ServletContextHandler@32e1e333{/,null,null} contextPath ends with /
11:01:54.959 [main] WARN  o.e.j.server.handler.ContextHandler - Empty contextPath
11:01:54.962 [main] INFO  org.eclipse.jetty.server.Server - jetty-9.0.5.v20130815
11:01:55.053 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.h.MovedContextHandler@7168eab5{/,null,AVAILABLE}
11:01:55.228 [main] INFO  o.e.j.w.StandardDescriptorProcessor - NO JSP Support for /webadmin, did not find org.apache.jasper.servlet.JspServlet
11:01:55.254 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.w.WebAppContext@20c22f4d{/webadmin,jar:file:/home/shaika/Workspace/neo4j-server/system/lib/neo4j-server-2.1.2-static-web.jar!/webadmin-html,AVAILABLE}
11:01:56.370 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@8180687{/db/manage,null,AVAILABLE}
11:01:56.871 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@7b94d4d4{/db/data,null,AVAILABLE}
11:01:56.900 [main] INFO  o.e.j.w.StandardDescriptorProcessor - NO JSP Support for /browser, did not find org.apache.jasper.servlet.JspServlet
11:01:56.902 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.w.WebAppContext@7422b0de{/browser,jar:file:/home/shaika/Workspace/neo4j-server/system/lib/neo4j-browser-2.1.2.jar!/browser,AVAILABLE}
11:01:57.071 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@32e1e333{/,null,AVAILABLE}
11:01:57.089 [main] INFO  o.e.jetty.server.ServerConnector - Started ServerConnector@39d07ac8{HTTP/1.1}{0.0.0.0:7474}
11:01:57.921 [main] INFO  o.e.jetty.server.ServerConnector - Started ServerConnector@1fc8e3d{SSL-HTTP/1.1}{0.0.0.0:7484}
11:15:05.270 [qtp273404639-71] WARN  o.e.jetty.servlet.ServletHandler - /db/data/transaction/commit

[Axel Morgner]

Hi,

can you be more precise what kind of data was missing? There are some changes in the way relationships are stored in Neo4j, and the previous release 2.1.1 had some issues related to databases upgraded from 2.0.x. But they seem to be fixed with 2.1.2, see http://neo4j.com/blog/neo4j-2-1-2/ and https://github.com/neo4j/neo4j/issues/2540.

Do you have a backup? If yes, I would suggest to go back to 2.0.3 for now.

Please open an issue on Github (https://github.com/neo4j/neo4j/issues) with a detailed description of your data, and offer the upgraded database to the Neo guys to investigate.

Best,
Axel

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Axel Morgner
CEO Structr (c/o Morgner UG) · Hanauer Landstr. 291a · 60314 Frankfurt · Germany
Twitter: @amorgner
Phone: +49 151 40522060
Skype: axel.morgner

Structr - Award-Winning Open Source CMS and Web Framework based on Neo4j
Structr Mailing List and Forum
Graph Database Usergroup "graphdb-frankfurt"

[shai katzir]

Of course I have backup, and it was only a test, but not a successful one.

It's hard to say what data is lost.

I think that the dense nodes (the ones that I know have a lot of incoming relationships) lost all their relationships. That is also correspond to the nature of the version update.

בתאריך יום חמישי, 19 ביוני 2014 23:14:21 UTC+3, מאת Axel:

2014-06-19 08:01:<span style="color: #066;" clas

...

[mat...@neotechnology.com]

Hi Shai,

I got access to your store and using the backed up nodestore and relationshipstore in upgrade_backup, I compared contents of the upgraded (2.1.2) store to that of the 2.0.3 store.

What I found found was that all nodes, properties and relationships are exactly the same. I dumped nodes, their properties and their relationships (and properties) to text files, sorted, for both stores and they have the same contents, down to the last character.

I also did this for just the INCOMING relationships, just for the heck of it, since you observed issues with those. And the outcome is the same.

Can you redo your checks for the upgraded store and provide more details about the problems?

Best,
Mattias

[shai katzir]

It's getting a little weird.

I run the next query :

match (pr:Property {name: "BLACK"})

match (pr)--(it)

return pr,it

where "pr:Property {name: "BLACK"} " is a "densed node".

Some times I get 0 results for this query in the browser, and then I restart the graph server and I get ~900 results.

I guess the data isn't lost, it's just sometimes doesn't get retrieved by the cypher queries ...

[Mattias Persson]

This could be a browser issue actually. I tried running your query directly via a Cypher ExecutionEngine, and no matter how many times I tried, it always gave me the 860 results back.

--
You received this message because you are subscribed to a topic in the Google Groups "Neo4j" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/neo4j/s3K4k_9orGQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to neo4j+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mattias Persson

Neo4j Hacker at Neo Technology

[shai katzir]

I tried it both on the browser AND from neo4j nodejs client. When it doesn't work, it doesn't work on both of them.

My os is ubuntu 12.04.

Like I said, sometimes it doesn't work, so i do 'neo4j restart' and then it works...

Shai Katzir

CTO, Donde 

M: (650) 687 7424