Neo4j crashes constantly.

Alx

unread,

Feb 17, 2014, 12:54:25 PM2/17/14

to ne...@googlegroups.com

I have a Mac with 16Gb of RAM and I am running Neo4j 2.01 with 70M nodes and 60M relationships with Java 7. The graph occupies 6Gb on my hdd.

The Neo4j server constantly crashes even when limiting my queries to 10. I updated the neo4j.properties file to:

neostore.nodestore.db.mapped_memory=250M

neostore.relationshipstore.db.mapped_memory=500M

neostore.propertystore.db.mapped_memory=900M

neostore.propertystore.db.strings.mapped_memory=1300M

neostore.propertystore.db.arrays.mapped_memory=1300M

and the neo4j-wrapper.conf file to:

wrapper.java.initmemory=1512

wrapper.java.maxmemory=5120

I also start the Neo4j by running the following command:

neo4j start -server -Xmx7G -XX:+UseConcMarkSweepGC

I have run out of ideas. Anyone could help? Thanks!

Michael Hunger

unread,

Feb 17, 2014, 2:37:19 PM2/17/14

to ne...@googlegroups.com

Can you please share your data/graph.db/messages.log and data/log/console.log with us?

Thanks a lot

Michael

You probably don't need that much memory for strings and arrays.

neostore.propertystore.db.strings.mapped_memory=1300M
neostore.propertystore.db.arrays.mapped_memory=1300M

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Alx

unread,

Feb 17, 2014, 2:56:39 PM2/17/14

to ne...@googlegroups.com

Hi Michael,

Thanks for the reply. Basically I am working on this DocGraph Neo4j db that David Fauth has posted: https://groups.google.com/forum/#!topic/docgraph/sKerXGNVOyQ

Please download and try to run a simple query like this:

match (n:provider {npi:"1366440703"}) -[:REFERRED]->m return n,m LIMIT 10;

First of all the query will take forever to execute and it will never end successfully. At least on my machine. You can share your configuration if it works on your machine.

Here is the message.log warnings:

2014-02-17 19:50:10.949+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 118ms [total block time: 10.01s]

2014-02-17 19:50:12.302+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 103ms [total block time: 10.113s]

2014-02-17 19:50:13.773+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 163ms [total block time: 10.276s]

2014-02-17 19:50:15.247+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 166ms [total block time: 10.442s]

2014-02-17 19:50:16.844+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 191ms [total block time: 10.633s]

2014-02-17 19:50:19.598+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 137ms [total block time: 10.77s]

2014-02-17 19:51:02.696+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 37449ms [total block time: 48.219s]

2014-02-17 19:51:44.214+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 40311ms [total block time: 88.53s]

2014-02-17 19:52:24.785+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 38763ms [total block time: 127.293s]

2014-02-17 19:53:05.926+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 39326ms [total block time: 166.619s]

And the console.log:

14:52:24.886 [qtp348042924-55] WARN o.e.jetty.servlet.ServletHandler - /db/manage/server/monitor/fetch

javax.ws.rs.WebApplicationException: javax.ws.rs.WebApplicationException: org.eclipse.jetty.io.EofException

at org.neo4j.server.rest.repr.OutputFormat$1.write(OutputFormat.java:174) ~[neo4j-server-2.0.1.jar:2.0.1]

at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71) ~[jersey-core-1.9.jar:1.9]

at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57) ~[jersey-core-1.9.jar:1.9]

at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306) ~[jersey-server-1.9.jar:1.9]

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437) ~[jersey-server-1.9.jar:1.9]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) ~[jersey-server-1.9.jar:1.9]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) ~[jersey-server-1.9.jar:1.9]

at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) ~[jersey-server-1.9.jar:1.9]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) ~[jersey-server-1.9.jar:1.9]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) ~[jersey-server-1.9.jar:1.9]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) ~[javax.servlet-3.0.0.v201112011016.jar:na]

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:698) ~[jetty-servlet-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1506) ~[jetty-servlet-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.neo4j.server.rest.security.SecurityFilter.doFilter(SecurityFilter.java:112) ~[neo4j-server-2.0.1.jar:2.0.1]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1477) ~[jetty-servlet-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503) [jetty-servlet-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:211) ~[jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1096) ~[jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:432) [jetty-servlet-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175) ~[jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1030) ~[jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) ~[jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) ~[jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) ~[jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.server.Server.handle(Server.java:445) ~[jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:268) ~[jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:229) ~[jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358) ~[jetty-io-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) ~[jetty-util-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) ~[jetty-util-9.0.5.v20130815.jar:9.0.5.v20130815]

at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]

Caused by: javax.ws.rs.WebApplicationException: org.eclipse.jetty.io.EofException

at org.neo4j.server.rest.repr.formats.StreamingJsonFormat$StreamingRepresentationFormat.flush(StreamingJsonFormat.java:401) ~[neo4j-server-2.0.1.jar:2.0.1]

at org.neo4j.server.rest.repr.formats.StreamingJsonFormat$StreamingRepresentationFormat.complete(StreamingJsonFormat.java:389) ~[neo4j-server-2.0.1.jar:2.0.1]

at org.neo4j.server.rest.repr.MappingRepresentation.serialize(MappingRepresentation.java:43) ~[server-api-2.0.1.jar:2.0.1]

at org.neo4j.server.rest.repr.OutputFormat$1.write(OutputFormat.java:160) ~[neo4j-server-2.0.1.jar:2.0.1]

... 30 common frames omitted

Caused by: org.eclipse.jetty.io.EofException: null

at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:186) ~[jetty-io-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:335) ~[jetty-io-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:125) ~[jetty-io-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.server.HttpConnection$ContentCallback.process(HttpConnection.java:784) ~[jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:79) ~[jetty-util-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:356) ~[jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:631) ~[jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:661) ~[jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815]

at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:151) ~[jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815]

at com.sun.jersey.spi.container.servlet.WebComponent$Writer.flush(WebComponent.java:315) ~[jersey-server-1.9.jar:1.9]

at com.sun.jersey.spi.container.ContainerResponse$CommittingOutputStream.flush(ContainerResponse.java:145) ~[jersey-server-1.9.jar:1.9]

at org.codehaus.jackson.impl.Utf8Generator.flush(Utf8Generator.java:1091) ~[jackson-core-asl-1.9.7.jar:1.9.7]

at org.neo4j.server.rest.repr.formats.StreamingJsonFormat$StreamingRepresentationFormat.flush(StreamingJsonFormat.java:397) ~[neo4j-server-2.0.1.jar:2.0.1]

... 33 common frames omitted

Caused by: java.io.IOException: Broken pipe

at sun.nio.ch.FileDispatcherImpl.writev0(Native Method) ~[na:1.7.0_25]

at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) ~[na:1.7.0_25]

at sun.nio.ch.IOUtil.write(IOUtil.java:149) ~[na:1.7.0_25]

at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:499) ~[na:1.7.0_25]

at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:167) ~[jetty-io-9.0.5.v20130815.jar:9.0.5.v20130815]

... 45 common frames omitted

Michael Hunger

unread,

Feb 17, 2014, 3:59:46 PM2/17/14

to ne...@googlegroups.com

I just looked at Dave's github repository: https://github.com/davidfauth/neo4JDocGraph

match (n:provider {npi:"1366440703"}) -[:REFERRED]->m return n,m LIMIT 10;

The last update to the repo was 6 months ago, it might well be that the index and label store files have to be rebuild for 2.0.1

You can do that by deleting data/graph.db/schema and starting the db (which rebuilds the label scan store)

And then recreate the indexes / constraints, which will take a while.

create index on :organization(name);
create index on :specialty(name);
create index on :specialty(code);
create index on :provider(name);
create index on :provider(address_postal_code);
create index on :provider(telephone_number);
create index on :provider(npi);
create index on :location(address_city_name);
create index on :location(address_state_name);
create index on :location(address_country_code);

you can check the status with the "schema" command in the neo4j-shell.

in the end it should show something like this:

Indexes
  ON :location(address_city_name)   ONLINE
  ON :location(address_state_name)   ONLINE
  ON :location(address_country_code) ONLINE
  ON :organization(name)     ONLINE
  ON :provider(name)     ONLINE
  ON :provider(address_postal_code)  ONLINE
  ON :provider(telephone_number)     ONLINE
  ON :provider(npi)   ONLINE
  ON :specialty(name)   ONLINE
  ON :specialty(code)   ONLINE

After doing that, and actually finding a provider that had referrals with

match (n:provider) where (n)-[:REFERRED]->() return n LIMIT 5;

I get things like that:

match (n:provider {npi:"1386642726"})-[:REFERRED]->m return m.npi, m.organization_name, m.address_first_line LIMIT 10;
+---------------------------------------------------------------------------------+
| m.npi | m.organization_name | m.address_first_line |
+---------------------------------------------------------------------------------+
| "1093743650" | "NURSING RESOURCES CORPORATION" | "3600 BRIARFIELD BLVD" |
| "1881890176" | "" | "3120 GLENDALE AVE" |
| "1093822850" | "" | "7119 CHAMBERSBURG DR" |
| "1760568745" | "" | "3000 ARLINGTON AVENUE" |
| "1548279714" | "FULTON COUNTY HEALTH CENTER" | "725 S SHOOP AVE" |
| "1164598769" | "CITY OF TOLEDO" | "ONE GOVERNMENT CENTER" |
| "1548410053" | "" | "3065 ARLINGTON AVE" |
| "1629087168" | "" | "3000 ARLINGTON AVE" |
| "1700849858" | "MEDICAL UNIVERSITY OF OHIO AT TOLEDO" | "3000 ARLINGTON AVE" |
| "1023003530" | "" | "3355 GLENDALE AVE" |
+---------------------------------------------------------------------------------+
10 rows
6 ms

I'll zip and upload the updated db to example-data.neo4j.org/files/DocGraph-neo4j-201.zip

But it will take a while :)

HTH,

Michael

Alx

unread,

Feb 17, 2014, 11:18:13 PM2/17/14

to ne...@googlegroups.com

Michael,

Thanks for the help! I was able to re-index the database and it seems it works better. I have a couple of questions for specific queries:

a) When I am running "MATCH (n:provider) - [:REFERRED]-> m RETURN n.name, m.name, count(DISTINCT m) AS Counter LIMIT 5; " the Neo4j server crashes. On messages.log I get the following:\

2014-02-18 04:04:00.380+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 113ms [total block time: 0.113s]

2014-02-18 04:04:00.667+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 187ms [total block time: 0.3s]

2014-02-18 04:04:34.821+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 11698ms [total block time: 11.998s]

2014-02-18 04:04:51.466+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 14943ms [total block time: 26.941s]

2014-02-18 04:05:07.027+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 13947ms [total block time: 40.888s]

2014-02-18 04:05:07.273+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 146ms [total block time: 41.034s]

2014-02-18 04:05:23.382+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 14800ms [total block time: 55.834s]

2014-02-18 04:05:37.460+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 13069ms [total block time: 68.903s]

2014-02-18 04:05:53.906+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 15538ms [total block time: 84.441s]

2014-02-18 04:05:54.374+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 368ms [total block time: 84.809s]

2014-02-18 04:06:10.941+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 15760ms [total block time: 100.569s]

2014-02-18 04:06:11.453+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 411ms [total block time: 100.98s]

2014-02-18 04:06:27.904+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 15647ms [total block time: 116.627s]

2014-02-18 04:06:28.433+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 429ms [total block time: 117.056s]

and on console.log I get:

Caused by: java.io.IOException: Broken pipe

How could I make this query run for the whole database and not limit to 5?

b) I want to run a second query to the the query above. I have managed to do this:

" MATCH (n:provider) - [:REFERRED]-> m WITH n,m MATCH n - [:SPECIALTY] -> a RETURN a.classification AS ProviderA "

But I want it also to return " MATCH m - [:SPECIALTY] -> b RETURN b.classification AS ProviderB " at the same second query above. How is it possible?

Thanks !

Michael Hunger

unread,

Feb 18, 2014, 4:00:09 AM2/18/14

to ne...@googlegroups.com

You still haven't provided your FULL messages.log without the diagnostics information I can't help you with regard to configuration.

These are queries that pull the whole database through memory, and compute an almost cross product of providers (have to aggregate all first before applying a limit).

try something like this

MATCH (n:provider) WITH n LIMIT 5 MATCH (n) - [:REFERRED]-> m RETURN n.name, m.name, count(DISTINCT m) AS Counter LIMIT 5;

But it's not so useful

You'd rather want to do:

MATCH (n:provider) WITH n LIMIT 5 MATCH (n) - [:REFERRED]-> m RETURN n.name, collect(distinct m.name), count(DISTINCT m) AS Counter LIMIT 5;

Which returns in 141ms for me.

Michael

Alx

unread,

Feb 19, 2014, 10:54:00 AM2/19/14

to ne...@googlegroups.com

Hi Michael,

Thanks for the response. I am attaching the log files.

I also querying the following:

" MATCH (n:provider) - [:REFERRED]-> m WITH n,m MATCH n - [:SPECIALTY] -> a RETURN a.classification AS ProviderA "

But I want it also to return " MATCH m - [:SPECIALTY] -> b RETURN b.classification AS ProviderB " at the same second query above. How is it possible?

Thanks a lot!

messages.log

console.log

neo4j.0.0.log

Reply all

Reply to author

Forward