I'm just starting with neo4j and running into some issues, and was hoping to get some help here.
I created a graph with about 600k nodes and 11M relationships - I have plenty of disk space left. I have three types of nodes and two indices. My data import script seemed to be running fine, and then inexplicably starting returning 400s.
I googled parts of the stack trace but couldn't find any relevant details. Every time I try to start the server I get the trace below.
Any help would be greatly appreciated. Thanks!
- Justin
21:14:04.884 [18183221@qtp-11467963-4103] ERROR org.mortbay.log - /db/data/node org.neo4j.graphdb.TransactionFailureException: Unable to begin transaction at org.neo4j.kernel.InternalAbstractGraphDatabase.beginTx(InternalAbstractGrap hDatabase.java:747) ~[neo4j-kernel-1.8.M06.jar:1.8.M06] at org.neo4j.kernel.TransactionBuilderImpl.begin(TransactionBuilderImpl.java:3 9) ~[neo4j-kernel-1.8.M06.jar:1.8.M06] at org.neo4j.server.rest.web.DatabaseActions.beginTx(DatabaseActions.java:228) ~[neo4j-server-1.8.M06.jar:1.8.M06] at org.neo4j.server.rest.web.DatabaseActions.createNode(DatabaseActions.java:2 10) ~[neo4j-server-1.8.M06.jar:1.8.M06] at org.neo4j.server.rest.web.RestfulGraphDatabase.createNode(RestfulGraphDatab ase.java:195) ~[neo4j-server-1.8.M06.jar:1.8.M06] at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp l.java:43) ~[na:1.6.0_20] at java.lang.reflect.Method.invoke(Method.java:616) ~[na:1.6.0_20] at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodIn vokerFactory.java:60) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDisp atchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchPro vider.java:205) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatch er.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.j ava:288) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPath Rule.java:147) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClass Rule.java:108) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPath Rule.java:147) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootRes ourceClassesRule.java:84) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(We bApplicationImpl.java:1469) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(We bApplicationImpl.java:1400) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(Web ApplicationImpl.java:1349) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(Web ApplicationImpl.java:1339) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java :416) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContai ner.java:537) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContai ner.java:699) ~[jersey-server-1.9.jar:1.9] at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) ~[servlet-api-2.5-20081211.jar:na] at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) ~[jetty-6.1.25.jar:6.1.25] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle r.java:1166) ~[jetty-6.1.25.jar:6.1.25] at org.neo4j.server.statistic.StatisticFilter.doFilter(StatisticFilter.java:62 ) ~[neo4j-server-1.8.M06.jar:1.8.M06] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle r.java:1157) ~[jetty-6.1.25.jar:6.1.25] at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) ~[jetty-6.1.25.jar:6.1.25] at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) [jetty-6.1.25.jar:6.1.25] at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) [jetty-6.1.25.jar:6.1.25] at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:1 14) [jetty-6.1.25.jar:6.1.25] at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) [jetty-6.1.25.jar:6.1.25] at org.mortbay.jetty.Server.handle(Server.java:326) [jetty-6.1.25.jar:6.1.25] at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) [jetty-6.1.25.jar:6.1.25] at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java :943) [jetty-6.1.25.jar:6.1.25] at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) [jetty-6.1.25.jar:6.1.25] at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) [jetty-6.1.25.jar:6.1.25] at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) [jetty-6.1.25.jar:6.1.25] at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410 ) [jetty-6.1.25.jar:6.1.25] at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:58 2) [jetty-util-6.1.25.jar:6.1.25] Caused by: javax.transaction.SystemException: TM has encountered some problem, please perform neccesary action (tx recovery/restart) at org.neo4j.kernel.impl.transaction.TxManager.assertTmOk(TxManager.java:362) ~[neo4j-kernel-1.8.M06.jar:1.8.M06] at org.neo4j.kernel.impl.transaction.TxManager.begin(TxManager.java:339) ~[neo4j-kernel-1.8.M06.jar:1.8.M06] at org.neo4j.kernel.InternalAbstractGraphDatabase.beginTx(InternalAbstractGrap hDatabase.java:742) ~[neo4j-kernel-1.8.M06.jar:1.8.M06] ... 40 common frames omitted
> I'm just starting with neo4j and running into some issues, and was hoping to get some help here.
> I created a graph with about 600k nodes and 11M relationships - I have plenty of disk space left. I have three types of nodes and two indices. My data import script seemed to be running fine, and then inexplicably starting returning 400s.
> I googled parts of the stack trace but couldn't find any relevant details. Every time I try to start the server I get the trace below.
> Any help would be greatly appreciated. Thanks!
> - Justin
> 21:14:04.884 [18183221@qtp-11467963-4103] ERROR org.mortbay.log - /db/data/node
> org.neo4j.graphdb.TransactionFailureException: Unable to begin transaction
> at org.neo4j.kernel.InternalAbstractGraphDatabase.beginTx(InternalAbstractGrap hDatabase.java:747) ~[neo4j-kernel-1.8.M06.jar:1.8.M06]
> at org.neo4j.kernel.TransactionBuilderImpl.begin(TransactionBuilderImpl.java:3 9) ~[neo4j-kernel-1.8.M06.jar:1.8.M06]
> at org.neo4j.server.rest.web.DatabaseActions.beginTx(DatabaseActions.java:228) ~[neo4j-server-1.8.M06.jar:1.8.M06]
> at org.neo4j.server.rest.web.DatabaseActions.createNode(DatabaseActions.java:2 10) ~[neo4j-server-1.8.M06.jar:1.8.M06]
> at org.neo4j.server.rest.web.RestfulGraphDatabase.createNode(RestfulGraphDatab ase.java:195) ~[neo4j-server-1.8.M06.jar:1.8.M06]
> at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) ~[na:na]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp l.java:43) ~[na:1.6.0_20]
> at java.lang.reflect.Method.invoke(Method.java:616) ~[na:1.6.0_20]
> at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodIn vokerFactory.java:60) ~[jersey-server-1.9.jar:1.9]
> at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDisp atchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchPro vider.java:205) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatch er.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.9.jar:1.9]
> at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.j ava:288) ~[jersey-server-1.9.jar:1.9]
> at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPath Rule.java:147) ~[jersey-server-1.9.jar:1.9]
> at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClass Rule.java:108) ~[jersey-server-1.9.jar:1.9]
> at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPath Rule.java:147) ~[jersey-server-1.9.jar:1.9]
> at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootRes ourceClassesRule.java:84) ~[jersey-server-1.9.jar:1.9]
> at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(We bApplicationImpl.java:1469) ~[jersey-server-1.9.jar:1.9]
> at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(We bApplicationImpl.java:1400) ~[jersey-server-1.9.jar:1.9]
> at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(Web ApplicationImpl.java:1349) ~[jersey-server-1.9.jar:1.9]
> at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(Web ApplicationImpl.java:1339) ~[jersey-server-1.9.jar:1.9]
> at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java :416) ~[jersey-server-1.9.jar:1.9]
> at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContai ner.java:537) ~[jersey-server-1.9.jar:1.9]
> at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContai ner.java:699) ~[jersey-server-1.9.jar:1.9]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) ~[servlet-api-2.5-20081211.jar:na]
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) ~[jetty-6.1.25.jar:6.1.25]
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle r.java:1166) ~[jetty-6.1.25.jar:6.1.25]
> at org.neo4j.server.statistic.StatisticFilter.doFilter(StatisticFilter.java:62 ) ~[neo4j-server-1.8.M06.jar:1.8.M06]
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle r.java:1157) ~[jetty-6.1.25.jar:6.1.25]
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) ~[jetty-6.1.25.jar:6.1.25]
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) [jetty-6.1.25.jar:6.1.25]
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) [jetty-6.1.25.jar:6.1.25]
> at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:1 14) [jetty-6.1.25.jar:6.1.25]
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) [jetty-6.1.25.jar:6.1.25]
> at org.mortbay.jetty.Server.handle(Server.java:326) [jetty-6.1.25.jar:6.1.25]
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) [jetty-6.1.25.jar:6.1.25]
> at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java :943) [jetty-6.1.25.jar:6.1.25]
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) [jetty-6.1.25.jar:6.1.25]
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) [jetty-6.1.25.jar:6.1.25]
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) [jetty-6.1.25.jar:6.1.25]
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410 ) [jetty-6.1.25.jar:6.1.25]
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:58 2) [jetty-util-6.1.25.jar:6.1.25]
> Caused by: javax.transaction.SystemException: TM has encountered some problem, please perform neccesary action (tx recovery/restart)
> at org.neo4j.kernel.impl.transaction.TxManager.assertTmOk(TxManager.java:362) ~[neo4j-kernel-1.8.M06.jar:1.8.M06]
> at org.neo4j.kernel.impl.transaction.TxManager.begin(TxManager.java:339) ~[neo4j-kernel-1.8.M06.jar:1.8.M06]
> at org.neo4j.kernel.InternalAbstractGraphDatabase.beginTx(InternalAbstractGrap hDatabase.java:742) ~[neo4j-kernel-1.8.M06.jar:1.8.M06]
> ... 40 common frames omitted
On Thursday, August 9, 2012 7:01:11 PM UTC-7, Michael Hunger wrote:
> Justin,
> could you please share your servers data/graph.db/messages.log > as well as the logs in data/log ?
> And please don't delete the database (yet) we might ask you for it.
> Thanks a lot
> Michael
> Am 10.08.2012 um 01:08 schrieb Justin Pincar:
> > I'm just starting with neo4j and running into some issues, and was > hoping to get some help here.
> > I created a graph with about 600k nodes and 11M relationships - I have > plenty of disk space left. I have three types of nodes and two indices. My > data import script seemed to be running fine, and then inexplicably > starting returning 400s.
> > I googled parts of the stack trace but couldn't find any relevant > details. Every time I try to start the server I get the trace below.
> > Any help would be greatly appreciated. Thanks!
> > - Justin
> > 21:14:04.884 [18183221@qtp-11467963-4103] ERROR org.mortbay.log - > /db/data/node > > org.neo4j.graphdb.TransactionFailureException: Unable to begin > transaction > > at > org.neo4j.kernel.InternalAbstractGraphDatabase.beginTx(InternalAbstractGrap hDatabase.java:747) > ~[neo4j-kernel-1.8.M06.jar:1.8.M06] > > at > org.neo4j.kernel.TransactionBuilderImpl.begin(TransactionBuilderImpl.java:3 9) > ~[neo4j-kernel-1.8.M06.jar:1.8.M06] > > at > org.neo4j.server.rest.web.DatabaseActions.beginTx(DatabaseActions.java:228) > ~[neo4j-server-1.8.M06.jar:1.8.M06] > > at > org.neo4j.server.rest.web.DatabaseActions.createNode(DatabaseActions.java:2 10) > ~[neo4j-server-1.8.M06.jar:1.8.M06] > > at > org.neo4j.server.rest.web.RestfulGraphDatabase.createNode(RestfulGraphDatab ase.java:195) > ~[neo4j-server-1.8.M06.jar:1.8.M06] > > at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) > ~[na:na] > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp l.java:43) > ~[na:1.6.0_20] > > at java.lang.reflect.Method.invoke(Method.java:616) > ~[na:1.6.0_20] > > at > com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodIn vokerFactory.java:60) > ~[jersey-server-1.9.jar:1.9] > > at > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDisp atchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchPro vider.java:205) > ~[jersey-server-1.9.jar:1.9] at > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatch er.dispatch(ResourceJavaMethodDispatcher.java:75) > ~[jersey-server-1.9.jar:1.9] > > at > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.j ava:288) > ~[jersey-server-1.9.jar:1.9] > > at > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPath Rule.java:147) > ~[jersey-server-1.9.jar:1.9] > > at > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClass Rule.java:108) > ~[jersey-server-1.9.jar:1.9] > > at > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPath Rule.java:147) > ~[jersey-server-1.9.jar:1.9] > > at > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootRes ourceClassesRule.java:84) > ~[jersey-server-1.9.jar:1.9] > > at > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(We bApplicationImpl.java:1469) > ~[jersey-server-1.9.jar:1.9] > > at > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(We bApplicationImpl.java:1400) > ~[jersey-server-1.9.jar:1.9] > > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(Web ApplicationImpl.java:1349) > ~[jersey-server-1.9.jar:1.9] > > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(Web ApplicationImpl.java:1339) > ~[jersey-server-1.9.jar:1.9] > > at > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java :416) > ~[jersey-server-1.9.jar:1.9] > > at > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContai ner.java:537) > ~[jersey-server-1.9.jar:1.9] > > at > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContai ner.java:699) > ~[jersey-server-1.9.jar:1.9] > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > ~[servlet-api-2.5-20081211.jar:na] > > at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > ~[jetty-6.1.25.jar:6.1.25] > > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle r.java:1166) > ~[jetty-6.1.25.jar:6.1.25] > > at > org.neo4j.server.statistic.StatisticFilter.doFilter(StatisticFilter.java:62 ) > ~[neo4j-server-1.8.M06.jar:1.8.M06] > > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle r.java:1157) > ~[jetty-6.1.25.jar:6.1.25] > > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) > ~[jetty-6.1.25.jar:6.1.25] > > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > [jetty-6.1.25.jar:6.1.25] > > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) > [jetty-6.1.25.jar:6.1.25] > > at > org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:1 14) > [jetty-6.1.25.jar:6.1.25] > > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > [jetty-6.1.25.jar:6.1.25] > > at org.mortbay.jetty.Server.handle(Server.java:326) > [jetty-6.1.25.jar:6.1.25] > > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > [jetty-6.1.25.jar:6.1.25] > > at > org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java :943) > [jetty-6.1.25.jar:6.1.25] > > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) > [jetty-6.1.25.jar:6.1.25] > > at > org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) > [jetty-6.1.25.jar:6.1.25] > > at > org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > [jetty-6.1.25.jar:6.1.25] > > at > org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410 ) > [jetty-6.1.25.jar:6.1.25] > > at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:58 2) > [jetty-util-6.1.25.jar:6.1.25] > > Caused by: javax.transaction.SystemException: TM has encountered some > problem, please perform neccesary action (tx recovery/restart) > > at > org.neo4j.kernel.impl.transaction.TxManager.assertTmOk(TxManager.java:362) > ~[neo4j-kernel-1.8.M06.jar:1.8.M06] > > at > org.neo4j.kernel.impl.transaction.TxManager.begin(TxManager.java:339) > ~[neo4j-kernel-1.8.M06.jar:1.8.M06] > > at > org.neo4j.kernel.InternalAbstractGraphDatabase.beginTx(InternalAbstractGrap hDatabase.java:742) > ~[neo4j-kernel-1.8.M06.jar:1.8.M06] > > ... 40 common frames omitted
Looking at the messages.log, two items stick out. One is the amount of GC pauses you are seeing, you might want to look into tuning your garbage collector. Specifically, try and make sure there are available cores for the GC thread (by for instance limiting the number of threads given to jetty in the neo4j server config), it seems it is getting paused by the OS almost constantly.
The other issue, which appears to be the cause of your Transaction Manager (TM) problems, is that Lucene has broken with this exception:
I've googled to try and find anyone else seeing this issue, and haven't had any luck. One hypothesis is that the shaky environment created by the continuous GC pauses has rustled up a race condition that hasn't been noticed in Lucene before.
You can try running the Lucene checkindex tool to make sure your indexes are intact. The various indexes are stored under graph.db/index/lucene/node|relationship/. Take a look here for how to use checkindex:
Please let us know what checkindex tells you and if you were able to resolve the issue with it, it will help tracing down what caused this in the first place.
/jake
Den söndagen den 12:e augusti 2012 kl. 19:07:50 UTC+2 skrev Justin Pincar:
> Let me know if there's anything else you need. Thanks for taking a look!
> - Justin
> On Thursday, August 9, 2012 7:01:11 PM UTC-7, Michael Hunger wrote:
>> Justin,
>> could you please share your servers data/graph.db/messages.log >> as well as the logs in data/log ?
>> And please don't delete the database (yet) we might ask you for it.
>> Thanks a lot
>> Michael
>> Am 10.08.2012 um 01:08 schrieb Justin Pincar:
>> > I'm just starting with neo4j and running into some issues, and was >> hoping to get some help here.
>> > I created a graph with about 600k nodes and 11M relationships - I have >> plenty of disk space left. I have three types of nodes and two indices. My >> data import script seemed to be running fine, and then inexplicably >> starting returning 400s.
>> > I googled parts of the stack trace but couldn't find any relevant >> details. Every time I try to start the server I get the trace below.
>> > Any help would be greatly appreciated. Thanks!
>> > - Justin
>> > 21:14:04.884 [18183221@qtp-11467963-4103] ERROR org.mortbay.log - >> /db/data/node >> > org.neo4j.graphdb.TransactionFailureException: Unable to begin >> transaction >> > at >> org.neo4j.kernel.InternalAbstractGraphDatabase.beginTx(InternalAbstractGrap hDatabase.java:747) >> ~[neo4j-kernel-1.8.M06.jar:1.8.M06] >> > at >> org.neo4j.kernel.TransactionBuilderImpl.begin(TransactionBuilderImpl.java:3 9) >> ~[neo4j-kernel-1.8.M06.jar:1.8.M06] >> > at >> org.neo4j.server.rest.web.DatabaseActions.beginTx(DatabaseActions.java:228) >> ~[neo4j-server-1.8.M06.jar:1.8.M06] >> > at >> org.neo4j.server.rest.web.DatabaseActions.createNode(DatabaseActions.java:2 10) >> ~[neo4j-server-1.8.M06.jar:1.8.M06] >> > at >> org.neo4j.server.rest.web.RestfulGraphDatabase.createNode(RestfulGraphDatab ase.java:195) >> ~[neo4j-server-1.8.M06.jar:1.8.M06] >> > at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) >> ~[na:na] >> > at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp l.java:43) >> ~[na:1.6.0_20] >> > at java.lang.reflect.Method.invoke(Method.java:616) >> ~[na:1.6.0_20] >> > at >> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodIn vokerFactory.java:60) >> ~[jersey-server-1.9.jar:1.9] >> > at >> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDisp atchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchPro vider.java:205) >> ~[jersey-server-1.9.jar:1.9] at >> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatch er.dispatch(ResourceJavaMethodDispatcher.java:75) >> ~[jersey-server-1.9.jar:1.9] >> > at >> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.j ava:288) >> ~[jersey-server-1.9.jar:1.9] >> > at >> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPath Rule.java:147) >> ~[jersey-server-1.9.jar:1.9] >> > at >> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClass Rule.java:108) >> ~[jersey-server-1.9.jar:1.9] >> > at >> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPath Rule.java:147) >> ~[jersey-server-1.9.jar:1.9] >> > at >> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootRes ourceClassesRule.java:84) >> ~[jersey-server-1.9.jar:1.9] >> > at >> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(We bApplicationImpl.java:1469) >> ~[jersey-server-1.9.jar:1.9] >> > at >> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(We bApplicationImpl.java:1400) >> ~[jersey-server-1.9.jar:1.9] >> > at >> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(Web ApplicationImpl.java:1349) >> ~[jersey-server-1.9.jar:1.9] >> > at >> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(Web ApplicationImpl.java:1339) >> ~[jersey-server-1.9.jar:1.9] >> > at >> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java :416) >> ~[jersey-server-1.9.jar:1.9] >> > at >> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContai ner.java:537) >> ~[jersey-server-1.9.jar:1.9] >> > at >> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContai ner.java:699) >> ~[jersey-server-1.9.jar:1.9] >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >> ~[servlet-api-2.5-20081211.jar:na] >> > at >> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >> ~[jetty-6.1.25.jar:6.1.25] >> > at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle r.java:1166) >> ~[jetty-6.1.25.jar:6.1.25] >> > at >> org.neo4j.server.statistic.StatisticFilter.doFilter(StatisticFilter.java:62 ) >> ~[neo4j-server-1.8.M06.jar:1.8.M06] >> > at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle r.java:1157) >> ~[jetty-6.1.25.jar:6.1.25] >> > at >> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) >> ~[jetty-6.1.25.jar:6.1.25] >> > at >> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) >> [jetty-6.1.25.jar:6.1.25] >> > at >> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) >> [jetty-6.1.25.jar:6.1.25] >> > at >> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:1 14) >> [jetty-6.1.25.jar:6.1.25] >> > at >> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >> [jetty-6.1.25.jar:6.1.25] >> > at org.mortbay.jetty.Server.handle(Server.java:326) >> [jetty-6.1.25.jar:6.1.25] >> > at >> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >> [jetty-6.1.25.jar:6.1.25] >> > at >> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java :943) >> [jetty-6.1.25.jar:6.1.25] >> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) >> [jetty-6.1.25.jar:6.1.25] >> > at >> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) >> [jetty-6.1.25.jar:6.1.25] >> > at >> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >> [jetty-6.1.25.jar:6.1.25] >> > at >> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410 ) >> [jetty-6.1.25.jar:6.1.25] >> > at >> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:58 2) >> [jetty-util-6.1.25.jar:6.1.25] >> > Caused by: javax.transaction.SystemException: TM has encountered some >> problem, please perform neccesary action (tx recovery/restart) >> > at >> org.neo4j.kernel.impl.transaction.TxManager.assertTmOk(TxManager.java:362) >> ~[neo4j-kernel-1.8.M06.jar:1.8.M06] >> > at >> org.neo4j.kernel.impl.transaction.TxManager.begin(TxManager.java:339) >> ~[neo4j-kernel-1.8.M06.jar:1.8.M06] >> > at >> org.neo4j.kernel.InternalAbstractGraphDatabase.beginTx(InternalAbstractGrap hDatabase.java:742) >> ~[neo4j-kernel-1.8.M06.jar:1.8.M06] >> > ... 40 common frames omitted
Thanks Jake, I'll try these things tonight and let you know how it turns
out.
- Justin
On Aug 28, 2012 3:12 AM, "Jacob Hansson" <jacob.hans...@neotechnology.com>
wrote:
> Looking at the messages.log, two items stick out. One is the amount of GC
> pauses you are seeing, you might want to look into tuning your garbage
> collector. Specifically, try and make sure there are available cores for
> the GC thread (by for instance limiting the number of threads given to
> jetty in the neo4j server config), it seems it is getting paused by the OS
> almost constantly.
> The other issue, which appears to be the cause of your Transaction Manager
> (TM) problems, is that Lucene has broken with this exception:
> I've googled to try and find anyone else seeing this issue, and haven't
> had any luck. One hypothesis is that the shaky environment created by
> the continuous GC pauses has rustled up a race condition that hasn't been
> noticed in Lucene before.
> You can try running the Lucene checkindex tool to make sure your indexes
> are intact. The various indexes are stored under
> graph.db/index/lucene/node|relationship/. Take a look here for how to use
> checkindex:
> Please let us know what checkindex tells you and if you were able to
> resolve the issue with it, it will help tracing down what caused this in
> the first place.
> /jake
> Den söndagen den 12:e augusti 2012 kl. 19:07:50 UTC+2 skrev Justin Pincar:
>> Let me know if there's anything else you need. Thanks for taking a look!
>> - Justin
>> On Thursday, August 9, 2012 7:01:11 PM UTC-7, Michael Hunger wrote:
>>> Justin,
>>> could you please share your servers data/graph.db/messages.log
>>> as well as the logs in data/log ?
>>> And please don't delete the database (yet) we might ask you for it.
>>> Thanks a lot
>>> Michael
>>> Am 10.08.2012 um 01:08 schrieb Justin Pincar:
>>> > I'm just starting with neo4j and running into some issues, and was
>>> hoping to get some help here.
>>> > I created a graph with about 600k nodes and 11M relationships - I have
>>> plenty of disk space left. I have three types of nodes and two indices. My
>>> data import script seemed to be running fine, and then inexplicably
>>> starting returning 400s.
>>> > I googled parts of the stack trace but couldn't find any relevant
>>> details. Every time I try to start the server I get the trace below.
>>> > Any help would be greatly appreciated. Thanks!
>>> > - Justin
>>> > 21:14:04.884 [18183221@qtp-11467963-4103] ERROR org.mortbay.log -
>>> /db/data/node
>>> > org.neo4j.graphdb.**TransactionFailureException: Unable to begin
>>> transaction
>>> > at org.neo4j.kernel.**InternalAbstractGraphDatabase.**beginTx(
>>> **InternalAbstractGraphDatabase.**java:747)
>>> ~[neo4j-kernel-1.8.M06.jar:1.**8.M06]
>>> > at org.neo4j.kernel.**TransactionBuilderImpl.begin(**
>>> TransactionBuilderImpl.java:**39) ~[neo4j-kernel-1.8.M06.jar:1.**8.M06]
>>> > at org.neo4j.server.rest.web.**DatabaseActions.beginTx(**DatabaseActions.java: 228)
>>> ~[neo4j-server-1.8.M06.jar:1.**8.M06]
>>> > at org.neo4j.server.rest.web.**DatabaseActions.createNode(**DatabaseActions.ja va:210)
>>> ~[neo4j-server-1.8.M06.jar:1.**8.M06]
>>> > at org.neo4j.server.rest.web.**RestfulGraphDatabase.**
>>> createNode(**RestfulGraphDatabase.java:195)
>>> ~[neo4j-server-1.8.M06.jar:1.**8.M06]
>>> > at sun.reflect.**GeneratedMethodAccessor83.**invoke(Unknown
>>> Source) ~[na:na]
>>> > at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
>>> DelegatingMethodAccessorImpl.**java:43) ~[na:1.6.0_20]
>>> > at java.lang.reflect.Method.**invoke(Method.java:616)
>>> ~[na:1.6.0_20]
>>> > at com.sun.jersey.spi.container.**JavaMethodInvokerFactory$1.*
>>> *invoke(**JavaMethodInvokerFactory.java:**60)
>>> ~[jersey-server-1.9.jar:1.9]
>>> > at com.sun.jersey.server.impl.**model.method.dispatch.**
>>> AbstractResourceMethodDispatch**Provider$ResponseOutInvoker._**dispatch(
>>> **AbstractResourceMethodDispatch**Provider.java:205)
>>> ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.server.impl.**
>>> model.method.dispatch.**ResourceJavaMethodDispatcher.**dispatch(**
>>> ResourceJavaMethodDispatcher.**java:75) ~[jersey-server-1.9.jar:1.9]
>>> > at com.sun.jersey.server.impl.**uri.rules.HttpMethodRule.**
>>> accept(HttpMethodRule.java:**288) ~[jersey-server-1.9.jar:1.9]
>>> > at com.sun.jersey.server.impl.**uri.rules.RightHandPathRule.**
>>> accept(RightHandPathRule.java:**147) ~[jersey-server-1.9.jar:1.9]
>>> > at com.sun.jersey.server.impl.**uri.rules.ResourceClassRule.**
>>> accept(ResourceClassRule.java:**108) ~[jersey-server-1.9.jar:1.9]
>>> > at com.sun.jersey.server.impl.**uri.rules.RightHandPathRule.**
>>> accept(RightHandPathRule.java:**147) ~[jersey-server-1.9.jar:1.9]
>>> > at com.sun.jersey.server.impl.**uri.rules.**
>>> RootResourceClassesRule.**accept(**RootResourceClassesRule.java:**84)
>>> ~[jersey-server-1.9.jar:1.9]
>>> > at com.sun.jersey.server.impl.**application.**
>>> WebApplicationImpl._**handleRequest(**WebApplicationImpl.java:1469)
>>> ~[jersey-server-1.9.jar:1.9]
>>> > at com.sun.jersey.server.impl.**application.**
>>> WebApplicationImpl._**handleRequest(**WebApplicationImpl.java:1400)
>>> ~[jersey-server-1.9.jar:1.9]
>>> > at com.sun.jersey.server.impl.**application.**
>>> WebApplicationImpl.**handleRequest(**WebApplicationImpl.java:1349)
>>> ~[jersey-server-1.9.jar:1.9]
>>> > at com.sun.jersey.server.impl.**application.**
>>> WebApplicationImpl.**handleRequest(**WebApplicationImpl.java:1339)
>>> ~[jersey-server-1.9.jar:1.9]
>>> > at com.sun.jersey.spi.container.**
>>> servlet.WebComponent.service(**WebComponent.java:416)
>>> ~[jersey-server-1.9.jar:1.9]
>>> > at com.sun.jersey.spi.container.**servlet.ServletContainer.**
>>> service(ServletContainer.java:**537) ~[jersey-server-1.9.jar:1.9]
>>> > at com.sun.jersey.spi.container.**servlet.ServletContainer.**
>>> service(ServletContainer.java:**699) ~[jersey-server-1.9.jar:1.9]
>>> > at javax.servlet.http.**HttpServlet.service(**HttpServlet.java:820)
>>> ~[servlet-api-2.5-20081211.**jar:na]
>>> > at org.mortbay.jetty.servlet.**ServletHolder.handle(**ServletHolder.java:511)
>>> ~[jetty-6.1.25.jar:6.1.25]
>>> > at org.mortbay.jetty.servlet.**ServletHandler$CachedChain.**
>>> doFilter(ServletHandler.java:**1166) ~[jetty-6.1.25.jar:6.1.25]
>>> > at org.neo4j.server.statistic.**StatisticFilter.doFilter(**StatisticFilter.jav a:62)
>>> ~[neo4j-server-1.8.M06.jar:1.**8.M06]
>>> > at org.mortbay.jetty.servlet.**ServletHandler$CachedChain.**
>>> doFilter(ServletHandler.java:**1157) ~[jetty-6.1.25.jar:6.1.25]
>>> > at org.mortbay.jetty.servlet.**ServletHandler.handle(**ServletHandler.java:388 )
>>> ~[jetty-6.1.25.jar:6.1.25]
>>> > at org.mortbay.jetty.servlet.**SessionHandler.handle(**SessionHandler.java:182 )
>>> [jetty-6.1.25.jar:6.1.25]
>>> > at org.mortbay.jetty.handler.**ContextHandler.handle(**ContextHandler.java:765 )
>>> [jetty-6.1.25.jar:6.1.25]
>>> > at org.mortbay.jetty.handler.**HandlerCollection.handle(**HandlerCollection.ja va:114)
>>> [jetty-6.1.25.jar:6.1.25]
>>> > at org.mortbay.jetty.handler.**HandlerWrapper.handle(**HandlerWrapper.java:152 )
>>> [jetty-6.1.25.jar:6.1.25]
>>> > at org.mortbay.jetty.Server.**handle(Server.java:326)
>>> [jetty-6.1.25.jar:6.1.25]
>>> > at org.mortbay.jetty.**HttpConnection.handleRequest(**HttpConnection.java:542)
>>> [jetty-6.1.25.jar:6.1.25]
>>> > at org.mortbay.jetty.**HttpConnection$RequestHandler.**
>>> content(HttpConnection.java:**943) [jetty-6.1.25.jar:6.1.25]
>>> > at
The garbage collection seems very likely - I switched to a higher memory
instance and things seem like they're going faster and smoother.
The index seems to be fine, output below.
At this point I'm just moving on with the new setup and making sure to
backup frequently, so I'm not going to worry about this issue too much.
Thanks for all the help!
> Thanks Jake, I'll try these things tonight and let you know how it turns
> out.
> - Justin
> On Aug 28, 2012 3:12 AM, "Jacob Hansson" <jacob.hans...@neotechnology.com>
> wrote:
>> Hey Justin, thanks for uploading the logs.
>> Looking at the messages.log, two items stick out. One is the amount of GC
>> pauses you are seeing, you might want to look into tuning your garbage
>> collector. Specifically, try and make sure there are available cores for
>> the GC thread (by for instance limiting the number of threads given to
>> jetty in the neo4j server config), it seems it is getting paused by the OS
>> almost constantly.
>> The other issue, which appears to be the cause of your Transaction
>> Manager (TM) problems, is that Lucene has broken with this exception:
>> I've googled to try and find anyone else seeing this issue, and haven't
>> had any luck. One hypothesis is that the shaky environment created by
>> the continuous GC pauses has rustled up a race condition that hasn't been
>> noticed in Lucene before.
>> You can try running the Lucene checkindex tool to make sure your indexes
>> are intact. The various indexes are stored under
>> graph.db/index/lucene/node|relationship/. Take a look here for how to use
>> checkindex:
>> Please let us know what checkindex tells you and if you were able to
>> resolve the issue with it, it will help tracing down what caused this in
>> the first place.
>> /jake
>> Den söndagen den 12:e augusti 2012 kl. 19:07:50 UTC+2 skrev Justin Pincar:
>>> Let me know if there's anything else you need. Thanks for taking a look!
>>> - Justin
>>> On Thursday, August 9, 2012 7:01:11 PM UTC-7, Michael Hunger wrote:
>>>> Justin,
>>>> could you please share your servers data/graph.db/messages.log
>>>> as well as the logs in data/log ?
>>>> And please don't delete the database (yet) we might ask you for it.
>>>> Thanks a lot
>>>> Michael
>>>> Am 10.08.2012 um 01:08 schrieb Justin Pincar:
>>>> > I'm just starting with neo4j and running into some issues, and was
>>>> hoping to get some help here.
>>>> > I created a graph with about 600k nodes and 11M relationships - I
>>>> have plenty of disk space left. I have three types of nodes and two
>>>> indices. My data import script seemed to be running fine, and then
>>>> inexplicably starting returning 400s.
>>>> > I googled parts of the stack trace but couldn't find any relevant
>>>> details. Every time I try to start the server I get the trace below.
>>>> > Any help would be greatly appreciated. Thanks!
>>>> > - Justin
>>>> > 21:14:04.884 [18183221@qtp-11467963-4103] ERROR org.mortbay.log -
>>>> /db/data/node
>>>> > org.neo4j.graphdb.**TransactionFailureException: Unable to begin
>>>> transaction
>>>> > at org.neo4j.kernel.**InternalAbstractGraphDatabase.**
>>>> beginTx(**InternalAbstractGraphDatabase.**java:747)
>>>> ~[neo4j-kernel-1.8.M06.jar:1.**8.M06]
>>>> > at org.neo4j.kernel.**TransactionBuilderImpl.begin(**
>>>> TransactionBuilderImpl.java:**39) ~[neo4j-kernel-1.8.M06.jar:1.**8.M06]
>>>> > at org.neo4j.server.rest.web.**DatabaseActions.beginTx(**DatabaseActions.java: 228)
>>>> ~[neo4j-server-1.8.M06.jar:1.**8.M06]
>>>> > at org.neo4j.server.rest.web.**DatabaseActions.createNode(**DatabaseActions.ja va:210)
>>>> ~[neo4j-server-1.8.M06.jar:1.**8.M06]
>>>> > at org.neo4j.server.rest.web.**RestfulGraphDatabase.**
>>>> createNode(**RestfulGraphDatabase.java:195)
>>>> ~[neo4j-server-1.8.M06.jar:1.**8.M06]
>>>> > at sun.reflect.**GeneratedMethodAccessor83.**invoke(Unknown
>>>> Source) ~[na:na]
>>>> > at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
>>>> DelegatingMethodAccessorImpl.**java:43) ~[na:1.6.0_20]
>>>> > at java.lang.reflect.Method.**invoke(Method.java:616)
>>>> ~[na:1.6.0_20]
>>>> > at com.sun.jersey.spi.container.**JavaMethodInvokerFactory$1.
>>>> **invoke(**JavaMethodInvokerFactory.java:**60)
>>>> ~[jersey-server-1.9.jar:1.9]
>>>> > at com.sun.jersey.server.impl.**model.method.dispatch.**
>>>> AbstractResourceMethodDispatch**Provider$ResponseOutInvoker._**
>>>> dispatch(**AbstractResourceMethodDispatch**Provider.java:205)
>>>> ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.server.impl.**
>>>> model.method.dispatch.**ResourceJavaMethodDispatcher.**dispatch(**
>>>> ResourceJavaMethodDispatcher.**java:75) ~[jersey-server-1.9.jar:1.9]
>>>> > at com.sun.jersey.server.impl.**uri.rules.HttpMethodRule.**
>>>> accept(HttpMethodRule.java:**288) ~[jersey-server-1.9.jar:1.9]
>>>> > at com.sun.jersey.server.impl.**uri.rules.RightHandPathRule.*
>>>> *accept(RightHandPathRule.java:**147) ~[jersey-server-1.9.jar:1.9]
>>>> > at com.sun.jersey.server.impl.**uri.rules.ResourceClassRule.*
>>>> *accept(ResourceClassRule.java:**108) ~[jersey-server-1.9.jar:1.9]
>>>> > at com.sun.jersey.server.impl.**uri.rules.RightHandPathRule.*
>>>> *accept(RightHandPathRule.java:**147) ~[jersey-server-1.9.jar:1.9]
>>>> > at com.sun.jersey.server.impl.**uri.rules.**
>>>> RootResourceClassesRule.**accept(**RootResourceClassesRule.java:**84)
>>>> ~[jersey-server-1.9.jar:1.9]
>>>> > at com.sun.jersey.server.impl.**application.**
>>>> WebApplicationImpl._**handleRequest(**WebApplicationImpl.java:1469)
>>>> ~[jersey-server-1.9.jar:1.9]
>>>> > at com.sun.jersey.server.impl.**application.**
>>>> WebApplicationImpl._**handleRequest(**WebApplicationImpl.java:1400)
>>>> ~[jersey-server-1.9.jar:1.9]
>>>> > at