Performance/Limits of closing a transaction after big insert

[José Cornado]

Hello!

I am running int the following:

After inserting around 30,000 nodes to a graph, I close the operation with transaction success. But the neo returns:

Unable to commit transaction.

Is there a hard limit on the size of a transaction? Doing it on a per node basis is too slow.

Thanks a lot!!!

[Michael Hunger]

Can you share your code and the full exception? As well as the graph.db/messages.log file

Thx

Sent from mobile device

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[José Cornado]

Let .me reproduce it again. I was able to wrap 29,000 in a transaction (side note)

--
You received this message because you are subscribed to a topic in the Google Groups "Neo4j" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to neo4j+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
José Cornado

--

home: http://www.efekctive.com
blog:   http://blogging.efekctive.com
----------------------

Everything has been said before, but since nobody listens we have to keep going back and beginning all over again.

Andre Gide

[José Cornado]

It is crashing before it gets where the original exception occurred.

java.lang.OutOfMemoryError: GC overhead limit exceeded

at org.neo4j.kernel.impl.api.KernelTransactionImplementation.acquireStatement(KernelTransactionImplementation.java:182)

at org.neo4j.kernel.impl.api.KernelTransactionImplementation.acquireStatement(KernelTransactionImplementation.java:63)

at org.neo4j.kernel.impl.core.ThreadToStatementContextBridge.instance(ThreadToStatementContextBridge.java:47)

at org.neo4j.kernel.impl.core.NodeProxy.addLabel(NodeProxy.java:468)

at MY LOGIC

at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:220)

at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)

at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4166)

at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1466)

at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1489)

at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1474)

at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:1279)

at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4012)

at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3651)

at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1113)

at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)

at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:997)

at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:138)

at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:610)

at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)

at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:567)

at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150)

at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:124)

at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)

at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)

at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)

at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:354)

[Michael Hunger]

Sorry, can't see where you shared your code / queries / statements.

Michael

<messages.log>

[José Cornado]

I managed to reproduce the same exception.

Trace and log file are included. I will put together a test case in a few

(org.neo4j.graphdb.TransactionFailureException) org.neo4j.graphdb.TransactionFailureException: Unable to commit transaction

Caused by: org.neo4j.graphdb.TransactionFailureException: commit threw exception

at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:498)

at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:397)

at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:122)

at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:124)

at endTransaction(...)

... 31 more

Caused by: javax.transaction.xa.XAException

at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:553)

at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:460)

... 38 more

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

at org.apache.lucene.util.BytesRef.<init>(BytesRef.java:77)

at org.apache.lucene.store.DataOutput.writeString(DataOutput.java:111)

at org.apache.lucene.index.FieldsWriter.writeField(FieldsWriter.java:212)

at org.apache.lucene.index.StoredFieldsWriterPerThread.addField(StoredFieldsWriterPerThread.java:58)

at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:265)

at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:766)

at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060)

at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2034)

at org.neo4j.kernel.api.impl.index.LuceneIndexAccessor.add(LuceneIndexAccessor.java:151)

at org.neo4j.kernel.api.impl.index.LuceneIndexAccessor$LuceneIndexUpdater.process(LuceneIndexAccessor.java:186)

at org.neo4j.kernel.impl.api.index.FlippableIndexProxy$LockingIndexUpdater.process(FlippableIndexProxy.java:337)

at org.neo4j.kernel.impl.api.index.ContractCheckingIndexProxy$1.process(ContractCheckingIndexProxy.java:102)

at org.neo4j.kernel.impl.api.index.IndexingService.processUpdateIfIndexExists(IndexingService.java:411)

at org.neo4j.kernel.impl.api.index.IndexingService.applyUpdates(IndexingService.java:359)

at org.neo4j.kernel.impl.api.index.IndexingService.updateIndexes(IndexingService.java:310)

at org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.applyCommit(WriteTransaction.java:817)

at org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.doCommit(WriteTransaction.java:751)

at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:322)

at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commitWriteTx(XaResourceManager.java:530)

at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:446)

at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64)

at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:545)

at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:460)

at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:397)

at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:122)

at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:124)

-> endTransaction(...)

[Michael Hunger]

Make sure your tx is not too big for your memory, according to your messages log your JVM only runs with a few megabytes of memory.

Try to limit your tx-size to 10k or 20k elements (nodes and rels) restarting the tx after calling 

if (count++ == 10000) {

tx.success(); tx.close();

tx = db.beginTx();

count = 0;

}

Cheers,

Michael

<messages.log>

[José Cornado]

that I know. What is the cost per element (node/rels) in a transaction? few kb?

[José Cornado]

The data is fed to an engine that is able to process work in the millions. The number of relations is really low compared to the number of nodes so the cost of node per transaction is the driver.

Thanks!

[Michael Hunger]

In my experience a it depends on what you do, I had good results with tx-sizes from 1k to 30k

But if you need to import millions at once, you might want to look into batch-insertion, see http://neo4j.org/develop/import

Cheers,

Michael

[José Cornado]

I am not importing. I am creating. These were very simple (one value == one node) It can be more complex than that. Where can I find docs about the memory usage and setup?

[José Cornado]

FYI,

I ran a little, crude experiment. I increased the -Xmx (max heap size) option from 512 to 1024 mb while incrementing the number of nodes added to the graph. It crashed around 78K nodes.

So a crude guideline to use is (78k - 30K)/512MB. In case you want to minimize the number of transactions used.

[Michael Hunger]

Yep, makes sense, in general it is sensible to stay in the 30k range so you get the best utilization between tx-commit disk-flushes and memory usage of large tx.

See: 

http://jexp.de/blog/2013/05/on-importing-data-in-neo4j-blog-series/

[José Cornado]

ok.