Logical Log write failed with no space left on device when there is plenty of disk space

[poly...@gmail.com]

I am running neo4j 2.1.6 embedded. Under load, I ran into "org.neo4j.graphdb.TransactionFailureException: Unable to write command to logical log" several times. The root cause says 'No space left on device' while there is plenty of disk space available. 

The logical log file size is the default 25MB and 10 files are kept. I noticed that file rotation occurred every 2 minutes.

Any idea why this error occurred? Any recommendations for configure change/tuning? 

Caused by: javax.transaction.xa.XAException: null

at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:560) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:448) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:385) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:123) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.transaction.BaseSpringTransactionImpl.commit(BaseSpringTransactionImpl.java:66) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1011) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]

at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:755) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]

at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:724) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]

at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:475) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]

at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:270) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]

at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.2.4.RELEASE.jar:3.2.4.RELEASE]

at com.ryantenney.metrics.spring.MeteredMethodInterceptor.invoke(MeteredMethodInterceptor.java:45) ~[metrics-spring-3.0.1.jar:na]

at com.ryantenney.metrics.spring.MeteredMethodInterceptor.invoke(MeteredMethodInterceptor.java:32) ~[metrics-spring-3.0.1.jar:na]

at com.ryantenney.metrics.spring.AbstractMetricMethodInterceptor.invoke(AbstractMetricMethodInterceptor.java:59) ~[metrics-spring-3.0.1.jar:na]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.2.4.RELEASE.jar:3.2.4.RELEASE]

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) ~[spring-aop-3.2.4.RELEASE.jar:3.2.4.RELEASE]

at com.sun.proxy.$Proxy68.setIntrinsicState(Unknown Source) ~[na:na]

at com.zenoss.dsa.impl.StateChangeProcessorImpl.processStateEvents(StateChangeProcessorImpl.java:51) ~[classes/:na]

... 23 common frames omitted

Caused by: org.neo4j.graphdb.TransactionFailureException: Unable to write command to logical log.

at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.addCommand(XaTransaction.java:254) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.doPrepare(NeoStoreTransaction.java:322) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.prepare(XaTransaction.java:307) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commitWriteTx(XaResourceManager.java:588) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:533) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:548) ~[neo4j-kernel-2.1.6.jar:2.1.6]

... 41 common frames omitted

Caused by: java.io.IOException: No space left on device

at java.nio.MappedByteBuffer.force0(Native Method) ~[na:1.7.0_75]

at java.nio.MappedByteBuffer.force(MappedByteBuffer.java:203) ~[na:1.7.0_75]

at org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow.force(MappedPersistenceWindow.java:91) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:319) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.flushAll(CommonAbstractStore.java:562) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.nioneo.store.PropertyStore.flushAll(PropertyStore.java:146) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.nioneo.store.NeoStore.flushAll(NeoStore.java:315) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.forceEverything(NeoStoreXaDataSource.java:522) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.access$1200(NeoStoreXaDataSource.java:119) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource$TransactionFactory.flushAll(NeoStoreXaDataSource.java:592) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.rotate(XaLogicalLog.java:1167) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.checkLogRotation(XaLogicalLog.java:532) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.writeCommand(XaLogicalLog.java:501) ~[neo4j-kernel-2.1.6.jar:2.1.6]

at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.addCommand(XaTransaction.java:250) ~[neo4j-kernel-2.1.6.jar:2.1.6]

[Chris Vest]

“No space left on device” is what the operating system gives us as reason for failing a write. Maybe you are running into quota limits, or the particular partition of the drive is getting full? Also note that Neo4j wants to create the next log file and complete the rotation, before it looks for old log files to delete.

--

Chris Vest

System Engineer, Neo Technology
[ skype: mr.chrisvest, twitter: chvest ]

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[poly...@gmail.com]

It is really strange. The disk or the partition is nowhere near being full. The open file limits on the system is set to a really high number as well. Any other quota I should check?

Any chance when the current logical log is approaching its size limit 25MB, if the last commit is too big, the error is raise?

Thanks. 

[Christian Morgner]

The "no space left on device" error can occur when there are many very
small files on your disk, since a file will always consume a certain
min. amount of space, but the disk usage utilities will report the
actual size, not the inode size.

What operating system and filesystem type do you use?


Best,
Christian

>> send an email to neo4j+un...@googlegroups.com <javascript:>.

>> For more options, visit https://groups.google.com/d/optout

>> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to neo4j+un...@googlegroups.com

> <mailto:neo4j+un...@googlegroups.com>.

[poly...@gmail.com]

Thanks Christian.

The OS is ubuntu 14.04 and file system for the neo4j storage partition is btrfs.

The partition size is 1.1TB and 625GB is used.

[Chris Vest]

Looks like Btrfs has a history of running into ENOSPC problems. Google turns up a lot of results where Btrfs runs out of space in situations where most other file systems would not.

Also try enabling the `enospc_debug` mount option and see if anything useful gets written to the syslog next time you run into the problem.

--

Chris Vest

System Engineer, Neo Technology
[ skype: mr.chrisvest, twitter: chvest ]

To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.

[poly...@gmail.com]

I think you are right. Most likely it is btrfs. Will add the mount option you recommended.

Thanks!