Axel Morgner
CEO Structr (c/o Morgner UG) · Hanauer Landstr. 291a · 60314
Frankfurt · Germany
Twitter: @amorgner
Phone: +49 151 40522060
Skype: axel.morgner
Structr - Award-Winning Open
Source CMS and Web Framework based on Neo4j
Structr
Mailing List and Forum
Graph
Database Usergroup "graphdb-frankfurt"
--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
This makes me want to dive into the ext4 code and see how they do barriers and journalling.
I wonder if we need journalling, since Neo4j has its own transaction log that it will replay upon recovery.
Disabling barriers is a bit more worrying. I don�t have a deep understanding of ext4 internals, but I�m guessing that they prevent NCQ from reordering commands across the barrier. Specifically, they prevent store file writes from reordering before�the transaction log commit command. In other words, they make sure that store files don�t contain data that never got committed. Furthermore, in the case of a power failure, the drive might be receiving junk data. If the junk goes to the store file, then we�d like our transaction log to repair the store file, and if the junk goes to the transaction log, then we would like to simply disregard it and consider any orphaned�commands as uncommitted.
I�m a little surprised that the effect of disabling barriers is so pronounced. I wonder how the SSD compare to the HDD when barriers are enabled.
--Chris Vest
[ skype: mr.chrisvest, twitter: chvest ]
On 10 Dec 2013, at 21:55, Axel Morgner <ax...@morgner.de> wrote:
Hi,
maybe some of you have experienced poor write performance on their Linux boxes as I did, esp. with small transactions.
In my tests I was able to increase the throughput by a factor of 15! Here's a blog post about my findings:
http://structr.org/blog/neo4j-performance-on-ext4
Comments?
Best
Axel
--
Axel Morgner
CEO Structr (c/o Morgner UG) � Hanauer Landstr. 291a � 60314 Frankfurt � Germany
Twitter: @amorgner
Phone: +49 151 40522060
Skype: axel.morgner
Structr - Award-Winning Open Source CMS and Web Framework based on Neo4j
Structr Mailing List and Forum
Graph Database Usergroup "graphdb-frankfurt"
--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Axel Morgner
CEO Structr (c/o Morgner UG) � Hanauer Landstr. 291a � 60314
Frankfurt � Germany
Twitter: @amorgner
Phone: +49 151 40522060
Skype: axel.morgner
Disabling barriers is a bit more worrying. I don’t have a deep understanding of ext4 internals, but I’m guessing that they prevent NCQ from reordering commands across the barrier. Specifically, they prevent store file writes from reordering before the transaction log commit command. In other words, they make sure that store files don’t contain data that never got committed. Furthermore, in the case of a power failure, the drive might be receiving junk data. If the junk goes to the store file, then we’d like our transaction log to repair the store file, and if the junk goes to the transaction log, then we would like to simply disregard it and consider any orphaned commands as uncommitted.
I’m a little surprised that the effect of disabling barriers is so pronounced. I wonder how the SSD compare to the HDD when barriers are enabled.
--Chris Vest
[ skype: mr.chrisvest, twitter: chvest ]
On 10 Dec 2013, at 21:55, Axel Morgner <ax...@morgner.de> wrote:
Hi,
maybe some of you have experienced poor write performance on their Linux boxes as I did, esp. with small transactions.
In my tests I was able to increase the throughput by a factor of 15! Here's a blog post about my findings:
http://structr.org/blog/neo4j-performance-on-ext4
Comments?
Best
Axel
--
Axel Morgner
CEO Structr (c/o Morgner UG) · Hanauer Landstr. 291a · 60314 Frankfurt · Germany
Twitter: @amorgner
Phone: +49 151 40522060
Skype: axel.morgner
Structr - Award-Winning Open Source CMS and Web Framework based on Neo4j
Structr Mailing List and Forum
Graph Database Usergroup "graphdb-frankfurt"
--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
Axel Morgner
CEO Structr (c/o Morgner UG) · Hanauer Landstr. 291a · 60314 Frankfurt · Germany
Twitter: @amorgner
Phone: +49 151 40522060
Skype: axel.morgner
Structr - Award-Winning Open Source CMS and Web Framework based on Neo4j
Structr Mailing List and Forum
Graph Database Usergroup "graphdb-frankfurt"
In principle, Neo4j should not cause meta-data changes, e.g. it should use fdatasync instead of fsync and similar, such that the in ode tree don�t need updating during normal operation. In practice I would be hesitant to rely on this. For instance, I don�t know at this point, if the force() calls we do to the MappedByteBuffers translate into msync() calls, which do update metadata, or if there are other metadata-updating file operations hiding in there. But what�s worse is that Lucene probably does a lot of fiddling about with files, causing metadata updates. Actually, the fact that such large gains can be had by turning journalling off might be indicative that we do a lot of metadata updates.
I guess you can argue that a UPS backed cluster of machines would be safe, since it allows you time to rebuild (rather than restart) any machine that might crash. It would certainly lower the probability of data loss either way.
I haven�t looked into raw devices. The trouble is the file and IO APIs we�re given. I�ve briefly looked at the Linux block device APIs. They look more in tune with the nature of NCQ and ATA, which is nice, but I�m not sure what�s available in user space. Also, we need to support OS X and Windows.
Disabling barriers is a bit more worrying. I don�t have a deep understanding of ext4 internals, but I�m guessing that they prevent NCQ from reordering commands across the barrier. Specifically, they prevent store file writes from reordering before�the transaction log commit command. In other words, they make sure that store files don�t contain data that never got committed. Furthermore, in the case of a power failure, the drive might be receiving junk data. If the junk goes to the store file, then we�d like our transaction log to repair the store file, and if the junk goes to the transaction log, then we would like to simply disregard it and consider any orphaned�commands as uncommitted.
I�m a little surprised that the effect of disabling barriers is so pronounced. I wonder how the SSD compare to the HDD when barriers are enabled.
--Chris Vest
[ skype: mr.chrisvest, twitter: chvest ]
On 10 Dec 2013, at 21:55, Axel Morgner <ax...@morgner.de> wrote:
Hi,
maybe some of you have experienced poor write performance on their Linux boxes as I did, esp. with small transactions.
In my tests I was able to increase the throughput by a factor of 15! Here's a blog post about my findings:
http://structr.org/blog/neo4j-performance-on-ext4
Comments?
Best
Axel
--
Axel Morgner
CEO Structr (c/o Morgner UG) � Hanauer Landstr. 291a � 60314 Frankfurt � Germany
Twitter: @amorgner
Phone: +49 151 40522060
Skype: axel.morgner
Structr - Award-Winning Open Source CMS and Web Framework based on Neo4j
Structr Mailing List and Forum
Graph Database Usergroup "graphdb-frankfurt"
--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
Axel Morgner
CEO Structr (c/o Morgner UG) � Hanauer Landstr. 291a � 60314 Frankfurt � Germany
Twitter: @amorgner
Phone: +49 151 40522060
Skype: axel.morgner
Structr - Award-Winning Open Source CMS and Web Framework based on Neo4j
Structr Mailing List and Forum
Graph Database Usergroup "graphdb-frankfurt"
--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Axel Morgner
CEO Structr (c/o Morgner UG) � Hanauer Landstr. 291a � 60314
Frankfurt � Germany
Twitter: @amorgner
Phone: +49 151 40522060
Skype: axel.morgner