Too many open files
Matt Chaput
Well, there's hope for the future, because I finally wised up to the whole buffer/memoryview thing, so in the next release (by default) each segment will be one read-only memory mapped file which can be shared between threads, instead of 5-7 open files (+ mmaps) per segment per thread.
It's a good thing there's so much I don't understand, it means there's always ways to improve
;)
Cheers,
Matt
Roger Binns
Hash: SHA1
On 16/12/11 15:14, Matt Chaput wrote:
> So, um, did everyone else know that creating a mmap object counts as an
> extra open file? Because I didn't (*sheepish*).
It has to keep a reference to the file so that the file is considered in
use, otherwise it could be deleted from underneath the memory map.
BTW you may want to look at what SQLite does with its write ahead logging.
Effectively changes are aggregated in a separate journal file. Clients
continue to read the main file and up to a point in the journal.
Eventually a checkpoint is done when everything gets locked and the
batched up changes in the journal are written into the main file and the
journal reset.
The net effect is that writers have little effect on readers, I/O is
sequential, and the duration of locks is really short. There are also a
total of two files involved at any point.
http://www.sqlite.org/wal.html
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
iEYEARECAAYFAk7r2m4ACgkQmOOfHg372QS2mgCguuWX3wA9sD3x1Rt3cCGPLAKn
zuoAoIYY3y+dIGpKiM0/NCfivVQw9+BI
=pHpE
-----END PGP SIGNATURE-----
Matt Chaput
> BTW you may want to look at what SQLite does with its write ahead logging.
> Effectively changes are aggregated in a separate journal file. Clients
> continue to read the main file and up to a point in the journal.
I am thinking along these lines... I think there might be a way to do lockless writing. Thanks for the link, I'll check it out :)
Matt
Roger Binns
Hash: SHA1
On 16/12/11 16:06, Matt Chaput wrote:
> I think there might be a way to do lockless writing.
Only if you don't care about data integrity. You can certainly get most
of the way there by doing atomic appends to a file, but at some point
you'll need to stop the world (aka lock) in order to rearrange things and
do garbage collection.
Personally I'm quite happy with improved performance with the tradeoff
being increased probability of data loss/corruption. The only need is to
know it has happened so I can restart and repopulate.
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
iEYEARECAAYFAk7sDpUACgkQmOOfHg372QQVVgCeLbcVYpcdSjC1W4kM+6Kp5Ot7
1ZkAn1BFVDuM/9+cRxNzFLYrWDNvSdJB
=0KD3
-----END PGP SIGNATURE-----
Thomas Waldmann
Just wanted to note that.
The problem is known, but rather embarassing if you relied on atomic appends in your code and then some user runs the stuff on NFS and gets corrupted file content.
Roger Binns
Hash: SHA1
On 17/03/12 19:39, Thomas Waldmann wrote:
> About "atomic append": there is no such thing on NFS.
The simple rule is to never run a database or anything substantially
database like over NFS anyway. Atomic append is just one of the many
things broken.
I'd advice just saying "do not do that" rather than trying to work around
it. Put something in the FAQ. Unfortunately it will appear to work most
of the time.
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
iEYEARECAAYFAk9lS2IACgkQmOOfHg372QScQwCeKN77ADwfUFLE6dPIlJtFEByp
pwEAoMyjYFsOsGdCtL+yyAIGDJvz/w1i
=iTWU
-----END PGP SIGNATURE-----