Q4M 0.9.2 prerelease available fixing data corruption on 32bit systems
Kazuho Oku
Thanks to a user of Q4M, I have found a bug that would likely lead to
data corruption on 32bit versions of Q4M. 64bit versions are
anaffected.
Q4M by default uses mmap(2) to read from data files. On 32bit
systems, it tries to map max. 1GB per each table into memory using
mmap. When mmap fails to map memory due to low memory, Q4M falls back
to file I/O to read the data.
However there was a bug in handling the response from mmap, that led
to reading corrupt data from database files when mmap(2) failed after
the size of the underlying file was grown / shrunk by Q4M. And since
Q4M writes back the corrupt data into the database file when rows are
being consumed, the bug will likely destroy the database files.
I have fixed the bug and have uploaded Q4M 0.9.2, into the prerelease
directory at http://q4m.31tools.com/dist/pre
If you are using 32bit versions of Q4M, I highly recommend either to
update to 0.9.2 or switch to 64bit versions if possible.
BTW in 0.9.2 release, I also changed the maximum mmap size per table
from 1GB to 256MB, to lower the possiblity of low memory. However
this countermeasure might not be sufficient in some cases, i.e.
databases having many Q4M tables or if other storage engines also used
a lot of memory. I am considering of just disable the use of mmap(2)
on 32bit systems in future releases. If you have any comments on
this, please let me know.
--
Kazuho Oku