tokudb crashing on start after unclean reboot

[Frank Homann]

hi!

i'm running a zabbix server on tokudb and after an unclean server reboot mysqld is crashing with the following log:

http://pastebin.com/NdPCcdcY

how to recover?

thank you.

[Frank Homann]

i finally solved by parsing each tokudb table with tokuftdump and finding out two tabes had corrupted indexes.

removing any file referencing the tables permitted me to start mysqld and dump what was left, however i've lost the affected tables data...

i still wonder if there is a better (lossless) recovery path and why the hell tokudb is crashing because an index corruption on an idle mysql server...

anyway, back to innodb.

[George Lorch]

Hello Frank,

As a new custodian of TokuDB I apologize profusely for the recovery issue you have. I have started a bit of an internal 'witch hunt' for any lingering data consistency issues that have not yet been identified and fixed. I wonder if you would be so kind to let me know which specific MySQL variant you encountered this on, TokuDB versions and any server options you might be using that stray from the defaults. If you are not comfortable revealing any of this on this public forum, please email me privately at firstname dot lastname at percona

[Frank Homann]

thank you george,

i'm using mariadb-10.0.21 and tokudb that comes with it (build and deployed on gentoo with jemalloc support)

huge tables was properly disabled at boot time as suggested by tokudb folks.

no special configuration was used apart from usual buffer and caching settings and the only custom tokudb settings was "tokudb_cache_size = 5G" (the server has 48 gb ram).

i think it's quite easy to reproduce by forcing a server reset while under heavy write load...

if you need further information, just ask... ;)

[Kshitij Banerjee]

I had a similar corruption issue, wherein I asked about it here https://groups.google.com/forum/#!topic/tokudb-user/RrE5MNJFCxw. 

I patched tokuftdump to dump hex and did some reverse engineering to recover my data from the main files. This is tedious and time consuming. 

Such unrecoverable failures prevent me from using it in production... Please fix these. In my case, some status files have gone missing suddenly.