Server crash and MariaDb with TokuDb does not start

[Elena Cuevas]

Hi, 

I have a server with mariadb 5.5.30 and tokudb-7.0.4. The server has crashed (storage problems) and now, once I have restored from backup, I cannot restart Toku. 

This is what I get on my error log (I have no idea of what the selected line is, I have never had this path):

mysqld_safe: Starting mysqld daemon with databases from /home/mysql/toku/data

mysqld: 131020 11:45:32 InnoDB: The InnoDB memory heap is disabled

mysqld: 131020 11:45:32 InnoDB: Mutexes and rw_locks use GCC atomic builtins

mysqld: 131020 11:45:32 InnoDB: Compressed tables use zlib 1.2.3

mysqld: 131020 11:45:32 InnoDB: Using Linux native AIO

mysqld: 131020 11:45:32 InnoDB: Initializing buffer pool, size = 128.0M

mysqld: 131020 11:45:32 InnoDB: Completed initialization of buffer pool

mysqld: InnoDB: Error: log file ./ib_logfile0 is of different size 0 52428800 bytes

mysqld: InnoDB: than specified in the .cnf file 0 5242880 bytes!

mysqld: 131020 11:45:32 [ERROR] Plugin 'InnoDB' init function returned error.

mysqld: 131020 11:45:32 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

mysqld: 131020 11:45:32 [Note] Plugin 'FEEDBACK' is disabled.

mysqld: Sun Oct 20 11:45:32 2013 Tokudb recovery starting in env /home/mysql/toku/data/

mysqld: Sun Oct 20 11:45:32 2013 Tokudb recovery scanning backward from 181431135

mysqld: Sun Oct 20 11:45:32 2013 Tokudb recovery bw_end_checkpoint at 181431135 timestamp 1379942415567966 xid 181430903 (bw_newer)

mysqld: Sun Oct 20 11:45:32 2013 Tokudb recovery bw_begin_checkpoint at 181430903 timestamp 1379942415203755 (bw_between)

mysqld: Sun Oct 20 11:45:32 2013 Tokudb recovery turning around at begin checkpoint 181430903 time 364211

mysqld: Sun Oct 20 11:45:32 2013 Tokudb recovery starts scanning forward to 181431135 from 181430903 left 232 (fw_between)

mysqld: /home/tokubuild/build-tokudb-7.0.4/mariadb/storage/tokudb/ft-index/ft/ft-serialize.cc:711 toku_deserialize_ft_from: Assertion `!((r0==0 && checkpoint_lsn_0.lsn > max_acceptable_lsn.lsn) && (r1==0 && checkpoint_lsn_1.lsn > max_acceptable_lsn.lsn))' failed (errno=2)

mysqld: : No such file or directory

mysqld: Backtrace: (Note: toku_do_assert=0x0x7f34495585d0)

mysqld: /home/mysql/toku/7.0.4/lib/plugin/ha_tokudb.so(+0x129428)[0x7f3449558428]

mysqld: /home/mysql/toku/7.0.4/lib/plugin/ha_tokudb.so(+0x1295cd)[0x7f34495585cd]

mysqld: /home/mysql/toku/7.0.4/lib/plugin/ha_tokudb.so(_Z24toku_deserialize_ft_fromi10__toku_lsnPP2ft+0x35d)[0x7f344950b53d]

mysqld: /home/mysql/toku/7.0.4/lib/plugin/ha_tokudb.so(_Z35toku_read_ft_and_store_in_cachefileP9ft_handleP9cachefile10__toku_lsnPP2ftPb+0x9a)[0x7f34494d4bba]

mysqld: /home/mysql/toku/7.0.4/lib/plugin/ha_tokudb.so(+0xcdbcd)[0x7f34494fcbcd]

mysqld: /home/mysql/toku/7.0.4/lib/plugin/ha_tokudb.so(_Z28toku_ft_handle_open_recoveryP9ft_handlePKciiP10cachetableP7tokutxn7FILENUM10__toku_lsn+0x2a)[0x7f34494fd2fa]

mysqld: /home/mysql/toku/7.0.4/lib/plugin/ha_tokudb.so(+0xebf74)[0x7f344951af74]

mysqld: /home/mysql/toku/7.0.4/lib/plugin/ha_tokudb.so(_Z14tokudb_recoverP13__toku_db_envPFvS0_P7tokutxnEPFvS0_P10cachetableEP10tokuloggerPKcSC_PFiP9__toku_dbPK10__toku_dbtSH_EPFiSE_SH_SH_SH_PFvSH_PvESK_EPFiSE_SE_PSF_SP_SH_SH_EPFiSE_SE_SP_SH_SH_Em+0x1c9)[0x7f344951bd69]

mysqld: /home/mysql/toku/7.0.4/lib/plugin/ha_tokudb.so(+0x6f4dc)[0x7f344949e4dc]

mysqld: /home/mysql/toku/7.0.4/lib/plugin/ha_tokudb.so(+0x4a055)[0x7f3449479055]

mysqld: /home/mysql/toku/7.0.4/bin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x7005a8]

mysqld: /home/mysql/toku/7.0.4/bin/mysqld[0x5d99d9]

mysqld: /home/mysql/toku/7.0.4/bin/mysqld(_Z11plugin_initPiPPci+0x923)[0x5dcca3]

mysqld: /home/mysql/toku/7.0.4/bin/mysqld[0x55785e]

mysqld: /home/mysql/toku/7.0.4/bin/mysqld(_Z11mysqld_mainiPPc+0x35b)[0x55835b]

mysqld: /lib/libc.so.6(__libc_start_main+0xfd)[0x7f346bd5cc8d]

mysqld: /home/mysql/toku/7.0.4/bin/mysqld[0x51bbed]

mysqld: Engine status function not available

mysqld: Arena 0:

mysqld: system bytes     =          0

mysqld: in use bytes     =          0

mysqld: Total (incl. mmap):

mysqld: system bytes     =          0

mysqld: in use bytes     =          0

mysqld: max mmap regions =          0

mysqld: max mmap bytes   =          0

mysqld: 131020 11:45:32 [ERROR] mysqld got signal 6 ;

mysqld: This could be because you hit a bug. It is also possible that this binary

mysqld: or one of the libraries it was linked against is corrupt, improperly built,

mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.

mysqld:

mysqld: To report this bug, see http://kb.askmonty.org/en/reporting-bugs

mysqld:

mysqld: We will try our best to scrape up some info that will hopefully help

mysqld: diagnose the problem, but since we have already crashed,

mysqld: something is definitely wrong and this may fail.

mysqld:

mysqld: Server version: 5.5.30-tokudb-7.0.4-MariaDB-log

mysqld: key_buffer_size=134217728

mysqld: read_buffer_size=2097152

mysqld: max_used_connections=0

mysqld: max_threads=132

mysqld: thread_count=0

mysqld: It is possible that mysqld could use up to

mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2565947 K  bytes of memory

mysqld: Hope that's ok; if not, decrease some variables in the equation.

mysqld:

mysqld: Thread pointer: 0x0x0

mysqld: Attempting backtrace. You can use the following information to find out

mysqld: where mysqld died. If you see no messages after this, something went

mysqld: terribly wrong...

mysqld: stack_bottom = 0x0 thread_stack 0x48000

mysqld: mysys/stacktrace.c:247(my_print_stacktrace)[0xada50b]

mysqld: sql/signal_handler.cc:153(handle_fatal_signal)[0x6fdf4e]

mysqld: ??:0(??)[0x7f346ca7fff0]

mysqld: ??:0(??)[0x7f346bd701b5]

mysqld: ??:0(??)[0x7f346bd72fc0]

mysqld: ??:0(??)[0x7f3449558527]

mysqld: ??:0(??)[0x7f34495585cd]

mysqld: ??:0(??)[0x7f344950b53d]

mysqld: ??:0(??)[0x7f34494d4bba]

mysqld: ??:0(??)[0x7f34494fcbcd]

mysqld: ??:0(??)[0x7f34494fd2fa]

mysqld: ??:0(??)[0x7f344951af74]

mysqld: ??:0(??)[0x7f344951bd69]

mysqld: ??:0(??)[0x7f344949e4dc]

mysqld: ??:0(??)[0x7f3449479055]

mysqld: sql/handler.cc:476(ha_initialize_handlerton(st_plugin_int*))[0x7005a8]

mysqld: sql/sql_plugin.cc:1354(plugin_initialize)[0x5d99d9]

mysqld: sql/sql_plugin.cc:1648(plugin_init(int*, char**, int))[0x5dcca3]

mysqld: sql/mysqld.cc:4289(init_server_components)[0x55785e]

mysqld: sql/mysqld.cc:4847(mysqld_main(int, char**))[0x55835b]

mysqld: ??:0(??)[0x7f346bd5cc8d]

mysqld: ??:0(_start)[0x51bbed]

mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

mysqld: information that should help you find out what is causing the crash.

mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended

I have tried to do several things to restart, but any has worked: 

• start with 

  innodb_force_recovery = 6 and

  innodb_purge_thread = 0 in order to restart the DB and launch mysqldump to recover it later.
  
• delete both ib_logfile0, ib_logfile1 and restart (this part has worked, but still continues whit the rest of the log , is the same).
  
• I have re-installed Toku, but still the same error.

I guess the problem is in the data, but I don't know how could I recover it. 

The backup failed, so I have only the information in this machine. 

Can you please help me?

Thanks a lot, 

[Elena Cuevas]

I have resovle the problem, maybe it's a bad practise, but .. has resolve my problem . 

1- move my data directory

2- re-create the database, this generates a new data directory

3- move new data directory and replace the old one

4- copy new data directory/* to the old one and replace everything but tokudb.directory, tokudb.environment and tokudb.meta_3_0_18.tokudb

5- restart mysql and.. voila , everything is working fine. I have checked one by one all the tables and select all the rows to check everything is all right.

6- backup

Thanks

Elena

[Rich Prohaska]

Hello Elena,

I am glad that you were able to get back running.  However, the crash during TokuDB recovery from a backup indicates that the recovery log was old WRT one of the TokuDB tables.  This tells me that either (1) your backup is bad, or (2) the restoration of the backup is bad.  How do you take a backup?

[Rich Prohaska]

Just copying the data directory while mysql is running would explain the mismatch between the TokuDB fractal tree file and the TokuDB recovery log.  This will not work with the community version of TokuDB.  One can use file system snapshots to get a consistent backup of the TokuDB files, or one get use mysqldump to get a logical copy of the data, or one can use the hot backup in the TokuDB enterprise version.