Percona 5.5.28 crashes with table corruption during a delete process

[Amol]

Hi i have a single node instance of Percona mysql running on ubuntu 10.04 the following version

mysql> show global variables like '%version%';

+-------------------------+------------------------------------+

| Variable_name           | Value                              |

+-------------------------+------------------------------------+

| innodb_version          | 1.1.8-rel29.1                      |

| protocol_version        | 10                                 |

| slave_type_conversions  |                                    |

| version                 | 5.5.28-29.1                        |

| version_comment         | Percona Server (GPL), Release 29.1 |

| version_compile_machine | x86_64                             |

| version_compile_os      | Linux                              |

+-------------------------+------------------------------------+

during last nights daily deletion process on the master, the slave crashed with innodb corruption with the following errors in the log

key_buffer_size=16777216

read_buffer_size=131072

max_used_connections=81

max_threads=300

thread_count=0

connection_count=0

It is possible that mysqld could use up to 

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 672861 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f7a80000930

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 7f7aa4181868 thread_stack 0x40000

/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7d3755]

/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x69f314]

/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f7b08aabcb0]

/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f7b07bd4445]

/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f7b07bd7bab]

/usr/sbin/mysqld[0x8bb46f]

/usr/sbin/mysqld[0x8aa83b]

/usr/sbin/mysqld[0x859307]

/usr/sbin/mysqld[0x859dea]

/usr/sbin/mysqld[0x82c0d8]

/usr/sbin/mysqld[0x82c7cb]

/usr/sbin/mysqld(_ZN14Rows_log_event8find_rowEPK14Relay_log_info+0x2bd)[0x749d5d]

/usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEPK14Relay_log_info+0x11)[0x74a101]

/usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEPK14Relay_log_info+0x204)[0x74d654]

/usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14Relay_log_info+0x12d)[0x5350bd]

/usr/sbin/mysqld[0x539752]

/usr/sbin/mysqld(handle_slave_sql+0x9a5)[0x53ac05]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f7b08aa3e9a]

/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f7b07c904bd]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0): is an invalid pointer

Connection ID (thread ID): 423651

Status: NOT_KILLED

130509  0:15:07  InnoDB: Error: space id and page n:o stored in the page

InnoDB: read in are 445248321:226492568, should be 2444:48429!

InnoDB: Database page corruption on disk or a failed

InnoDB: file read of page 48429.

InnoDB: You may have to recover from a backup.

130509  0:15:07  InnoDB: Page dump in ascii and hex (16384 bytes):

http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html

when i later started the slave with innodb_force_recovery is set to 1, i see these messages in the error log

130509  0:47:44  InnoDB: Database was not shut down normally!

InnoDB: Starting crash recovery.

InnoDB: Reading tablespace information from the .ibd files...

InnoDB: Restoring possible half-written data pages from the doublewrite

InnoDB: buffer...

InnoDB: 1 transaction(s) which must be rolled back or cleaned up

InnoDB: in total 6345 row operations to undo

InnoDB: Trx id counter is 7574F00

InnoDB: Starting in background the rollback of uncommitted transactions

130509  0:47:51  InnoDB: Rolling back trx with id 7574DC3, 6345 rows to undo

InnoDB: Progress in percents: 1130509  0:47:51  InnoDB: Waiting for the background threads to start

 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100

InnoDB: Rolling back of trx id 7574DC3 completed

130509  0:47:51  InnoDB: Rollback of non-prepared transactions completed

130509  0:47:52 Percona XtraDB (http://www.percona.com) 1.1.8-rel29.1 started; log sequence number 82831398584

130509  0:47:52 InnoDB: !!! innodb_force_recovery is set to 1 !!!

130509  0:47:52 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306

130509  0:47:52 [Note]   - '0.0.0.0' resolves to '0.0.0.0';

130509  0:47:52 [Note] Server socket created on IP: '0.0.0.0'.

130509  0:47:52 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.000648' at position 7096207, relay log './iprod5-slave-relay-bin.000058' position: 7096353

130509  0:47:52 [Note] Slave I/O thread: connected to master 'slave...@xx.xx.xx.xx:3306',replication started in log 'mysql-bin.000648' at position 55172904

130509  0:47:52 [Note] Event Scheduler: Loaded 0 events

130509  0:47:52 [Note] /usr/sbin/mysqld: ready for connections.

Version: '5.5.28-29.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Percona Server (GPL), Release 29.1

InnoDB: A new raw disk partition was initialized or

InnoDB: innodb_force_recovery is on: we do not allow

InnoDB: database modifications by the user. Shut down

InnoDB: mysqld and edit my.cnf so that newraw is replaced

InnoDB: with raw, and innodb_force_... is removed.

130509  0:47:53 [ERROR] Slave SQL: Could not execute Delete_rows event on table setdb.Location; Got error -1 from storage engine, Error_code: 1030; handler error No Error!; the event's master log mysql-bin.000648, end_log_pos 7097314, Error_code: 1030

130509  0:47:53 [Warning] Slave: Got error -1 from storage engine Error_code: 1030

130509  0:47:53 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql-bin.000648' position 7096207

130509  0:47:55 [ERROR] /usr/sbin/mysqld: Table './mysql/proc' is marked as crashed and should be repaired

130509  0:47:55 [Warning] Checking table:   './mysql/proc'

130509  0:47:56 [ERROR] /usr/sbin/mysqld: Table './setdb/Location2' is marked as crashed and should be repaired

130509  0:47:56 [Warning] Checking table:   './setdb/Location2'

so i dropped the Location and Location2 tables and reimported the slave data from master and started the slave back(without the innodb recover parameter)

but what i am unable to understand is the main reason for this crash?

how do i know if it was running out of any parameter for the operation?