Date: Sat, 10 Nov 2012 09:01:17 -0800 (PST) From: amol <ajke...@gmail.com> To: percona-discussion@googlegroups.com Message-Id: <525dadd8-9435-4f86-ac56-0038320f916d@googlegroups.com> In-Reply-To: <CAFHKPu_VsyJSK5WHmdYAsAD1syoUuy4ZhYPx2-X0BOLGq+RuKQ@mail.gmail.com> References: <2140214f-8437-4bfb-883b-9ae484e62c0e@googlegroups.com> <CAFHKPu_VsyJSK5WHmdYAsAD1syoUuy4ZhYPx2-X0BOLGq+RuKQ@mail.gmail.com> Subject: Re: [percona-group] Percona server 5.5.27-29.0 crashed during a bulk delete process MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_598_1390945.1352566878022" ------=_Part_598_1390945.1352566878022 Content-Type: multipart/alternative; boundary="----=_Part_599_23740821.1352566878022" ------=_Part_599_23740821.1352566878022 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Hi Laurynas, thanks for much for taking interest in this and replying, yes here is the log from the night the db crashed, you will the timings according to the activity during crash please let me know if you need anything else 121108 0:15:31 InnoDB: Error: space id and page n:o stored in the page InnoDB: read in are 808728115:3377681476, should be 239:31651! InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 31651. InnoDB: You may have to recover from a backup. 121108 0:15:31 InnoDB: Page dump in ascii and hex (16384 bytes): InnoDB: End of page dump 121108 0:15:31 InnoDB: Page checksum 2335292886 (32bit_calc: 2805692839), prior-to-4.0.14-form checksum 3860048700 InnoDB: stored checksum 2071986176, prior-to-4.0.14-form stored checksum 4101310499 InnoDB: Page lsn 943336760 875443507, low 4 bytes of lsn at page end 1798844638 InnoDB: Page number (if stored to page already) 3377681476, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 808728115 InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 31651. InnoDB: You may have to recover from a backup. InnoDB: It is also possible that your operating InnoDB: system has corrupted its own file cache InnoDB: and rebooting your computer removes the InnoDB: error. InnoDB: If the corrupt page is an index page InnoDB: you can also try to fix the corruption InnoDB: by dumping, dropping, and reimporting InnoDB: the corrupt table. You can use CHECK InnoDB: TABLE to scan your table for corruption. InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 121108 0:15:31 InnoDB: Error: space id and page n:o stored in the page InnoDB: read in are 808728115:3377681476, should be 239:31651! InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 31651. InnoDB: You may have to recover from a backup. 121108 0:15:31 InnoDB: Page dump in ascii and hex (16384 bytes): nnoDB: End of page dump 121108 0:15:32 InnoDB: Page checksum 2335292886 (32bit_calc: 2805692839), prior-to-4.0.14-form checksum 3860048700 InnoDB: stored checksum 2071986176, prior-to-4.0.14-form stored checksum 4101310499 InnoDB: Page lsn 943336760 875443507, low 4 bytes of lsn at page end 1798844638 InnoDB: Page number (if stored to page already) 3377681476, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 808728115 InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 31651. InnoDB: You may have to recover from a backup. InnoDB: It is also possible that your operating InnoDB: system has corrupted its own file cache InnoDB: and rebooting your computer removes the InnoDB: error. InnoDB: If the corrupt page is an index page InnoDB: you can also try to fix the corruption InnoDB: by dumping, dropping, and reimporting InnoDB: the corrupt table. You can use CHECK InnoDB: TABLE to scan your table for corruption. InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 121108 0:15:32 InnoDB: Error: space id and page n:o stored in the page InnoDB: read in are 808728115:3377681476, should be 239:31651! InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 31651. InnoDB: You may have to recover from a backup. 121108 0:15:32 InnoDB: Page dump in ascii and hex (16384 bytes): 21108 0:15:51 InnoDB: Assertion failure in thread 140346818901760 in file buf0buf.c line 2614 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 05:15:51 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Please help us make Percona Server better by reporting any bugs at http://bugs.percona.com/ key_buffer_size=33554432 read_buffer_size=131072 max_used_connections=158 max_threads=500 thread_count=1 connection_count=1 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1126896 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x5b5fbe0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7fa50a492e88 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7d3545] /usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x69f394] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7fa5aa135cb0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7fa5a925e445] /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7fa5a9261bab] /usr/sbin/mysqld[0x8b968f] /usr/sbin/mysqld[0x8a87cb] /usr/sbin/mysqld[0x8576b7] /usr/sbin/mysqld[0x85819a] /usr/sbin/mysqld[0x82d91d] /usr/sbin/mysqld(_Z13rr_sequentialP11READ_RECORD+0x1f)[0x77362f] /usr/sbin/mysqld(_Z12mysql_deleteP3THDP10TABLE_LISTP4ItemP10SQL_I_ListI8st_orderEyy+0x9c5)[0x78fd05] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x9ed)[0x5992dd] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x333)[0x59d643] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x15df)[0x59eccf] /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0xdf)[0x63e2df] /usr/sbin/mysqld(handle_one_connection+0x51)[0x63e411] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fa5aa12de9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fa5a931a4bd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (7fa53457e350): is an invalid pointer Connection ID (thread ID): 20225462 Status: NOT_KILLED You may download the Percona Server operations manual by visiting http://www.percona.com/software/percona-server/. You may find information in the manual which will help you identify the cause of the crash. 121108 00:15:52 mysqld_safe mysqld from pid file /var/lib/mysql/iform-prod5.pid ended InnoDB: Error: Unable to read tablespace 239 page no 31651 into the buffer pool after 100 attempts InnoDB: The most probable cause of this error may be that the table has been corrupted. InnoDB: You can try to fix this problem by using innodb_force_recovery. InnoDB: Please see reference manual for more details. InnoDB: Aborting... 121108 0:15:51 InnoDB: Assertion failure in thread 140346818901760 in file buf0buf.c line 2614 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 05:15:51 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Please help us make Percona Server better by reporting any bugs at http://bugs.percona.com/ key_buffer_size=33554432 read_buffer_size=131072 max_used_connections=158 max_threads=500 thread_count=1 connection_count=1 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1126896 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. On Saturday, November 10, 2012 4:56:29 AM UTC-5, Laurynas Biveinis wrote: > > Hi - > > Do you have the 1st crash message in the log? > > This could be caused by > https://bugs.launchpad.net/percona-server/+bug/1042640, which was fixed > in 5.5.28-29.1 release. > > > On Thu, Nov 8, 2012 at 5:53 PM, amol <ajk...@gmail.com <javascript:>>wrote: > >> Hi >> i am running 5.5.27-29.0 version of Percona db for my production database >> and last night during a bulk delete process the mysql instance crashed and >> would not start even when i added >> >> innodb_force_recovery = 4 >> >> the errors in the mysql error log were >> >> >> InnoDB: Reading tablespace information from the .ibd files... >> InnoDB: Restoring possible half-written data pages from the doublewrite >> InnoDB: buffer... >> InnoDB: 1 transaction(s) which must be rolled back or cleaned up >> InnoDB: in total 298015 row operations to undo >> InnoDB: Trx id counter is 131DE00 >> 121108 1:16:18 InnoDB: Waiting for the background threads to start >> 121108 1:16:19 InnoDB: Waiting for the background threads to start >> >> >> >> so then i killed the pid again and started with >> >> innodb_force_recovery = 1 >> >> >> it started with these messages in the log >> >> InnoDB: End of page dump >> 121108 1:55:48 InnoDB: Page checksum 2477130822 (32bit_calc: 2803992502), prior-to-4.0.14-form checksum 2861862647 >> InnoDB: stored checksum 3571226133, prior-to-4.0.14-form stored checksum 2861862647 >> InnoDB: Page lsn 1 2921818088, low 4 bytes of lsn at page end 2921818088 >> InnoDB: Page number (if stored to page already) 12280, >> InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 239 >> InnoDB: Page may be an index page where index id is 441 >> InnoDB: (index "PRIMARY" of table "prod"."Location") >> InnoDB: Database page corruption on disk or a failed >> InnoDB: file read of page 12280. >> InnoDB: You may have to recover from a backup. >> InnoDB: It is also possible that your operating >> InnoDB: system has corrupted its own file cache >> InnoDB: and rebooting your computer removes the >> InnoDB: error. >> InnoDB: If the corrupt page is an index page >> InnoDB: you can also try to fix the corruption >> InnoDB: by dumping, dropping, and reimporting >> InnoDB: the corrupt table. You can use CHECK >> InnoDB: TABLE to scan your table for corruption. >> InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html >> InnoDB: about forcing recovery. >> 121108 1:55:48 Percona XtraDB (http://www.percona.com) 1.1.8-rel29.0 started; log sequence number 7298862733 >> 121108 1:55:48 InnoDB: !!! innodb_force_recovery is set to 1 !!! >> 121108 1:55:48 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306 >> 121108 1:55:48 [Note] - '0.0.0.0' resolves to '0.0.0.0'; >> 121108 1:55:48 [Note] Server socket created on IP: '0.0.0.0'. >> 121108 1:55:49 [Warning] 'user' entry 'root@iform-prod5' ignored in --skip-name-resolve mode. >> 121108 1:55:49 [Warning] 'proxies_priv' entry '@ root@iform-prod5' ignored in --skip-name-resolve mode. >> 121108 1:55:49 [Note] Event Scheduler: Loaded 0 events >> 121108 1:55:49 [Note] /usr/sbin/mysqld: ready for connections. >> Version: '5.5.27-29.0' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona Server (GPL), Release 29.0 >> InnoDB: A new raw disk partition was initialized or >> InnoDB: innodb_force_recovery is on: we do not allow >> InnoDB: database modifications by the user. Shut down >> InnoDB: mysqld and edit my.cnf so that newraw is replaced >> InnoDB: with raw, and innodb_force_... is removed. >> InnoDB: A new raw disk partition was initialized or >> InnoDB: innodb_force_recovery is on: we do not allow >> InnoDB: database modifications by the user. Shut down >> InnoDB: mysqld and edit my.cnf so that newraw is replaced >> InnoDB: with raw, and innodb_force_... is removed. >> 2 3InnoDB: A new raw disk partition was initialized or >> InnoDB: innodb_force_recovery is on: we do not allow >> InnoDB: database modifications by the user. Shut down >> InnoDB: mysqld and edit my.cnf so that newraw is replaced >> InnoDB: with raw, and innodb_force_... is removed. >> 4InnoDB: A new raw disk partition was initialized or >> InnoDB: innodb_force_recovery is on: we do not allow >> >> >> >> >> and then i had to perform a restore from backup of that table >> >> any idea why the db would have crashed, it was doing a bulk delete on a >> table that has high amount of inserts >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Percona Discussion" group. >> To post to this group, send email to percona-d...@googlegroups.com<javascript:> >> . >> >> >> > > > > -- > Laurynas > www.percona.com > > ------=_Part_599_23740821.1352566878022 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <div>Hi Laurynas,</div><div>thanks for much for taking interest in this and= replying, </div><div><br></div><div>yes here is the log from the nigh= t the db crashed, you will the timings according to the activity during cra= sh</div><div><br></div><div>please let me know if you need anything else</d= iv><div><br></div><div><br></div><div>121108 0:15:31 InnoDB: Er= ror: space id and page n:o stored in the page</div><div>InnoDB: read in are= 808728115:3377681476, should be 239:31651!</div><div>InnoDB: Database page= corruption on disk or a failed</div><div>InnoDB: file read of page 31651.<= /div><div>InnoDB: You may have to recover from a backup.</div><div>121108 &= nbsp;0:15:31 InnoDB: Page dump in ascii and hex (16384 bytes):</div><= div><br></div><div>InnoDB: End of page dump</div><div>121108 0:15:31 = InnoDB: Page checksum 2335292886 (32bit_calc: 2805692839), prior-to-4= .0.14-form checksum 3860048700</div><div>InnoDB: stored checksum 2071986176= , prior-to-4.0.14-form stored checksum 4101310499</div><div>InnoDB: Page ls= n 943336760 875443507, low 4 bytes of lsn at page end 1798844638</div><div>= InnoDB: Page number (if stored to page already) 3377681476,</div><div>InnoD= B: space id (if created with >=3D MySQL-4.1.1 and stored already) 808728= 115</div><div>InnoDB: Database page corruption on disk or a failed</div><di= v>InnoDB: file read of page 31651.</div><div>InnoDB: You may have to recove= r from a backup.</div><div>InnoDB: It is also possible that your operating<= /div><div>InnoDB: system has corrupted its own file cache</div><div>InnoDB:= and rebooting your computer removes the</div><div>InnoDB: error.</div><div= >InnoDB: If the corrupt page is an index page</div><div>InnoDB: you can als= o try to fix the corruption</div><div>InnoDB: by dumping, dropping, and rei= mporting</div><div>InnoDB: the corrupt table. You can use CHECK</div><div>I= nnoDB: TABLE to scan your table for corruption.</div><div>InnoDB: See also = http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html</div><d= iv>InnoDB: about forcing recovery.</div><div>121108 0:15:31 Inn= oDB: Error: space id and page n:o stored in the page</div><div>InnoDB: read= in are 808728115:3377681476, should be 239:31651!</div><div>InnoDB: Databa= se page corruption on disk or a failed</div><div>InnoDB: file read of page = 31651.</div><div>InnoDB: You may have to recover from a backup.</div><div>1= 21108 0:15:31 InnoDB: Page dump in ascii and hex (16384 bytes):= </div><div><br></div><div>nnoDB: End of page dump</div><div>121108 0:= 15:32 InnoDB: Page checksum 2335292886 (32bit_calc: 2805692839), prio= r-to-4.0.14-form checksum 3860048700</div><div>InnoDB: stored checksum 2071= 986176, prior-to-4.0.14-form stored checksum 4101310499</div><div>InnoDB: P= age lsn 943336760 875443507, low 4 bytes of lsn at page end 1798844638</div= ><div>InnoDB: Page number (if stored to page already) 3377681476,</div><div= >InnoDB: space id (if created with >=3D MySQL-4.1.1 and stored already) = 808728115</div><div>InnoDB: Database page corruption on disk or a failed</d= iv><div>InnoDB: file read of page 31651.</div><div>InnoDB: You may have to = recover from a backup.</div><div>InnoDB: It is also possible that your oper= ating</div><div>InnoDB: system has corrupted its own file cache</div><div>I= nnoDB: and rebooting your computer removes the</div><div>InnoDB: error.</di= v><div>InnoDB: If the corrupt page is an index page</div><div>InnoDB: you c= an also try to fix the corruption</div><div>InnoDB: by dumping, dropping, a= nd reimporting</div><div>InnoDB: the corrupt table. You can use CHECK</div>= <div>InnoDB: TABLE to scan your table for corruption.</div><div>InnoDB: See= also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html</= div><div>InnoDB: about forcing recovery.</div><div>121108 0:15:32 &nb= sp;InnoDB: Error: space id and page n:o stored in the page</div><div>InnoDB= : read in are 808728115:3377681476, should be 239:31651!</div><div>InnoDB: = Database page corruption on disk or a failed</div><div>InnoDB: file read of= page 31651.</div><div>InnoDB: You may have to recover from a backup.</div>= <div>121108 0:15:32 InnoDB: Page dump in ascii and hex (16384 b= ytes):</div><div><br></div><div>21108 0:15:51 InnoDB: Assertion= failure in thread 140346818901760 in file buf0buf.c line 2614</div><div>In= noDB: We intentionally generate a memory trap.</div><div>InnoDB: Submit a d= etailed bug report to http://bugs.mysql.com.</div><div>InnoDB: If you get r= epeated assertion failures or crashes, even</div><div>InnoDB: immediately a= fter the mysqld startup, there may be</div><div>InnoDB: corruption in the I= nnoDB tablespace. Please refer to</div><div>InnoDB: http://dev.mysql.com/do= c/refman/5.5/en/forcing-innodb-recovery.html</div><div>InnoDB: about forcin= g recovery.</div><div>05:15:51 UTC - mysqld got signal 6 ;</div><div>This c= ould be because you hit a bug. It is also possible that this binary</div><d= iv>or one of the libraries it was linked against is corrupt, improperly bui= lt,</div><div>or misconfigured. This error can also be caused by malfunctio= ning hardware.</div><div>We will try our best to scrape up some info that w= ill hopefully help</div><div>diagnose the problem, but since we have alread= y crashed,</div><div>something is definitely wrong and this may fail.</div>= <div>Please help us make Percona Server better by reporting any</div><div>b= ugs at http://bugs.percona.com/</div><div><br></div><div>key_buffer_size=3D= 33554432</div><div>read_buffer_size=3D131072</div><div>max_used_connections= =3D158</div><div>max_threads=3D500</div><div>thread_count=3D1</div><div>con= nection_count=3D1</div><div>It is possible that mysqld could use up to</div= ><div>key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = =3D 1126896 K bytes of memory</div><div>Hope that's ok; if not, decre= ase some variables in the equation.</div><div><br></div><div>Thread pointer= : 0x5b5fbe0</div><div>Attempting backtrace. You can use the following infor= mation to find out</div><div>where mysqld died. If you see no messages afte= r this, something went</div><div>terribly wrong...</div><div>stack_bottom = =3D 7fa50a492e88 thread_stack 0x40000</div><div>/usr/sbin/mysqld(my_print_s= tacktrace+0x35)[0x7d3545]</div><div>/usr/sbin/mysqld(handle_fatal_signal+0x= 4a4)[0x69f394]</div><div>/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7= fa5aa135cb0]</div><div>/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7fa5= a925e445]</div><div>/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7fa5a926= 1bab]</div><div>/usr/sbin/mysqld[0x8b968f]</div><div>/usr/sbin/mysqld[0x8a8= 7cb]</div><div>/usr/sbin/mysqld[0x8576b7]</div><div>/usr/sbin/mysqld[0x8581= 9a]</div><div>/usr/sbin/mysqld[0x82d91d]</div><div>/usr/sbin/mysqld(_Z13rr_= sequentialP11READ_RECORD+0x1f)[0x77362f]</div><div>/usr/sbin/mysqld(_Z12mys= ql_deleteP3THDP10TABLE_LISTP4ItemP10SQL_I_ListI8st_orderEyy+0x9c5)[0x78fd05= ]</div><div>/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x9ed)[0x5992dd= ]</div><div>/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x333)[= 0x59d643]</div><div>/usr/sbin/mysqld(_Z16dispatch_command19enum_server_comm= andP3THDPcj+0x15df)[0x59eccf]</div><div>/usr/sbin/mysqld(_Z24do_handle_one_= connectionP3THD+0xdf)[0x63e2df]</div><div>/usr/sbin/mysqld(handle_one_conne= ction+0x51)[0x63e411]</div><div>/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e= 9a)[0x7fa5aa12de9a]</div><div>/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0= x7fa5a931a4bd]</div><div><br></div><div>Trying to get some variables.</div>= <div>Some pointers may be invalid and cause the dump to abort.</div><div>Qu= ery (7fa53457e350): is an invalid pointer</div><div>Connection ID (thread I= D): 20225462</div><div>Status: NOT_KILLED</div><div><br></div><div>You may = download the Percona Server operations manual by visiting</div><div>http://= www.percona.com/software/percona-server/. You may find information</div><di= v>in the manual which will help you identify the cause of the crash.</div><= div>121108 00:15:52 mysqld_safe mysqld from pid file /var/lib/mysql/iform-p= rod5.pid ended</div><div><br></div><div><br></div><div>InnoDB: Error: Unabl= e to read tablespace 239 page no 31651 into the buffer pool after 100 attem= pts</div><div>InnoDB: The most probable cause of this error may be that the= table has been corrupted.</div><div>InnoDB: You can try to fix this proble= m by using innodb_force_recovery.</div><div>InnoDB: Please see reference ma= nual for more details.</div><div>InnoDB: Aborting...</div><div>121108  = ;0:15:51 InnoDB: Assertion failure in thread 140346818901760 in file = buf0buf.c line 2614</div><div>InnoDB: We intentionally generate a memory tr= ap.</div><div>InnoDB: Submit a detailed bug report to http://bugs.mysql.com= .</div><div>InnoDB: If you get repeated assertion failures or crashes, even= </div><div>InnoDB: immediately after the mysqld startup, there may be</div>= <div>InnoDB: corruption in the InnoDB tablespace. Please refer to</div><div= >InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.htm= l</div><div>InnoDB: about forcing recovery.</div><div>05:15:51 UTC - mysqld= got signal 6 ;</div><div>This could be because you hit a bug. It is also p= ossible that this binary</div><div>or one of the libraries it was linked ag= ainst is corrupt, improperly built,</div><div>or misconfigured. This error = can also be caused by malfunctioning hardware.</div><div>We will try our be= st to scrape up some info that will hopefully help</div><div>diagnose the p= roblem, but since we have already crashed,</div><div>something is definitel= y wrong and this may fail.</div><div>Please help us make Percona Server bet= ter by reporting any</div><div>bugs at http://bugs.percona.com/</div><div><= br></div><div>key_buffer_size=3D33554432</div><div>read_buffer_size=3D13107= 2</div><div>max_used_connections=3D158</div><div>max_threads=3D500</div><di= v>thread_count=3D1</div><div>connection_count=3D1</div><div>It is possible = that mysqld could use up to</div><div>key_buffer_size + (read_buffer_size += sort_buffer_size)*max_threads =3D 1126896 K bytes of memory</div><di= v>Hope that's ok; if not, decrease some variables in the equation.</div><di= v><br></div><br>On Saturday, November 10, 2012 4:56:29 AM UTC-5, Laurynas B= iveinis wrote:<blockquote class=3D"gmail_quote" style=3D"margin: 0;margin-l= eft: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;">Hi - <br><br>Do = you have the 1st crash message in the log?<br><br>This could be caused by <= a href=3D"https://bugs.launchpad.net/percona-server/+bug/1042640" target=3D= "_blank">https://bugs.launchpad.net/<wbr>percona-server/+bug/1042640</a>, w= hich was fixed in 5.5.28-29.1 release.<br> <div><br><br><div class=3D"gmail_quote">On Thu, Nov 8, 2012 at 5:53 PM, amo= l <span dir=3D"ltr"><<a href=3D"javascript:" target=3D"_blank" gdf-obfus= cated-mailto=3D"8if722MCshYJ">ajk...@gmail.com</a>></span> wrote:<br><bl= ockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #= ccc solid;padding-left:1ex"> Hi <div>i am running 5.5.27-29.0 version of Percona db for my pro= duction database and last night during a bulk delete process the mysql inst= ance crashed and would not start even when i added </div><div><pre sty= le=3D"line-height:20px;max-width:720px;background-color:rgb(238,238,238);fo= nt-family:'Courier New',Courier,fixed,monospace;outline:0px;padding:2px">in= nodb_force_recovery =3D 4</pre></div><div>the errors in the mysql error log= were</div><div><br></div><div><br></div><div><pre>InnoDB: Reading tablespa= ce information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 298015 row operations to undo InnoDB: Trx id counter is 131DE00 121108 1:16:18 InnoDB: Waiting for the background threads to start 121108 1:16:19 InnoDB: Waiting for the background threads to start</pre><= pre><br></pre><pre><br></pre><pre>so then i killed the pid again and starte= d with </pre><pre><pre style=3D"line-height:20px;max-width:720px;background= -color:rgb(238,238,238);font-family:'Courier New',Courier,fixed,monospace;o= utline:0px;padding:2px">innodb_force_recovery =3D 1</pre></pre></div><div><= br></div><div>it started with these messages in the log</div><div><br></div= ><div><pre>InnoDB: End of page dump 121108 1:55:48 InnoDB: Page checksum 2477130822 (32bit_calc: 2803992502),= prior-to-4.0.14-form checksum 2861862647 InnoDB: stored checksum 3571226133, prior-to-4.0.14-form stored checksum 28= 61862647 InnoDB: Page lsn 1 2921818088, low 4 bytes of lsn at page end 2921818088 InnoDB: Page number (if stored to page already) 12280, InnoDB: space id (if created with >=3D MySQL-4.1.1 and stored already) 2= 39 InnoDB: Page may be an index page where index id is 441 InnoDB: (index "PRIMARY" of table "prod"."Location") InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 12280. InnoDB: You may have to recover from a backup. InnoDB: It is also possible that your operating InnoDB: system has corrupted its own file cache InnoDB: and rebooting your computer removes the InnoDB: error. InnoDB: If the corrupt page is an index page InnoDB: you can also try to fix the corruption InnoDB: by dumping, dropping, and reimporting InnoDB: the corrupt table. You can use CHECK InnoDB: TABLE to scan your table for corruption. InnoDB: See also <a href=3D"http://dev.mysql.com/doc/refman/5.5/en/forcing-= innodb-recovery.html" target=3D"_blank">http://dev.mysql.com/doc/<wbr>refma= n/5.5/en/forcing-innodb-<wbr>recovery.html</a> InnoDB: about forcing recovery. 121108 1:55:48 Percona XtraDB (<a href=3D"http://www.percona.com" target= =3D"_blank">http://www.percona.com</a>) 1.1.8-rel29.0 started; log sequence= number 7298862733 121108 1:55:48 InnoDB: !!! innodb_force_recovery is set to 1 !!! 121108 1:55:48 [Note] Server hostname (bind-address): '0.0.0.0'; port: 330= 6 121108 1:55:48 [Note] - '0.0.0.0' resolves to '0.0.0.0'; 121108 1:55:48 [Note] Server socket created on IP: '0.0.0.0'. 121108 1:55:49 [Warning] 'user' entry 'root@iform-prod5' ignored in --skip= -name-resolve mode. 121108 1:55:49 [Warning] 'proxies_priv' entry '@ root@iform-prod5' ignored= in --skip-name-resolve mode. 121108 1:55:49 [Note] Event Scheduler: Loaded 0 events 121108 1:55:49 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.27-29.0' socket: '/var/run/mysqld/mysqld.sock' port: 3306 = Percona Server (GPL), Release 29.0 InnoDB: A new raw disk partition was initialized or InnoDB: innodb_force_recovery is on: we do not allow InnoDB: database modifications by the user. Shut down InnoDB: mysqld and edit my.cnf so that newraw is replaced InnoDB: with raw, and innodb_force_... is removed. InnoDB: A new raw disk partition was initialized or InnoDB: innodb_force_recovery is on: we do not allow InnoDB: database modifications by the user. Shut down InnoDB: mysqld and edit my.cnf so that newraw is replaced InnoDB: with raw, and innodb_force_... is removed. 2 3InnoDB: A new raw disk partition was initialized or InnoDB: innodb_force_recovery is on: we do not allow InnoDB: database modifications by the user. Shut down InnoDB: mysqld and edit my.cnf so that newraw is replaced InnoDB: with raw, and innodb_force_... is removed. 4InnoDB: A new raw disk partition was initialized or InnoDB: innodb_force_recovery is on: we do not allow</pre></div><div><br></= div><div><br></div><div><br></div><div>and then i had to perform a restore = from backup of that table</div><div><br></div><div>any idea why the db woul= d have crashed, it was doing a bulk delete on a table that has high amount = of inserts </div> <span><font color=3D"#888888"><div><br></div><div><br></div><div><br></div> <p></p> -- <br> You received this message because you are subscribed to the Google Groups "= Percona Discussion" group.<br> To post to this group, send email to <a href=3D"javascript:" target=3D"_bla= nk" gdf-obfuscated-mailto=3D"8if722MCshYJ">percona-d...@<wbr>googlegroups.c= om</a>.<br> <br> <br> </font></span></blockquote></div><br><br clear=3D"all"><br>-- <br>Laurynas<= br><a href=3D"http://www.percona.com" target=3D"_blank">www.percona.com</a>= <br><br> </div> </blockquote> ------=_Part_599_23740821.1352566878022-- ------=_Part_598_1390945.1352566878022--
Message from discussion Percona server 5.5.27-29.0 crashed during a bulk delete process
| Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy |
| ©2013 Google |