InnoDB: Assertion failure during running of xtrabackup

[Johann Vincent Paul Tagle]

We have Percona Server 5.1.57-rel12.8-log and we do full and incremental backups on it using xtrabackup.  The command we use for full backup is:

innobackupex-1.5.1 --slave-info                                                 \

  --defaults-file=/etc/my.cnf                                 \

  --user=root                                                             \

  --password=password                              \

  --socket=/var/run/mysqld/mysqld.sock                                       \

  --safe-slave-backup                                                     \

  --no-timestamp                                                          \

  2> /backup/xtrabackups/xtrabackup-timestamped-filename.log        \

  /backup/xtrabackups/timestamped-directory

It's been running smoothly for I think 2 months now.  But last week I started getting errors like the one below.

>> log scanned up to (291781603905)

>> log scanned up to (291781611845)

>> log scanned up to (291781620026)

>> log scanned up to (291781629903)

InnoDB: Error: tablespace id is 710 in the data dictionary

InnoDB: but in file ./vbulletin/threadviews.ibd it is 712!

110918 18:40:39  InnoDB: Assertion failure in thread 139974588552960 in file fil/fil0fil.c line 740

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html

InnoDB: about forcing recovery.

Aborted

innobackupex-1.5.1: Error: ibbackup child process has died at /usr/bin/innobackupex-1.5.1 line 336.

Last week it gets fixed after I restart the database then rerun the backup script.  But it keeps popping up again.  The table threadviews is actually empty, so I've tried dropping and recreating, but it doesn't fix it.

Any ideas?  Thanks

Johann

[Alexey Kopytov]

Hi,

On 19.09.11 12:36, Johann Vincent Paul Tagle wrote:
>
>>> log scanned up to (291781603905)
>>> log scanned up to (291781611845)
>>> log scanned up to (291781620026)
>>> log scanned up to (291781629903)
> InnoDB: Error: tablespace id is 710 in the data dictionary
> InnoDB: but in file ./vbulletin/threadviews.ibd it is 712!
> 110918 18:40:39 InnoDB: Assertion failure in thread 139974588552960 in file
> fil/fil0fil.c line 740
> InnoDB: We intentionally generate a memory trap.
> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
> InnoDB: If you get repeated assertion failures or crashes, even
> InnoDB: immediately after the mysqld startup, there may be
> InnoDB: corruption in the InnoDB tablespace. Please refer to
> InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html
> InnoDB: about forcing recovery.
> Aborted
> innobackupex-1.5.1: Error: ibbackup child process has died at
> /usr/bin/innobackupex-1.5.1 line 336.

It's https://bugs.launchpad.net/percona-xtrabackup/+bug/722638 which was
fixed last week. The fix is going to be released in XtraBackup 1.6.3.

Best regards,
Alexey.

[Johann Vincent Paul Tagle]

Awesome work as usual.  Thanks!

> --
> You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
> To post to this group, send email to percona-d...@googlegroups.com.
> To unsubscribe from this group, send email to percona-discuss...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/percona-discussion?hl=en.
>
>

[Johann Vincent Paul Tagle]

I'm sorry.  This is me being silly - how do I get the patch?  I'm new to Launchpad and can't seem to find my way around.

Thanks

Johann

[Johann Vincent Paul Tagle]

Okay think I got it.  Installed Bazaar VCS then did:

bzr branch lp:percona-xtrabackup/1.6 percona-xtrabackup-1.6

[Johann Vincent Paul Tagle]

Okay, so I followed the build instructions that came with the code, uploaded the created innobackupex, soft-linked innobackupex-1.5.1 to it, then tried to do my backup - still same error.  

Attached the file I created - just added the following lines to it to cover a previous issue:

mysql_send "set interactive_timeout=28800;

mysql_send "set wait_timeout=28800;

Any ideas?  Thanks.

Johann

[Alexey Kopytov]

Johann,

On 20.09.11 4:18, Johann Vincent Paul Tagle wrote:
> Okay, so I followed the build instructions that came with the code, uploaded
> the created innobackupex, soft-linked innobackupex-1.5.1 to it, then tried
> to do my backup - still same error.
>

You also have to make sure that xtrabackup binaries are installed and
used by the innobackupex script (i.e. they are in PATH). There is
information on where the binaries can be found at the bottom of BUILD.txt.

Best regards,
Alexey.

[Johann Vincent Paul Tagle]

Ah, it's another case of PEBKAC-NFI* for me... my bad.  Thanks!

Johann

*Problem Exists Between Keyboard And Chair - Not Following Instructions :-)

Best regards,
Alexey.