streaming backup failed with non-helpful error message

[David Stainton]

Greetings,

I need to perform streaming xtradb backups because my database machines don't have enough local disk

available for snapshots. I've done it successfully a couple times however recently an attempt failed.

Is this not a totally useless error message? Or is there some information I'm not seeing here?

innobackupex --stream=xbstream /tmp --defaults-file=/etc/my.cnf --slave-info | pv | ssh root@backup-boats-db 'xbstream -x -C /disk1/mysql/test'

...

innobackupex: Backing up file '/var/lib/mysql/boats_08/cats.frm'

120515 06:22:56  innobackupex: Finished backing up .frm, .MRG, .MYD, .MYI, .TRG, .TRN, .ARM, .ARZ, .CSV, .CSM and .opt files

innobackupex: Resuming ibbackup

xtrabackup: The latest check point (for incremental): '13759611938184'

xtrabackup: Stopping log copying thread.

xtrabackup: Error: log_copying_thread failed.

120515 06:22:58  innobackupex: All tables unlocked                                                                  <=>                                                                ]

120515 06:22:58  innobackupex: Connection to database server closed

innobackupex: Backup created in directory '/tmp'

innobackupex: MySQL binlog position: filename 'boats-db01-binlog.014002', position 920231549

innobackupex: MySQL slave binlog position: master host '', filename '', position

120515 06:22:58  innobackupex: xtrabackup failed! (exit code 1)  The backup may not be complete.

 903GB 11:28:24 [22.4MB/s] [                                                                                           <=>                                                             ]

[root@boats-db01 ~]# innobackupex --stream=xbstream /tmp --defaults-file=/etc/my.cnf --slave-info | pv | ssh root@backup-boats-db 'xbstream -x -C /disk1/mysql/test'

I'm curious what people think about why this failed!

It coulda been the ssh connection that failed?

Or an on disk error?

Cheers,

David

[David Stainton]

Hi,

How can I find out why this is failing?

I tried it again and it again failed with the same error message.

David

[David Stainton]

We are running Percona 5.1.55-rel12.6-log and using percona-xtrabackup-2.0.0-417.rhel5.x86_64.rpm

Perhaps there is a bug in our mysql version or with that version of xtrabackup?

David

[David Stainton]

Could it have anything to do with this warning message in the beginning of running innobackupex? :

120515 17:27:24  InnoDB: Warning: allocated tablespace 772, old maximum was 9

in here:

[root@boat-db01 ~]# innobackupex --stream=xbstream /tmp --defaults-file=/etc/my.cnf | nc backup-boat-db 9666

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy

and Percona Inc 2009-2012.  All Rights Reserved.

This software is published under

the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

120515 17:27:16  innobackupex: Starting mysql with options:  --defaults-file='/etc/my.cnf' --unbuffered --

120515 17:27:16  innobackupex: Connected to database with mysql child process (pid=10144)

120515 17:27:22  innobackupex: Connection to database server closed

IMPORTANT: Please check that the backup run completes successfully.

           At the end of a successful backup run innobackupex

           prints "completed OK!".

innobackupex: Using mysql  Ver 14.14 Distrib 5.1.55, for unknown-linux-gnu (x86_64) using readline 5.1

innobackupex: Using mysql server version Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.

innobackupex: Created backup directory /tmp

120515 17:27:22  innobackupex: Starting mysql with options:  --defaults-file='/etc/my.cnf' --unbuffered --

120515 17:27:22  innobackupex: Connected to database with mysql child process (pid=10258)

120515 17:27:24  innobackupex: Connection to database server closed

120515 17:27:24  innobackupex: Starting ibbackup with command: xtrabackup  --defaults-file="/etc/my.cnf" --backup --suspend-at-end --target-dir=/tmp --stream=xbstream

innobackupex: Waiting for ibbackup (pid=10300) to suspend

innobackupex: Suspend file '/tmp/xtrabackup_suspended'

xtrabackup version 2.0.0 for Percona Server 5.1.59 unknown-linux-gnu (x86_64) (revision id: 417)

xtrabackup: uses posix_fadvise().

xtrabackup: cd to /var/lib/mysql

xtrabackup: Target instance is assumed as followings.

xtrabackup:   innodb_data_home_dir = ./

xtrabackup:   innodb_data_file_path = ibdata1:1000M:autoextend

xtrabackup:   innodb_log_group_home_dir = ./

xtrabackup:   innodb_log_files_in_group = 2

xtrabackup:   innodb_log_file_size = 1073741824

xtrabackup: using O_DIRECT

120515 17:27:24  InnoDB: Warning: allocated tablespace 772, old maximum was 9

>> log scanned up to (13769977313464)

[01] Streaming ./ibdata1

>> log scanned up to (13769978544611)

>> log scanned up to (13769980201056)

>> log scanned up to (13769981458447)

>> log scanned up to (13769982853138)

>> log scanned up to (13769983888634)

[Alexey Kopytov]

Hi David,

On Tue, 15 May 2012 11:19:59 -0700, David Stainton wrote:

> Greetings,
>
>
> I need to perform streaming xtradb backups because my database machines
> don't have enough local disk
> available for snapshots. I've done it successfully a couple times however
> recently an attempt failed.
>
> Is this not a totally useless error message? Or is there some information
> I'm not seeing here?

I don't see any messages that might be relevant either, except the
"log_copying_thread failed" one. I don't see any code paths that would
result in log copying failure without additional diagnostics.

It could have been a failing ssh connection, or an xbstream failure.
Though I would expect some additional error messages in both cases.

Can you try saving the stream to a file on the remote host rather than
unpacking it with xbstream simultaneously? That is, does the following
fail as well?

innobackupex --stream=xbstream /tmp --defaults-file=/etc/my.cnf
--slave-info | pv | ssh root@backup-boats-db 'cat -
/disk1/mysql/test/backup.xbstream'

Best regards,
Alexey.

[David Stainton]

I actually got it working with an older version of xtrabackup; xtrabackup-1.6.6-332.rhel5.x86_64.rpm

Performed a flawless streaming tar backup to the remote machine over netcat without a problem.

I therefore suspect a interaction bug between Percona 5.1.55-rel12.6-log and percona-xtrabackup-2.0.0-417.rhel5.x86_64.rpm;

Cheers,

David

--
You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
To post to this group, send email to percona-d...@googlegroups.com.
To unsubscribe from this group, send email to percona-discuss...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/percona-discussion?hl=en.