innobackupex-1.5.1: Waiting for log copying to finish (for 3+ hours so far)

[Andrew McQuerry]

Hi everyone -

I'm currently using "innobackupex-1.5.1" to perform a backup of our primary mysql DB, pushing it to a secondary server which will become our Slave.  I'm following the guide at this link. I've performed this procedure a few times before, and I even used this procedure to migrate this same DB from an old machine to it's current host.

The overall DB Size is 1.2TB, and it has taken about 21 hours just for the backup of data.  However, at this time, it's been sitting at the following status for about 3 hours, and I'm very worried about it failing before it's able to complete.

140605 06:36:20  innobackupex-1.5.1: Continuing after ibbackup has suspended

140605 06:36:20  innobackupex-1.5.1: Starting to lock all tables...

140605 06:36:20  innobackupex-1.5.1: All tables locked and flushed to disk

140605 06:36:20  innobackupex-1.5.1: Starting to backup non-InnoDB tables and files

innobackupex-1.5.1: in subdirectories of '/var/lib/mysql'

innobackupex-1.5.1: Backing up file '/var/lib/mysql/epc/db.opt'

innobackupex-1.5.1: Backing up files '/var/lib/mysql/ecds_statistics/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (91 files)

innobackupex-1.5.1: Backing up files '/var/lib/mysql/enodeb/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (38 files)

innobackupex-1.5.1: Backing up files '/var/lib/mysql/performance_schema/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (53 files)

innobackupex-1.5.1: Backing up files '/var/lib/mysql/mysql/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (74 files)

innobackupex-1.5.1: Backing up file '/var/lib/mysql/util/generator_4k.frm'

innobackupex-1.5.1: Backing up file '/var/lib/mysql/util/numbers.frm'

innobackupex-1.5.1: Backing up file '/var/lib/mysql/util/generator_64k.frm'

innobackupex-1.5.1: Backing up file '/var/lib/mysql/util/db.opt'

innobackupex-1.5.1: Backing up file '/var/lib/mysql/util/generator_256.frm'

innobackupex-1.5.1: Backing up file '/var/lib/mysql/util/generator_16.frm'

innobackupex-1.5.1: Backing up files '/var/lib/mysql/phpmyadmin/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (16 files)

innobackupex-1.5.1: Backing up files '/var/lib/mysql/lte/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (14 files)

140605 06:36:22  innobackupex-1.5.1: Finished backing up non-InnoDB tables and files

140605 06:36:22  innobackupex-1.5.1: Waiting for log copying to finish

A bit of history:

 - I've had problems getting the backup to the 2nd machine, primarily due to the InnoDB Log File size.  I've steadily increased it from 50M, to 600M, to 1.2G, and now to it's current value of 5G x 2 log files.  Each time I increased it, I was able to get further along in the backup before seeing the "log file has wrapped" error message causing the backup to die.  

My questions are:

 - What exact logs are being copied here? and how can I monitor that to ensure things are still moving in the background?

 - When I look at the processlist of the Master DB, it shows that "root" user has been in for 2.72 hours.  All of my other normal DB queries (Load Data, Select, etc) are showing "Waiting for global read lock".  Is there anything truly going on? or is there a way that I can kickstart it to continue on?

*************************** 1. row ***************************

     Id: 3

   User: event_scheduler

   Host: localhost

     db: NULL

Command: Daemon

   Time: 85721

  State: Waiting for next activation

   Info: NULL

*************************** 2. row ***************************

     Id: 436

   User: root

   Host: localhost

     db: NULL

Command: Sleep

   Time: 9821

  State: 

   Info: NULL

 

I thank anyone that's able to provide input.  As stated, I'm really scared of this thing failing.. after spending 23 hours pushing the 1.2TB of data to the other server.  I'd really hate for it to fail if there's something that I can do to get it over this speed bump.

Thanks!!!!

Andrew McQuerry

[Andrew McQuerry]

A short update.  My connection to the server died, killing the process with it.  I tried using the --prepare option, but the xtrabackup_logfile was not on the new slave for it to perform that task.  I'm re-starting the entire process again (I don't know of any other way to resume where it left off).  Another 20-23 hours of data transfer, and then i'll see if the script hangs at the same spot as today.

But please, if you have input, I'd still like to hear it.