After power problems in the datacenter(master server powered off all of a sudden) my master and replicating slave became not synced. I decided to re-create the slave using innobackupex as follows:
After about a minute this command crashes with the following output:
<skipped>
>> log scanned up to (1008663005500) >> log scanned up to (1008663286889) >> log scanned up to (1008663565421) >> log scanned up to (1008663831324)
innobackupex-1.5.1: Backing up files '/var/lib/mysql/db/*.ibd' (40 files)
>> log scanned up to (1008664127219) >> log scanned up to (1008664430098) >> log scanned up to (1008664707455) >> log scanned up to (1008664991956) >> log scanned up to (1008665255594) >> log scanned up to (1008665527730) >> log scanned up to (1008665769134)
The file 'db/foo.ibd' may not be InnoDB datafile or may be corrupted. tar_append_tree("db/foo.ibd", "db/foo.ibd"): Input/output error innobackupex-1.5.1: tar returned with exit code 255. innobackupex-1.5.1: Error: Failed to stream '/var/lib/mysql/db/foo.ibd': at /usr/bin/innobackupex-1.5.1 line 336.
This command worked just fine in the very first time when I setup the slave but now it looks like there is a data corruption problem in the master, right? However, there are no error entries in the master log so far...
I'm kinda new to this, but aren't you supposed to failover to slave and recreate your master from it in such a situation? Otherwise what was the point of slave?
On Thu, 28 Jul 2011 09:50:09 +0400, Pavel Shevaev wrote: > Hi!
> After power problems in the datacenter(master server powered off all > of a sudden) my master and replicating slave became not synced. I > decided to re-create the slave using innobackupex as follows:
> After about a minute this command crashes with the following output:
> <skipped> >>> log scanned up to (1008663005500) >>> log scanned up to (1008663286889) >>> log scanned up to (1008663565421) >>> log scanned up to (1008663831324) > innobackupex-1.5.1: Backing up files '/var/lib/mysql/db/*.ibd' (40 > files) >>> log scanned up to (1008664127219) >>> log scanned up to (1008664430098) >>> log scanned up to (1008664707455) >>> log scanned up to (1008664991956) >>> log scanned up to (1008665255594) >>> log scanned up to (1008665527730) >>> log scanned up to (1008665769134) > The file 'db/foo.ibd' may not be InnoDB datafile or may be corrupted. > tar_append_tree("db/foo.ibd", "db/foo.ibd"): Input/output error > innobackupex-1.5.1: tar returned with exit code 255. > innobackupex-1.5.1: Error: Failed to stream > '/var/lib/mysql/db/foo.ibd': at /usr/bin/innobackupex-1.5.1 line > 336.
> This command worked just fine in the very first time when I setup the > slave but now it looks like there is a data corruption problem in the > master, right? However, there are no error entries in the master log > so far...
> What would you recommend in this situation?
> -- > Best regards, Pavel
-- Alexey Yurchenko, Codership Oy, www.codership.com Skype: alexey.yurchenko, Phone: +358-400-516-011
<alexey.yurche...@codership.com> wrote: > I'm kinda new to this, but aren't you supposed to failover to slave and > recreate your master from it in such a situation? Otherwise what was the > point of slave?
You are right and this is what I did in the past but in my situation master is fine(or pretends to be fine). Switching to a slave, restoring the master and switching back to it is quite a long procedure which I'd really to avoid unless there is a good reason.
> After power problems in the datacenter(master server powered off all
> of a sudden) my master and replicating slave became not synced. I
> decided to re-create the slave using innobackupex as follows:
> After about a minute this command crashes with the following output:
> <skipped>>> log scanned up to (1008663005500)
> >> log scanned up to (1008663286889)
> >> log scanned up to (1008663565421)
> >> log scanned up to (1008663831324)
> innobackupex-1.5.1: Backing up files '/var/lib/mysql/db/*.ibd' (40 files)>> log scanned up to (1008664127219)
> >> log scanned up to (1008664430098)
> >> log scanned up to (1008664707455)
> >> log scanned up to (1008664991956)
> >> log scanned up to (1008665255594)
> >> log scanned up to (1008665527730)
> >> log scanned up to (1008665769134)
> The file 'db/foo.ibd' may not be InnoDB datafile or may be corrupted.
> tar_append_tree("db/foo.ibd", "db/foo.ibd"): Input/output error
> innobackupex-1.5.1: tar returned with exit code 255.
> innobackupex-1.5.1: Error: Failed to stream
> '/var/lib/mysql/db/foo.ibd': at /usr/bin/innobackupex-1.5.1 line 336.
> This command worked just fine in the very first time when I setup the
> slave but now it looks like there is a data corruption problem in the
> master, right? However, there are no error entries in the master log
> so far...
On Wed, Aug 3, 2011 at 3:36 AM, Percona on-call <onc...@percona.com> wrote: > Pavel,
> Sorry for the delay.
> Do you mean an xtrabackup backup to a local disk is not possible > because it takes long or was it for another reason?
Looks like the table in question is corrupted :( I tried the non-stream version of xtrabackup and it also complained about this particular table. The database server currently works just fine, reads/writes do happen with this table.
If anyone can recommend any tool which could tell exactly what is wrong with this table I'll be grateful. I'm scared to run check within mysql itself since I've experienced a couple of time server crashes during this procedure when checking corrupted tables.
