Advice on dealing with permanent errors
60 views
Skip to first unread message
Budric
May 30, 2012, 2:03:50 PM5/30/12
to zfs-fuse
Hi,
I wanted to replace a drive in a functional RaidZ ("raid 5")
consisting of 3 drives. The new drive is bigger in size, and I want
to make the pool bigger by replacing all 3 drives eventually. The
last scrub completed without errors Monday May 28th, so I did not
issue another scrub before starting this on Wednesday. I removed one
of the drives, started the system, pool was in degraded mode (one
drive couldn't be opened obviously). I issued "zpool replace tank /
dev/old_drive /dev/new_drive" and it went on resilvering.
After a few hours of resilvering I see that there are 368 permanent
errors in various files. Some in a snapshot, some on the pool
itself. So I think there must have been errors in the 2 remaining
drives and now as it's trying to recover the information it can't.
I know hindsight is 20/20. I should have scrubbed the pool before
replacing drives. I should have left the drive in the pool and
replaced it while it was active, not doing it in degraded mode. But,
any advice on what I can do now? I still have the drive I removed.
If shutdown the machine now (as it's still resilvering), put the drive
in along side the new drive and restart the computer what will
happen? Will the device that could not be opened be part of a pool
again? Will the pool be in good state and resilver onto the bigger
drive will work?
Thanks for any suggestions.
I wanted to replace a drive in a functional RaidZ ("raid 5")
consisting of 3 drives. The new drive is bigger in size, and I want
to make the pool bigger by replacing all 3 drives eventually. The
last scrub completed without errors Monday May 28th, so I did not
issue another scrub before starting this on Wednesday. I removed one
of the drives, started the system, pool was in degraded mode (one
drive couldn't be opened obviously). I issued "zpool replace tank /
dev/old_drive /dev/new_drive" and it went on resilvering.
After a few hours of resilvering I see that there are 368 permanent
errors in various files. Some in a snapshot, some on the pool
itself. So I think there must have been errors in the 2 remaining
drives and now as it's trying to recover the information it can't.
I know hindsight is 20/20. I should have scrubbed the pool before
replacing drives. I should have left the drive in the pool and
replaced it while it was active, not doing it in degraded mode. But,
any advice on what I can do now? I still have the drive I removed.
If shutdown the machine now (as it's still resilvering), put the drive
in along side the new drive and restart the computer what will
happen? Will the device that could not be opened be part of a pool
again? Will the pool be in good state and resilver onto the bigger
drive will work?
Thanks for any suggestions.
Emmanuel Anne
May 30, 2012, 3:22:37 PM5/30/12
to zfs-...@googlegroups.com
368 permanent errors which appear 2 days after a full scrub just when you replace one of the drives, it sounds very much like there is a big bug somewhere or you are one of the most unlucky person on earth !... A nightmare in this situation. Not sure there is much solution here...
--
my zfs-fuse git repository : http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
Maybe playing with options --log-uberblocks / --min-uberblock-txg but not sure it would help here, the errors are probably at the pool level and these options are for the fs level.
In the same kind of idea : remove the new disk, run scrub again, but it will be long and it's unlikely to work.
Except that no idea... Hum, maybe try to import the pool with another zfs version, eventually one from opensolaris using some live dvd if there are still some available to see how it will work. I know nothing about opensolaris so I can't help here though.
2012/5/30 Budric <bud...@gmail.com>
--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/
my zfs-fuse git repository : http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary
Björn Kahl
May 30, 2012, 3:46:20 PM5/30/12
to zfs-...@googlegroups.com
Am 30.05.12 20:03, schrieb Budric:
How are the three drives connected to the system? The original three
and the new ones?
To me, this looks like you get random bit errors while reading, and /
or suffer from misdirected writes (blocks meant for the new drive
hitting one of the two old drives).
I had a similar problem once with an external 4-bay eSATA enclosure.
My specific combination of enclosure, controller card and driver
managed to screw up the port multiplexing and occasionally wrote blocks
to the wrong drive. Even ZFS doesn't like such hardware.
Best
Björn
--
| Bjoern Kahl +++ Siegburg +++ Germany |
| "googlelogin@-my-domain-" +++ www.bjoern-kahl.de |
| Languages: German, English, Ancient Latin (a bit :-)) |
> I wanted to replace a drive in a functional RaidZ ("raid 5")
> consisting of 3 drives. The new drive is bigger in size, and I want
> to make the pool bigger by replacing all 3 drives eventually. The
> last scrub completed without errors Monday May 28th, so I did not
> issue another scrub before starting this on Wednesday.
> consisting of 3 drives. The new drive is bigger in size, and I want
> to make the pool bigger by replacing all 3 drives eventually. The
> last scrub completed without errors Monday May 28th, so I did not
> issue another scrub before starting this on Wednesday.
> After a few hours of resilvering I see that there are 368 permanent
> errors in various files.
More than 360 errors in two days sounds wrong.
> errors in various files.
How are the three drives connected to the system? The original three
and the new ones?
To me, this looks like you get random bit errors while reading, and /
or suffer from misdirected writes (blocks meant for the new drive
hitting one of the two old drives).
I had a similar problem once with an external 4-bay eSATA enclosure.
My specific combination of enclosure, controller card and driver
managed to screw up the port multiplexing and occasionally wrote blocks
to the wrong drive. Even ZFS doesn't like such hardware.
Best
Björn
--
| Bjoern Kahl +++ Siegburg +++ Germany |
| "googlelogin@-my-domain-" +++ www.bjoern-kahl.de |
| Languages: German, English, Ancient Latin (a bit :-)) |
Seth Heeren
May 30, 2012, 4:31:44 PM5/30/12
to zfs-fuse
My thought would be: kill the resilver and check the hardware.
Try to move to a different machine (PSU, memory and Sata controllers).
You might move to another zfs implementation but, my main suspect
would be the hardware there.
Try to move to a different machine (PSU, memory and Sata controllers).
You might move to another zfs implementation but, my main suspect
would be the hardware there.
Bud Bundy
May 31, 2012, 10:34:15 AM5/31/12
to zfs-...@googlegroups.com
Thanks for the suggestions. I shutdown the system, put the drive I took out back in, and turned it back on to start resilvering. The errors remained. There are 444 now. Most of the errors were in a snapshot I took, I didn't need it so I erased it, bringing the errors down to 85 files, which I'll try to recover from backup or lose.
The resilver process that I started a second time eventually crashed. Syslog has this, but to me that's not very helpful. A segfault from zfs-fuse followed by cron-hourly complaining.
May 31 07:16:35 backup kernel: [44605.511111] zfs-fuse[1157]: segfault at 2f93b8c6a6d0 ip 00002b93b9038e84 sp 00002b93bbd64d00 error 4 in libpthread-2.$
May 31 07:17:02 backup CRON[6993]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
May 31 07:19:11 backup kernel: [44761.792350] INFO: task zfs-fuse:1153 blocked for more than 120 seconds.
May 31 07:19:11 backup kernel: [44761.792494] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 31 07:19:11 backup kernel: [44761.792638] zfs-fuse D 0000000000000000 0 1153 1 0x00000000
May 31 07:19:11 backup kernel: [44761.792652] ffff88020ccf5cc8 0000000000000082 0000000000000000 ffffffffffffffe0
May 31 07:19:11 backup kernel: [44761.792667] ffff88020ccf5fd8 ffff88020ccf5fd8 ffff88020ccf5fd8 0000000000013780
May 31 07:19:11 backup kernel: [44761.792679] ffff88021068c4d0 ffff88020d49c4d0 ffff88020d49c4d0 ffff88020e4dad80
May 31 07:19:11 backup kernel: [44761.792692] Call Trace:
May 31 07:19:11 backup kernel: [44761.792713] [<ffffffff8165a88f>] schedule+0x3f/0x60
May 31 07:19:11 backup kernel: [44761.792726] [<ffffffff8106b915>] exit_mm+0x85/0x130
May 31 07:19:11 backup kernel: [44761.792737] [<ffffffff8106bb2e>] do_exit+0x16e/0x420
May 31 07:19:11 backup kernel: [44761.792748] [<ffffffff8109d9bf>] ? __unqueue_futex+0x3f/0x80
May 31 07:19:11 backup kernel: [44761.792761] [<ffffffff8107a2ca>] ? __dequeue_signal+0x6a/0xb0
May 31 07:19:11 backup kernel: [44761.792772] [<ffffffff8106bf84>] do_group_exit+0x44/0xa0
May 31 07:19:11 backup kernel: [44761.792783] [<ffffffff8107ce0c>] get_signal_to_deliver+0x21c/0x420
May 31 07:19:11 backup kernel: [44761.792795] [<ffffffff81013865>] do_signal+0x45/0x130
May 31 07:19:11 backup kernel: [44761.792806] [<ffffffff8106652b>] ? do_fork+0x15b/0x2e0
May 31 07:19:11 backup kernel: [44761.792816] [<ffffffff810a0a0c>] ? do_futex+0x7c/0x1b0
May 31 07:19:11 backup kernel: [44761.792825] [<ffffffff810a0c4a>] ? sys_futex+0x10a/0x1a0
May 31 07:19:11 backup kernel: [44761.792835] [<ffffffff81013b15>] do_notify_resume+0x65/0x80
May 31 07:19:11 backup kernel: [44761.792846] [<ffffffff8101c668>] ? sys_clone+0x28/0x30
May 31 07:19:11 backup kernel: [44761.792856] [<ffffffff81665050>] int_signal+0x12/0x17
My kernel version is: Linux backup 3.2.0-24-generic #39-Ubuntu, Ubuntu 12.04.
At the moment I don't have different hardware to try it on. It's resilvering for the third time, if that fails I'll try memtest or something.
Reply all
Reply to author
Forward
0 new messages