Lost ZFS pools after reboot due to different HD order

[ssc]

Lost ZFS pools after reboot due to different HD order


Hi all,

I am running Kubuntu 8.10 64 on an AMD Athlon(tm) 64 X2 Dual Core
Processor 5200+ system.
The system itself is running from a rather conventional ext3 / md HD
mirror connected to the motherboard IDE controller.

In addition to that, I have 3 off-board PCI-card IDE controllers, each
having 4 harddrives attached which are configured as one raidz1 ZFS
pool each, so 3 pools in total.

Everything was fine for a while, I did some work with the data in the
pool and rebooted quite a few times.

Now for some reason, the IDE controllers seem to be detected in a
different order and therefore the harddrives are assigned different
names (sda, sdb, etc.). Therefore, the zpools are unavailable.

This is the info I got when I imported the pools from FreeBSD where
they were created:

user@host:~$ sudo zpool import
pool: zpool01
id: 5464946640449943486
state: ONLINE
status: The pool is formatted using an older on-disk version.
action: The pool can be imported using its name or numeric
identifier, though
some features will not be available without an explicit
'zpool upgrade'.
config:

zpool01 ONLINE
raidz1 ONLINE
sdi ONLINE
sdj ONLINE
sdk ONLINE
sdl ONLINE

pool: zpool02
id: 7739701221919534393
state: ONLINE
status: The pool is formatted using an older on-disk version.
action: The pool can be imported using its name or numeric
identifier, though
some features will not be available without an explicit
'zpool upgrade'.
config:

zpool02 ONLINE
raidz1 ONLINE
sde ONLINE
sdf ONLINE
sdg ONLINE
sdh ONLINE

pool: zpool03
id: 512158936459898782
state: ONLINE
status: The pool is formatted using an older on-disk version.
action: The pool can be imported using its name or numeric
identifier, though
some features will not be available without an explicit
'zpool upgrade'.
config:

zpool03 ONLINE
raidz1 ONLINE
sda ONLINE
sdb ONLINE
sdc ONLINE
sdd ONLINE


This is the info I get now:

user@host:~$ sudo zpool status
pool: zpool01
state: UNAVAIL
status: One or more devices could not be opened. There are
insufficient
replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool
online'.
see: http://www.sun.com/msg/ZFS-8000-3C
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
zpool01 UNAVAIL 0 0 0 insufficient replicas
raidz1 UNAVAIL 0 0 0 insufficient replicas
sdi UNAVAIL 0 0 0 cannot open
sdj UNAVAIL 0 0 0 cannot open
sdk ONLINE 0 0 0
sdl ONLINE 0 0 0

pool: zpool02
state: UNAVAIL
status: The pool is formatted using an older on-disk format. The
pool can
still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is
done, the
pool will no longer be accessible on older software versions.
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
zpool02 UNAVAIL 0 0 0 insufficient replicas
raidz1 UNAVAIL 0 0 0 corrupted data
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
sdg ONLINE 0 0 0
sdh ONLINE 0 0 0

pool: zpool03
state: UNAVAIL
status: One or more devices could not be used because the label is
missing
or invalid. There are insufficient replicas for the pool to
continue
functioning.
action: Destroy and re-create the pool from a backup source.
see: http://www.sun.com/msg/ZFS-8000-5E
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
zpool03 UNAVAIL 0 0 0 insufficient replicas
raidz1 UNAVAIL 0 0 0 insufficient replicas
sda FAULTED 0 0 0 corrupted data
sdb FAULTED 0 0 0 corrupted data
sdc FAULTED 0 0 0 corrupted data
sdd FAULTED 0 0 0 corrupted data


When I work my way through dmesg to see which disk is actually
configured as what sda sdb etc. device, I see that the order is
different than what it used to be when I imported the pools.

The system disks are unaffected of this arbitrary change as they are
referenced by UUIDs.

How can I fix this and how can I prevent this from happening again ?
Why does this occur at all ? I thought ZFS searches the disks in the
system and figures out itself what disk belongs to what pool ?

Any help would be greatly appreciated, I am looking at losing ~3TB of
private media.

Cheers,

SSC

[Frank]

0) DON'T PANIC! (Apologies to Douglas Adams)

1) zpool import -d /dev/disk/by-id

This should list your pool or pools. If so, proceed to 2).

If not, see 0, and consider stuffing around with forcing disk
orderings and/or temporarily physically disconnecting any other disks
that might have been added after you first imported your pool(s). Once
the same set of disks is attached to the same set of controllers as
was extant in your previous import *and nothing else*, udev *should*
reattach with the same /dev/sd* nomenclature (with the phase of the
moon correct and the stars aligned). In this case, your pool will
import correctly. IF you get to this point, IMMEDIATELY execute "zpool
export poolname" (without the quotes and substituting appropriately
for "poolname"). Then proceed to 2).

2) Execute "zpool import -d /dev/disk/by-id poolname" (without the
quotes, and substitute appropriately for "poolname"). This will import
things with reasonably unique identifiers (that happen not to be uuids
- which don't seem to work here for some reason or other) and
importantly are not volatile w.r.t. udev stuffing around at boottime.

With a little luck, this should bring your pool(s) back with disk
identifiers that are more stable than /dev/sd* identifiers under udev.

Good luck (and remember point 0)!

HTH,
Frank Horowitz

[Steven Samuel Cole]

Hey Frank,

what can I say, your reply is a full 10 out of 10.

The controller / HD order happened to be right again the next time I
booted and I exported the pools right away to keep them safe.
Was trying to get my head around udev rules already when I got your
email, saved me loads of work! :-)))

Thank you _very_ much!

Cheers,

SSC


Frank schrieb: