Importing a pool exported from OpenSolaris guest (over a dm-crypt device)
Jan Ploski
I'm using VirtualBox and having trouble importing a pool exported from
an OpenSolaris guest (snv_134 = the current 'dev' release) using zfs-
fuse 0.6.9 beta 3 on the Linux host.
Particularly, I created a dm-crypt block device over partition /dev/
sdb1, which I named "green", I passed it as a raw SATA disk to the
OpenSolaris guest using VirtualBox and then created a zfs pool
(version 22) named "green" there using this whole virtual disk, which
was called c11t1d0. Finally, I exported the pool.
After shutting down the guest, i'm now trying to import that pool on
the host. Having read an earlier thread in this forum I created a
symlink /dev/dsk/c11t1d0 ->
/dev/disk/by-id/dm-name-green on the host. However, try as I may, I'm
getting this error:
# ./zpool import green -d /dev/dsk
cannot import 'green': no such pool available
strace tells me that it is actually opening /dev/dsk/c11t1d0 and
reading from it:
stat64("/dev/dsk/c11t1d0", {st_mode=S_IFBLK|0660, st_rdev=makedev(253,
2), ...}) = 0
open("/dev/dsk/c11t1d0", O_RDONLY|O_LARGEFILE) = 6
fstat64(6, {st_mode=S_IFBLK|0660, st_rdev=makedev(253, 2), ...}) = 0
ioctl(6, BLKGETSIZE64, 0xbf851588) = 0
mmap2(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0xb73aa000
pread64(6,
"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 262144,
0) = 262144
pread64(6,
"\0\0\0\0\0\0\0\0\26\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 262144,
262144) = 262144
pread64(6, "\250G\375\345\6\27\235\310\0177\314t
\3\216\235\333,]\365"..., 262144, 1099510841344) = 262144
pread64(6, "<\224p\334F\2549\335>a\251\33\272]\224\227\304\251nX
\265"..., 262144, 1099511103488) = 262144
munmap(0xb73aa000, 266240) = 0
close(6) = 0
getdents64(5, /* 0 entries */, 32768) = 0
close(5) = 0
write(2, "cannot import \'green\': no such p"..., 46cannot import
'green': no such pool available
Am I doing anything wrong? Can you give me any tips how to get it
working or gather more information?
Best regards,
Jan Ploski
Jan Ploski
the same dm-crypt device (with -o version=22), I can import it in the
OpenSolaris guest, apparently without any problems. So I can use this
as a solution instead. Nevertheless, it is puzzling why it doesn't
work the other way around.
Tino
http://hub.opensolaris.org/bin/download/Community+Group+zfs/docs/ondiskformat0822.pdf
Note that with an 75% proabability I AM WRONG about what I write.
Perhaps my ideas or questions are nuts anyway, but others might read
this thread and it might be helpful.
On 26 Mai, 22:28, Jan Ploski <jpl...@gmx.de> wrote:
> I should add that after creating the same zfs pool using zfs-fuse over
> the same dm-crypt device (with -o version=22), I can import it in the
> OpenSolaris guest, apparently without any problems. So I can use this
Is this really the same pool? Have you written some (different!) data
to it from Linux and Solaris and seen this data again in the other OS?
(Yes this question sounds silly. But sometimes one does not spot the
most easy thing.)
ZFS is like a cockroach. You cannot easily elliminate it, if there is
some debris left. This is by purpose, of course, and a good thing
that it is very hard to kill. ZFS I mean, not cockroaches ;)
So even destroying a pool does not mean it is gone for real, you can
import it if you give the name. (It says - on the first invocation
only - that it imported a destroyed pool.)
It may be that Solaris sees a DIFFERENT pool now, possibly you created
TWO ZFS volumes which cover the same disk but both have a slighly
different layout. Due to the ZFS self healing capabilities it might
even take quite some time until you notice that, if you use them in
parallel!
Ah, and well, I think it is a little bit dangerous to have a ZFS
kernel run in both OS as well. This is why there is the "zoned" flag
in ZFS (but Linux does not know about Zones and Solaris thinks it is
not zoned because it is no zone) to accidentally prevent the use of a
ZFS outside a zone.
So please keep in mind that there is danger ahead, as when ZFS-FUSE is
able to access the Solaris ZFS in parallel (accidentally), this might
lead to data corruption. So perhaps having Linux unable to see the
Solaris ZFS should be considered a safety feature instead of a
problem ;)
> as a solution instead. Nevertheless, it is puzzling why it doesn't
> work the other way around.
It may be that Solaris natively adds some extra room at the start of
the raw device (extra solaris device header), such that the ZFS
structures (the first 4 MB of the device - and some sectors at the
logical end) land on a different place. Usually you can compensate
that on the Linux side using losetup:
(NOTE THAT THIS IS AN EXAMPLE, I DO NOT EVEN THINK THIS MIGHT WORK
THIS WAY)
modprobe loop
losetup -o 1024 /dev/loop0 /dev/disk/by-id/dm-name-green
zpool -d /dev/disk import
(I am not sure that loop works on head of dm-crypt or with ZFS this
way, so be warned, this example may break something.)
However Solaris also is able to read and write non-solaris disks,
too. Therefor it is possible (but I am not sure) that Solaris can
"mount" whole disk pools created by Linux, but Linux does not find the
on-disk structures if they were written with Solaris because Linux is
not Solaris-aware (in a dm-crypt setup).
There is a way to find out if this is the case (I am no ZFS nor
Solaris expert, it's all according to the docs):
You have to NULL the first and the last 5 MB of the device (all
commands in this message run on the Linux side):
dd if=/dev/null of=/dev/disk/by-id/dm-name-green bs=1048576 count=5
dd if=/dev/null of=/dev/disk/by-id/dm-name-green bs=1048576 count=5
seek=END-OF-DEVICE-MINUS-5M
(you have to find out the right number of "END-OF-DEVICE-MINUS-5M")
Then create the ZFS under Solaris.
Now look at the first 5 MB of the device again:
dd if=/dev/disk/by-id/dm-name-green bs=1048576 count=5 | od -tx1z >
FORMAT-SOLARIS.dump
It shall be mostly NUL bytes, though.
Then NULL the device again (as above) and create the drive with ZFS-
FUSE and do:
dd if=/dev/disk/by-id/dm-name-green bs=1048576 count=5 | od -tx1z >
FORMAT-LINUX.dump
Looking at the two outputs one usually can easily spot if there is a
displacement somewhere.
Notes:
If you do not NULL the end of the device, Solaris might find old
debris (even after destroying a pool) from the previous pool creation,
so be sure to NULL the end of the device, too.
If this is too much effort, just NULL the complete device:
dd if=/dev/zero of=/dev/disk/by-id/dm-name-green bs=1048576
According to the docs, only the first 4MB are the header, so I
extended it 1M to cover possible displacements on the solaris side.
Nulling 5MB at the end of the device is probably overkill, but better
be safe than sorrow.
And as always: Drive carefully! Read: It is dangerous to directly
work with raw devices as shown here. If you do not understand it or
are puzzled, then stop immediately with your action. It's quite too
easy to destroy the wrong thing by accident.
-Tino
Jan Ploski
I might have found an important difference: when I run Linux fdisk the
dm-crypt device in which I created a pool using zfs-fuse, I can see a
single partition of type GPT (id = ee) spanning over the entire
(virtual) disk. However, when I do the same over the dm-crypt device
in which I created a pool using the OpenSolaris guest, then I see no
partitions at all. On the other hand, displaying the partition table
of each "disk" using the "format" command within the OpenSolaris guest
shows the correct ZFS-or-EFI-label partitioning in both cases (i.e. 8
slices, last one marked as reserved).
Could it be that zfs-fuse requires this 'ee' partition to treat the
pool as valid? Can someone reproduce this "problem" independently? I
was not sure whether I should file a bug report for this or whether
it's just a stupid mistake on my part, but I now lean toward the first
option. :-)
Also, I am definitely able to import the zfs-fuse pool in OpenSolaris
and see its contents. I'm also aware of the dangers of using the same
pool in both environments in parallel, so I export/import it each
time. I'm going to implement additional safeguards to prevent myself
from starting the VM while the pool is still in use on the host.
Best regards,
Jan Ploski