Bug#557817: nbd has I/O errors with md-over-dmcrypt-over-nbd
martin f krafft
Version: 1:2.9.14-1
Severity: normal
I am trying to investigate a remote backup solution: nbd-server
gives me a block device from remote, which I locally crypt with
dmcrypt, and then use as a component of a RAID1. The backups are
written to the RAID1, and by rsync'ing the RAID, I effectively
generate an encrypted live-backup remotely.
Unfortunatly, that is only the theory, for in practice, the
dmcrypt+nbd combination doesn't seem to work (it might well be
a dmcrypt problem, though not likely):
% dd if=/dev/zero of=nbd-file bs=1M count=1 seek=1023
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00447344 s, 234 MB/s
% nbd-server 127.0.0.1:12345 `pwd`/nbd-file 1024M -C/dev/null
% netstat -natp | grep :12345
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 127.0.0.1:12345 0.0.0.0:* LISTEN 16385/nbd-server
% sudo nbd-client localhost 12345 /dev/nbd0
Negotiation: ..size = 1048576KB
bs=1024, sz=1048576
% echo foo > key
% sudo cryptsetup luksFormat /dev/nbd0 ./key
WARNING!
========
This will overwrite data on /dev/nbd0 irrevocably.
Are you sure? (Type uppercase yes): YES
% grep nbd /var/log/syslog | tail
Nov 24 17:24:37 piper nbd_server[16385]: connect from 127.0.0.1, assigned file is /home/madduck/.tmp/cdt.hPOcvv/nbd-file
Nov 24 17:24:37 piper nbd_server[16385]: Can't open authorization file /etc/nbd-server/allow (No such file or directory).
Nov 24 17:24:37 piper nbd_server[16385]: Authorized client
Nov 24 17:24:37 piper nbd_server[16405]: Starting to serve
Nov 24 17:24:37 piper nbd_server[16405]: Size of exported file/device is 1073741824
Nov 24 17:24:37 piper kernel: [537982.733434] nbd0: unknown partition table
% sudo cryptsetup luksOpen -d ./key /dev/nbd0 cryptnbd
Key slot 0 unlocked.
% dd if=/dev/zero of=loop-file bs=1M count=1 seek=1023
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00413988 s, 253 MB/s
% sudo losetup -vf ./loop-file
Loop device is /dev/loop0
% sudo mdadm -Cl1 -n2 -ayes /dev/md99 /dev/mapper/cryptnbd /dev/loop0
mdadm: array /dev/md99 started.
% cat /proc/mdstat
Personalities : [raid1]
md99 : active (auto-read-only) raid1 loop0[1] dm-7[0]
1047936 blocks [2/2] [UU]
resync=PENDING
unused devices: <none>
So far so good. Now as soon as I write anything to the array, it will start
sync'ing. And this is when the I/O errors start, but the RAID syncs fine
actually.
% tail -f /var/log/syslog | grep nbd&
[1] 17024 17025
% dd if=/dev/zero of=/dev/md99 bs=1 count=1
dd: opening `/dev/md99': Permission denied
% sudo dd if=/dev/zero of=/dev/md99 bs=1 count=1
1+0 records in
1+0 records out
1 byte (1 B) copied, 0.0993314 s, 0.0 kB/s
Nov 24 17:41:45 piper kernel: [539010.683028] end_request: I/O error, dev nbd0, sector 0
[…] (34 in between)
Nov 24 17:42:41 piper kernel: [539066.182929] md: md99: resync done.
Nov 24 17:42:41 piper kernel: [539066.212929] end_request: I/O error, dev nbd0, sector 0
Nov 24 17:42:41 piper kernel: [539066.435511] end_request: I/O error, dev nbd0, sector 0
Nov 24 17:42:43 piper mdadm[2599]: RebuildFinished event detected on md device /dev/md99
Nov 24 17:42:43 piper kernel: [539068.098878] RAID1 conf printout:
Nov 24 17:42:43 piper kernel: [539068.098883] --- wd:2 rd:2
Nov 24 17:42:43 piper kernel: [539068.098887] disk 0, wo:0, o:1, dev:dm-7
Nov 24 17:42:43 piper kernel: [539068.098891] disk 1, wo:0, o:1, dev:loop0
Nov 24 17:42:43 piper kernel: [539068.099206] end_request: I/O error, dev nbd0, sector 0
% cat /proc/mdstat
Personalities : [raid1]
md99 : active raid1 loop0[1] dm-7[0]
1047936 blocks [2/2] [UU]
976510912 blocks [2/2] [UU]
unused devices: <none>
Subsequently, I can read the entire array (I tried with dd) without
any further I/O errors, but when I write to it, the I/O errors
re-appear.
The problem only appears when dmcrypt is in between nbd and md, not
if I use /dev/nbd0 directly for the RAID1.
Thus this problem might well be with dmcrypt, but it's unlikely.
I tried the same setup with a loop device instead of nbd (loop0
directly, dmcrypt-on-loop1, RAID1 across loop0 and dmcrypt device),
and there were no I/O errors.
-- System Information:
Debian Release: squeeze/sid
APT prefers unstable
APT policy: (500, 'unstable'), (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.31-1-amd64 (SMP w/1 CPU core)
Locale: LANG=en_GB, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages nbd-server depends on:
ii adduser 3.111 add and remove users and groups
ii debconf [debconf-2.0] 1.5.28 Debian configuration management sy
ii libc6 2.10.1-7 GNU C Library: Shared libraries
ii libglib2.0-0 2.22.2-2 The GLib library of C routines
ii ucf 3.0025 Update Configuration File: preserv
nbd-server recommends no packages.
nbd-server suggests no packages.
-- debconf information excluded
--
.''`. martin f. krafft <madduck@d.o> Related projects:
: :' : proud Debian developer http://debiansystem.info
`. `'` http://people.debian.org/~madduck http://vcs-pkg.org
`- Debian - when you have better things to do than fixing systems