ZFS: Invalid SEND/RECV stream still

[Larry Rosenman]

between 10.0-BETA1 and 11.0-CURRENT.
I've been fighting this battle for a LONG time. I had hoped(!) that it
would be solved by the systems being
closer in rev but that didn't happen.

I'd LOVE someone to look at it.

I can provide ssh / root access to both machines.


Error Message:
received 320KB stream in 1 seconds (320KB/sec)
receiving incremental stream of vault/var@2013-10-25 into
zroot/backups/TBH/var@2013-10-25
cannot receive incremental stream: invalid backup stream
$

Script:
$ cat backup-TBH-ZFS.sh
#!/bin/sh
DATE=`date "+%Y-%m-%d"`
#DATE2=2013-03-24
DATE2=`date -v "-1d" "+%Y-%m-%d"`
# snap the source
ssh ro...@tbh.lerctr.org zfs snapshot -r vault@${DATE}
# zfs copy the source to here.
ssh ro...@tbh.lerctr.org "zfs send -R -D -I vault@${DATE2}
vault@${DATE} | \
ssh home.lerctr.org \"zfs recv -F -u -v -d zroot/backups/TBH\""
# make sure we NEVER allow the backup stuff to automount.
/sbin/zfs list -H -t filesystem -r zroot/backups/TBH| \
awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh
$

Uname's:
$ uname -a
FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #80 r257016:
Wed Oct 23 17:36:15 CDT 2013
ro...@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64
$ ssh tbh uname -a
FreeBSD thebighonker.lerctr.org 10.0-BETA1 FreeBSD 10.0-BETA1 #1
r256791: Sun Oct 20 12:28:55 CDT 2013
ro...@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER amd64
$

What else do y'all need?



--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: l...@lerctr.org
US Mail: 108 Turvey Cove, Hutto, TX 78634-5688
_______________________________________________
freeb...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-...@freebsd.org"

[Ian Lepore]

I notice you're using ssh and a pipe -- does your shell logon script
echo anything to stdout when you connect? I've had that interfere with
piped ssh stuff. Most shells provide a way for the logon script to
determine "is this an interactive session?" and you can use that to
avoid echoing anything that would interfere with non-interactive
automation.

-- Ian

[Larry Rosenman]

no, it does NOT.

I've verified that, plus this error is a LONG way into the stream. (I
cut a bunch
of the previous output).

I've even gathered tmp files from this before, but no one had the time
to look at it.

Thanks for at least looking.

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: l...@lerctr.org
US Mail: 108 Turvey Cove, Hutto, TX 78634-5688

[Larry Rosenman]

On 2013-10-25 14:33, Garrett Wollman wrote:
> In article <c0a42b456769e22f...@webmail.lerctr.org>,

> Larry Rosenman <l...@lerctr.org> wrote:
>
>> Error Message:
>> received 320KB stream in 1 seconds (320KB/sec)
>> receiving incremental stream of vault/var@2013-10-25 into
>> zroot/backups/TBH/var@2013-10-25
>> cannot receive incremental stream: invalid backup stream
>

> I've seen this happen in one very specific case: if the receiving
> machine is doing automatic snapshots, and happens to do a snapshot of
> the filesystem being received *while it's being received*. In that
> case, it appears that the filesystem on disk gets subtly corrupted,
> and the only remedy is to destroy it on the receiving machine and
> start all over (making sure to disable automatic snapshots this time).
>
> "zfs receive -F" will rollback snapshots made on the receiving system
> that don't exist on the sending system, but it only does this at the
> beginning of the stream -- it can't prevent further snapshots from
> being taken during reception of an incremental stream which foul up
> the works.
>
> -GAWollman
There are NO automatic snapshots in play here at all :(

[J David]

On Fri, Oct 25, 2013 at 3:36 PM, Larry Rosenman <l...@lerctr.org> wrote:
>>> cannot receive incremental stream: invalid backup stream

It may help to decouple the steps. I.e. do a "zfs send blah blah
>file" then copy the file and do a "zfs recv blah blah <file" on the
second host.

If nothing else, it should help you demonstrate that it's not the
machine-to-machine copy that's at fault, because you can take the
sha256 sum of the file on both sides and prove it's the same.

Good luck!

[Larry Rosenman]

On 2013-10-25 17:09, J David wrote:
> On Fri, Oct 25, 2013 at 3:36 PM, Larry Rosenman <l...@lerctr.org> wrote:
>>>> cannot receive incremental stream: invalid backup stream
>
> It may help to decouple the steps. I.e. do a "zfs send blah blah
>> file" then copy the file and do a "zfs recv blah blah <file" on the
> second host.
>
> If nothing else, it should help you demonstrate that it's not the
> machine-to-machine copy that's at fault, because you can take the
> sha256 sum of the file on both sides and prove it's the same.
>
> Good luck!

we've tried this, and still the same issue.

So. I'm not sure how to diagnose it.

The message out of zfs send/recv is not very informative.

Other ideas?

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: l...@lerctr.org
US Mail: 108 Turvey Cove, Hutto, TX 78634-5688

[J David]

On Fri, Oct 25, 2013 at 6:11 PM, Larry Rosenman <l...@lerctr.org> wrote:
> we've tried this, and still the same issue.

Post that and the shorter local-only reproduction then, so people
don't spend time looking at the network layer.

> So. I'm not sure how to diagnose it.
>
> The message out of zfs send/recv is not very informative.
>
> Other ideas?

Two:

1) Do a non-incremental send/receive and see if it works.

2) Set up this exact test between two 10.0-BETA1 machines and then two
11-CURRENT machines to demonstrate whether the problem is specific to
one version or the other, or limited to the interaction between the
two versions. If the latter, then you may have to start bisecting
commits to find the breaking point.

Good luck!

[Larry Rosenman]

On 2013-10-25 17:17, J David wrote:
> On Fri, Oct 25, 2013 at 6:11 PM, Larry Rosenman <l...@lerctr.org> wrote:
>> we've tried this, and still the same issue.
>
> Post that and the shorter local-only reproduction then, so people
> don't spend time looking at the network layer.
>
>> So. I'm not sure how to diagnose it.
>>
>> The message out of zfs send/recv is not very informative.
>>
>> Other ideas?
>
> Two:
>
> 1) Do a non-incremental send/receive and see if it works.

This worked.

>
> 2) Set up this exact test between two 10.0-BETA1 machines and then two
> 11-CURRENT machines to demonstrate whether the problem is specific to
> one version or the other, or limited to the interaction between the
> two versions. If the latter, then you may have to start bisecting
> commits to find the breaking point.

Hard to do, as my choices are the 2 machines I have.

This has been going on since 8.x (see posts from me in the list
archives from March to now).


>
> Good luck!

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: l...@lerctr.org
US Mail: 108 Turvey Cove, Hutto, TX 78634-5688