dump | restore fails: unknown tape header type 1853384566

[Mikhail T.]

Hello!

I'm trying to migrate a filesystem from one disk to another using:

dump a0hCf 0 32 - /old | restore -rf -

(/old is already mounted read-only). The process runs for a while and
then stops with:

[...]
DUMP: 22.85% done, finished in 3:57 at Tue Mar 24 01:03:21 2009
DUMP: 24.66% done, finished in 3:50 at Tue Mar 24 01:00:58 2009
DUMP: 26.44% done, finished in 3:43 at Tue Mar 24 00:59:14 2009
unknown tape header type 1853384566
abort? [yn]

Any idea, what's going on? Why can't FreeBSD's restore read FreeBSD's
dump's output?

The system runs 7.0-STABLE from July 6th, i386. I just tried updating
the dump and restore from source (latest 7.x) -- the error is the same...

Please, advise. Thanks!

-mi

_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"

[Daniel O'Connor]

On Tuesday 24 March 2009 11:55:07 Mikhail T. wrote:
> I'm trying to migrate a filesystem from one disk to another using:
>
> dump a0hCf 0 32 - /old | restore -rf -
>
> (/old is already mounted read-only). The process runs for a while and
> then stops with:
>
> [...]
> DUMP: 22.85% done, finished in 3:57 at Tue Mar 24 01:03:21 2009
> DUMP: 24.66% done, finished in 3:50 at Tue Mar 24 01:00:58 2009
> DUMP: 26.44% done, finished in 3:43 at Tue Mar 24 00:59:14 2009
> unknown tape header type 1853384566
> abort? [yn]
>
> Any idea, what's going on? Why can't FreeBSD's restore read FreeBSD's
> dump's output?

What happens if you don't use the cache?
Also, do you really want -h 0? That means you skip nodump marked files at a
level 0 dump.

--
Daniel O'Connor software and network engineer
for Genesis Software - http://www.gsoft.com.au
"The nice thing about standards is that there
are so many of them to choose from."
-- Andrew Tanenbaum
GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C

[Mikhail T.]

Daniel O'Connor написав(ла):

> On Tuesday 24 March 2009 11:55:07 Mikhail T. wrote:
>
>> I'm trying to migrate a filesystem from one disk to another using:
>>
>> dump a0hCf 0 32 - /old | restore -rf -
>>
>> (/old is already mounted read-only). The process runs for a while and
>> then stops with:
>>
>> [...]
>> DUMP: 22.85% done, finished in 3:57 at Tue Mar 24 01:03:21 2009
>> DUMP: 24.66% done, finished in 3:50 at Tue Mar 24 01:00:58 2009
>> DUMP: 26.44% done, finished in 3:43 at Tue Mar 24 00:59:14 2009
>> unknown tape header type 1853384566
>> abort? [yn]
>>
>> Any idea, what's going on? Why can't FreeBSD's restore read FreeBSD's
>> dump's output?
>>
>
> What happens if you don't use the cache?
>

No big difference:

dump a0f - /old | restore -rf -
[...]
DUMP: 17.25% done, finished in 3:27 at Tue Mar 24 05:42:00 2009
DUMP: 20.36% done, finished in 3:09 at Tue Mar 24 05:28:13 2009
DUMP: 23.83% done, finished in 2:50 at Tue Mar 24 05:14:32 2009
unknown tape header type -621260722
abort? [yn]

Looks like a junk value somewhere... Unitialized variable or some such.

[Danny Braniss]

> Daniel O'Connor написав(ла):
>
> On Tuesday 24 March 2009 11:55:07 Mikhail T. wrote:
>
>
> >> I'm trying to migrate a filesystem from one disk to another using:
> >>
> >> dump a0hCf 0 32 - /old | restore -rf -
> >>
> >> (/old is already mounted read-only). The process runs for a while and> >> then stops with:
> >>
> >> [...]
> >> DUMP: 22.85% done, finished in 3:57 at Tue Mar 24 01:03:21 2009
> >> DUMP: 24.66% done, finished in 3:50 at Tue Mar 24 01:00:58 2009
> >> DUMP: 26.44% done, finished in 3:43 at Tue Mar 24 00:59:14 2009
> >> unknown tape header type 1853384566> >> abort? [yn]
> >>
> >> Any idea, what's going on? Why can't FreeBSD's restore read FreeBSD's
> >> dump's output?
> >>
> >>
> What happens if you don't use the cache?
> >
> No big difference:
> > dump a0f - /old | restore -rf -
> [...]
> DUMP: 17.25% done, finished in 3:27 at Tue Mar 24 05:42:00 2009
> DUMP: 20.36% done, finished in 3:09 at Tue Mar 24 05:28:13 2009
> DUMP: 23.83% done, finished in 2:50 at Tue Mar 24 05:14:32 2009
> unknown tape header type -621260722> abort? [yn]
>
> Looks like a junk value somewhere... Unitialized variable or some such.
>

can you try splitting it in 2, ie no pipe?
dump a0f some.file /old (or dump 0f - /old | gzip -c > file.dump.gz)
restore rf some.file

danny

[Mikhail T.]

Danny Braniss написав(ла):

Well, the first part (the dump) runs almost to the completion, but hangs
at the very end for some reason:

dump 0aCf 64 /ibm/ibmo.0.2009-03-24.dump /old
DUMP: WARNING: should use -L when dumping live read-write filesystems!
DUMP: Date of this level 0 dump: Tue Mar 24 05:59:27 2009
DUMP: Date of last level 0 dump: the epoch
DUMP: Dumping /dev/ad2s1e (/ibmo) to /ibm/ibmo.0.2009-03-24.dump
DUMP: mapping (Pass I) [regular files]
DUMP: Cache 64 MB, blocksize = 65536
DUMP: mapping (Pass II) [directories]
DUMP: estimated 152357442 tape blocks.
DUMP: dumping (Pass III) [directories]
DUMP: dumping (Pass IV) [regular files]
DUMP: 0.83% done, finished in 9:59 at Tue Mar 24 16:04:19 2009
DUMP: 2.74% done, finished in 5:55 at Tue Mar 24 12:05:07 2009
DUMP: 4.66% done, finished in 5:06 at Tue Mar 24 11:21:27 2009
DUMP: 6.58% done, finished in 4:43 at Tue Mar 24 11:03:37 2009
...
DUMP: 91.54% done, finished in 0:23 at Tue Mar 24 10:38:15 2009
DUMP: 93.41% done, finished in 0:18 at Tue Mar 24 10:38:02 2009
DUMP: 95.27% done, finished in 0:13 at Tue Mar 24 10:37:50 2009
DUMP: 97.15% done, finished in 0:07 at Tue Mar 24 10:37:36 2009
DUMP: 99.03% done, finished in 0:02 at Tue Mar 24 10:37:23 2009
DUMP: DUMP: 152769349 tape blocks on 1 volume
DUMP: finished in 16706 seconds, throughput 9144 KBytes/sec
[... Hang ...]
load: 0.18 cmd: dump 10105 [sbwait] 72.53u 383.14s 0% 73048k
load: 0.19 cmd: dump 10102 [sbwait] 164.93u 314.87s 0% 75008k
load: 0.10 cmd: dump 10102 [running] 164.93u 314.87s 0% 75008k

The timestamp on the output file is, indeed, 10:38 and the dumping
process is hanging ever since then (over 90 minutes already).

Yours,

-mi

[Mikhail T.]

Danny Braniss написав(ла):

> can you try splitting it in 2, ie no pipe?
> dump a0f some.file /old (or dump 0f - /old | gzip -c > file.dump.gz)
> restore rf some.file
>
>

Same problem:

restore -rf ibmo.0.2009-03-24.dump
load: 0.55 cmd: restore 11303 [nbufkv] 3.53u 3.91s 4% 27980k
unknown tape header type 213474529
abort? [yn]

Please, advise. Thanks! Yours,

-mi

[Andrew Snow]

Mikhail T. wrote:
> dump 0aCf 64 /ibm/ibmo.0.2009-03-24.dump /old
> DUMP: WARNING: should use -L when dumping live read-write filesystems!

I thought you said it was a read-only filesystem?

In my experience, restore can sometimes throw warnings if you dump a
live filesystem. It might be causing your errors? If possible, can you
try completely unmounting the filesystem you are dumping and trying again?

[Yoshihiro Ota]

-a option seems to be the problem.
Try without it.

Hiro

[Mikhail T.]

Andrew Snow написав(ла):

> Mikhail T. wrote:
>> dump 0aCf 64 /ibm/ibmo.0.2009-03-24.dump /old
>> DUMP: WARNING: should use -L when dumping live read-write
>> filesystems!
>
> I thought you said it was a read-only filesystem?

It was yesterday. Today I remounted it rw to remove some junk-files,
which I don't need to transfer. I don't believe, this is causing the
problems.

> In my experience, restore can sometimes throw warnings if you dump a
> live filesystem. It might be causing your errors? If possible, can
> you try completely unmounting the filesystem you are dumping and
> trying again?

I don't think, restore can even figure this out, much less throw a
warning -- it is dump, that complains... But the dump started this
morning is still hanging (in sbwait) -- I've never seen this before. I'm
also very troubled, that such an important functionality (dump/restore!)
is sooo problem-prone, and yet so few people seem to care...

Is the official view, that dump is obsolete (and already bit-rotten),
perhaps, and use of tar is encouraged instead?

-mi

[Daniel O'Connor]

On Wednesday 25 March 2009 11:48:56 Mikhail T. wrote:
> Andrew Snow написав(ла):
> > Mikhail T. wrote:
> >> dump 0aCf 64 /ibm/ibmo.0.2009-03-24.dump /old
> >> DUMP: WARNING: should use -L when dumping live read-write
> >> filesystems!
> >
> > I thought you said it was a read-only filesystem?
>
> It was yesterday. Today I remounted it rw to remove some junk-files,
> which I don't need to transfer. I don't believe, this is causing the
> problems.
>
> > In my experience, restore can sometimes throw warnings if you dump a
> > live filesystem. It might be causing your errors? If possible, can
> > you try completely unmounting the filesystem you are dumping and
> > trying again?
>
> I don't think, restore can even figure this out, much less throw a
> warning -- it is dump, that complains... But the dump started this

restore will emit a warning if dump writes a stream that is out of order
because of a live file system but that is not what you are seeing.

> morning is still hanging (in sbwait) -- I've never seen this before. I'm
> also very troubled, that such an important functionality (dump/restore!)
> is sooo problem-prone, and yet so few people seem to care...

Well, "works for me".

> Is the official view, that dump is obsolete (and already bit-rotten),
> perhaps, and use of tar is encouraged instead?

I've never had dump fail but it IS rather crusty and slow.. That said tar
doesn't cover all the information I believe.

[Mikhail T.]

Daniel O'Connor написав(ла):

>> morning is still hanging (in sbwait) -- I've never seen this before. I'm
>> also very troubled, that such an important functionality (dump/restore!)
>> is sooo problem-prone, and yet so few people seem to care...
>>
>
> Well, "works for me".
>

Well, would like a login on this system to take a look for yourself? I
can reproduce the problem easily.

>> Is the official view, that dump is obsolete (and already bit-rotten),
>> perhaps, and use of tar is encouraged instead?
>>
>
> I've never had dump fail but it IS rather crusty and slow.. That said tar
> doesn't cover all the information I believe.

So, if dump/restore ain't it, does FreeBSD have a supported way of
making filesystem-level backups, that's both modern and covers all
aspects (like flags)?

That said, I point out, that for me, dump is not failing (although it
did hang this morning). It is the restore, which fails to read dump's
output:

unknown tape header type 213474529

abort? [yn] n
resync restore, skipped 502 blocks
expected next file 54, got 0
unknown tape header type -954356454
abort? [yn] n
resync restore, skipped 29 blocks
expected next file 54, got 0
unknown tape header type -1754938223
abort? [yn] n
resync restore, skipped 482 blocks
expected next file 54, got 0
unknown tape header type -915868704
abort? [yn] n
resync restore, skipped 29 blocks
expected next file 54, got 0
unknown tape header type 1790084751
abort? [yn] n
resync restore, skipped 482 blocks
expected next file 54, got 0
unknown tape header type 903667267
abort? [yn] n
...

Thanks!

[Mikhail T.]

Yoshihiro Ota написав(ла):

>> No big difference:
>> dump a0f - /old | restore -rf -
>> [...]
>> DUMP: 17.25% done, finished in 3:27 at Tue Mar 24 05:42:00 2009
>> DUMP: 20.36% done, finished in 3:09 at Tue Mar 24 05:28:13 2009
>> DUMP: 23.83% done, finished in 2:50 at Tue Mar 24 05:14:32 2009
>> unknown tape header type -621260722
>> abort? [yn]
>>
>> Looks like a junk value somewhere... Unitialized variable or some such.
>>
>> -mi
>>
>
> -a option seems to be the problem.
> Try without it.
>

As could be expected, it failed exactly the same without the obviously
unrelated -a option:

root@symbion:/ibm (113) dump 0f - /ibmo | restore -rf -
DUMP: Date of this level 0 dump: Tue Mar 24 21:31:19 2009

DUMP: Date of last level 0 dump: the epoch

DUMP: Dumping /dev/ad2s1e (/ibmo) to standard output

DUMP: mapping (Pass I) [regular files]

DUMP: mapping (Pass II) [directories]
DUMP: estimated 152357442 tape blocks.
DUMP: dumping (Pass III) [directories]
DUMP: dumping (Pass IV) [regular files]

DUMP: 0.16% done, finished in 242:05 at Sat Apr 4 00:00:21 2009
DUMP: 4.29% done, finished in 10:24 at Wed Mar 25 08:24:02 2009
DUMP: 8.56% done, finished in 5:52 at Wed Mar 25 03:56:43 2009
DUMP: 11.76% done, finished in 4:35 at Wed Mar 25 02:43:29 2009
DUMP: 16.00% done, finished in 3:38 at Wed Mar 25 01:52:05 2009
DUMP: 19.28% done, finished in 3:15 at Wed Mar 25 01:33:43 2009
DUMP: 22.74% done, finished in 2:55 at Wed Mar 25 01:18:49 2009
unknown tape header type 1431655765
abort? [yn] n
resync restore, skipped 9 blocks
expected next file 1599492, got 0
DUMP: 24.50% done, finished in 3:27 at Wed Mar 25 02:05:41 2009
unknown tape header type 1508167078
abort? [yn] n
resync restore, skipped 66 blocks
expected next file 1599492, got 0
unknown tape header type -1493630979
abort? [yn] y
dump core? [yn] n
DUMP: Broken pipe
DUMP: The ENTIRE dump is aborted.

Now can one get /real/ support for the most basic functionality of the
most advanced modern Unix in the world? Thanks,

-mi

[Andrew Snow]

Mikhail T. wrote:
> Now can one get /real/ support for the most basic functionality of the
> most advanced modern Unix in the world? Thanks,

I think before this goes any further, you will need to try
rebooting/unmouting it, running fsck on it, and then dump the unmounted
partition and see how that goes.

- Andrew

[Daniel O'Connor]

On Wednesday 25 March 2009 12:38:04 Mikhail T. wrote:
> Daniel O'Connor написав(ла):
> >> morning is still hanging (in sbwait) -- I've never seen this before. I'm
> >> also very troubled, that such an important functionality (dump/restore!)
> >> is sooo problem-prone, and yet so few people seem to care...
> >
> > Well, "works for me".
>
> Well, would like a login on this system to take a look for yourself? I
> can reproduce the problem easily.

I don't know the internals of dump/restore :(

> >> Is the official view, that dump is obsolete (and already bit-rotten),
> >> perhaps, and use of tar is encouraged instead?
> >
> > I've never had dump fail but it IS rather crusty and slow.. That said tar
> > doesn't cover all the information I believe.
>
> So, if dump/restore ain't it, does FreeBSD have a supported way of
> making filesystem-level backups, that's both modern and covers all
> aspects (like flags)?

I would try a pax archive, eg..
tar --format pax --one-file-system -pcf - -C / . | tar -pxf - -C /mnt/newdisk

According to the libarchive-formats page this handles ACLs & flags, my testing
shows it handles flags (at least). eg..

[midget 13:30] /tmp/test2 >touch foo
[midget 13:30] /tmp/test2 >chflags uchg foo
[midget 13:30] /tmp/test2 >tar --format pax -zpcf /tmp/test.pax.gz foo
[midget 13:30] /tmp/test2 >rm -f foo
rm: foo: Operation not permitted
[midget 13:30] /tmp/test2 >chflags nouchg foo
[midget 13:30] /tmp/test2 >rm foo
[midget 13:30] /tmp/test2 >tar -pxf /tmp/test.pax.gz
[midget 13:30] /tmp/test2 >ls -lao
total 30
drwxr-xr-x 2 darius wheel - 512 Mar 25 13:30 .
drwxrwxrwt 53 root wheel - 28672 Mar 25 13:29 ..
-rw-r--r-- 1 darius wheel uchg 0 Mar 25 13:30 foo
[midget 13:30] /tmp/test2 >rm -f foo
rm: foo: Operation not permitted

> That said, I point out, that for me, dump is not failing (although it
> did hang this morning). It is the restore, which fails to read dump's
> output:

You can't tell the difference between dump producing mangled output or restore
bombing out on valid input..

[Mikhail T.]

Andrew Snow написав(ла):

> Mikhail T. wrote:
>
>> Now can one get /real/ support for the most basic functionality of the
>> most advanced modern Unix in the world? Thanks,
>>
>
> I think before this goes any further, you will need to try
> rebooting/unmouting it, running fsck on it, and then dump the unmounted
> partition and see how that goes.
>

The system's uptime is only 3 days -- I had to reboot it to put in the
new disk...

But I will try your suggestion next, after the current attempt (using
restore's -v switch) ends. If it chokes on the same file every time,
that would be a clue... Thanks,

-mi

[Mikhail T.]

Daniel O'Connor написав(ла):

>> That said, I point out, that for me, dump is not failing (although it
>> did hang this morning). It is the restore, which fails to read dump's
>> output:
>>
>
> You can't tell the difference between dump producing mangled output or restore
> bombing out on valid input..
>

That's true. I just wanted to point out, that someone running dump only
(to make backups) is not going to know, whether his dumps are usable
(for whichever of the two reasons), until he needs them...

[Daniel O'Connor]

On Wednesday 25 March 2009 13:10:23 Mikhail T. wrote:
> Now can one get /real/ support for the most basic functionality of the
> most advanced modern Unix in the world? Thanks,

Maybe you should return it to the shop and ask for your money back.

[Mikhail T.]

Daniel O'Connor написав(ла):

> On Wednesday 25 March 2009 13:10:23 Mikhail T. wrote:
>
>> Now can one get /real/ support for the most basic functionality of the
>> most advanced modern Unix in the world? Thanks,
>>
>
> Maybe you should return it to the shop and ask for your money back.
>

Well, if this response is the best one can get, may be I should also
revoke my own 15 years worth of contributions to the project...

Except, why would I? I always supported people, who had problems with
any of my work -- and the attitude of the rest of the contributors /used
to be/ the same...

[Daniel O'Connor]

On Wednesday 25 March 2009 14:35:01 Mikhail T. wrote:
> Daniel O'Connor написав(ла):
> > On Wednesday 25 March 2009 13:10:23 Mikhail T. wrote:
> >> Now can one get /real/ support for the most basic functionality of the
> >> most advanced modern Unix in the world? Thanks,
> >
> > Maybe you should return it to the shop and ask for your money back.
>
> Well, if this response is the best one can get, may be I should also
> revoke my own 15 years worth of contributions to the project...
>
> Except, why would I? I always supported people, who had problems with
> any of my work -- and the attitude of the rest of the contributors /used
> to be/ the same...

People ARE helping you, just because they haven't come up with an answer is no
reason to send snarky comments to the list.

Outrage (fake or otherwise) that people don't seem to be taking your
particular problem seriously is unhelpful and probably counter-productive.

[Mikhail T.]

Daniel O'Connor написав(ла):

> People ARE helping you, just because they haven't come up with an answer is no
> reason to send snarky comments to the list.
>

No, sorry, people aren't. They are trying, yes, but not even close. The
suggestion to eliminate the -a switch (a no-op, in fact) was
particularly unhelpful -- and deserving of mockery. Later on someone
with a similar problem will find this thread with a search engine and
will be trying to follow the posted advices -- to no avail, of course,
plunging FreeBSD further into disrepute.

> Outrage (fake or otherwise) that people don't seem to be taking your
> particular problem seriously is unhelpful and probably counter-productive.

It is fairly obvious by now, that no real help will be forthcoming, for
whatever reason. Thus any talk of "productivity" is moot. Thanks for trying.

[Mikhail T.]

Andrew Snow написав(ла):

> I think before this goes any further, you will need to try
> rebooting/unmouting it, running fsck on it, and then dump the unmounted
> partition and see how that goes.
>
>

Rebooted, reran `fsck -y /old' (all clean). Same problem...

-mi

[Mark Kirkwood]

Is this link any help?

http://www.mail-archive.com/freebsd-...@freebsd.org/msg197899.html

Other than that, I'd suggest checking the disk(s) with smartmontools to
try to rule out hardware problems.

regards

Mark

[Greg Black]

On 2009-03-24, Mikhail T. wrote:
> That's true. I just wanted to point out, that someone running dump only
> (to make backups) is not going to know, whether his dumps are usable
> (for whichever of the two reasons), until he needs them...

Such a person is not making backups and deserves what he gets.

I haven't got anything to say about dump/restore because I haven't
bothered with them for years. I do know that dumps from mounted file
systems will often appear to work, but will fail when it matters. This
is not a bug and is expected behaviour to which the solution is obvious.

[Peter Jeremy]

On 2009-Mar-25 00:29:36 -0400, "Mikhail T." <mi+...@aldan.algebra.com> wrote:
>Daniel O'Connor ???????(??):

>> People ARE helping you, just because they haven't come up with an answer is no
>> reason to send snarky comments to the list.
>>
>No, sorry, people aren't. They are trying, yes, but not even close.

People are _trying_ to help you. The problem is that there's no real
leads on where the problem might be so people are clutching at straws.
It's virtually impossible to debug that sort of problem remotely and
AFAIK no-one has been able to reproduce it. I've been using dump for
decades on a variety of different systems and don't recall ever seeing
the problem you are having.

Are you restoring to an empty (freshly newfs'd) filesystem? The '-r'
option to restore is only supposed to be used this way (though I've
ignored this in the past).

Are there any snapshots in the source filesystem?

Are you able to reproduce the problem using a different source filesystem?
(preferably something small)

Can you rule out hardware problems?

Have you tried moving the source disk to another computer and seeing
if it will dump/restore there?

Do you have enough spare space to be able to dd the problematic FS
somewhere else for experimenting? If so, does the problem go away
if you empty all the files (so all the files are there but all are
zero-length)? Or if you replace all the file contents with NULs?

>It is fairly obvious by now, that no real help will be forthcoming,
>for whatever reason.

Throwing a hissy-fit won't help.

--
Peter Jeremy

[Daniel O'Connor]

On Wednesday 25 March 2009 14:59:36 Mikhail T. wrote:
> Daniel O'Connor написав(ла):
> > People ARE helping you, just because they haven't come up with an answer
> > is no reason to send snarky comments to the list.
>
> No, sorry, people aren't. They are trying, yes, but not even close. The
> suggestion to eliminate the -a switch (a no-op, in fact) was
> particularly unhelpful -- and deserving of mockery. Later on someone
> with a similar problem will find this thread with a search engine and
> will be trying to follow the posted advices -- to no avail, of course,
> plunging FreeBSD further into disrepute.

I don't see how whining about it's going to change it. Insulting people for
having a helpful attitude (even if it didn't solve your problem) is not going
to reward them for their time and effort.

> > Outrage (fake or otherwise) that people don't seem to be taking your
> > particular problem seriously is unhelpful and probably
> > counter-productive.
>
> It is fairly obvious by now, that no real help will be forthcoming, for
> whatever reason. Thus any talk of "productivity" is moot. Thanks for
> trying.

Except that I offered you a way of transferring your files that would preserve
file flags and so on.

Yes, dump is broken for you, deal with it. It is quite possible your FS is
corrupt, and/or your disk is damaged.

[Daniel O'Connor]

On Wednesday 25 March 2009 18:37:17 Bartosz Stec wrote:
> > Yes, dump is broken for you, deal with it. It is quite possible your FS
> > is corrupt, and/or your disk is damaged.
>

> ..and/or it is some other hardware problem, maybe you also should test
> your memory with memtest or something similiar? I'm using dump/restore
> very frequently and I had never seen such problem. Neither on -RELAESE,
> -STABLE, nor -CURRENT.
> So I think you should make sure that your problem is not
> hardware/filesystem dependent before you point dump/restore as a couse
> of the problem. Peter Jeremy already gives you good hints to do that.

One other thing would be to make absolutely sure that your version of dump &
restore are in sync, the are very machine/version dependent.

[Sean]

On 25/03/2009, at 7:55 PM, Daniel O'Connor wrote:

> On Wednesday 25 March 2009 18:37:17 Bartosz Stec wrote:
>>> Yes, dump is broken for you, deal with it. It is quite possible
>>> your FS
>>> is corrupt, and/or your disk is damaged.
>>
>> ..and/or it is some other hardware problem, maybe you also should
>> test
>> your memory with memtest or something similiar? I'm using dump/
>> restore
>> very frequently and I had never seen such problem. Neither on -
>> RELAESE,
>> -STABLE, nor -CURRENT.
>> So I think you should make sure that your problem is not
>> hardware/filesystem dependent before you point dump/restore as a
>> couse
>> of the problem. Peter Jeremy already gives you good hints to do that.
>
> One other thing would be to make absolutely sure that your version
> of dump &
> restore are in sync, the are very machine/version dependent.
>

And the system is compiled without strange CFLAGS in /etc/make.conf

Great way to cause inexplicable problems because an unsafe
optimisation ran rampant, which is really noticeable when it's two
programs that have to be in sync.

> --
> Daniel O'Connor software and network engineer
> for Genesis Software - http://www.gsoft.com.au
> "The nice thing about standards is that there
> are so many of them to choose from."
> -- Andrew Tanenbaum
> GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C
>

_______________________________________________

[Jack L. Stone]

At 07:25 PM 3.25.2009 +1030, Daniel O'Connor wrote:
>On Wednesday 25 March 2009 18:37:17 Bartosz Stec wrote:

>> > Yes, dump is broken for you, deal with it. It is quite possible your FS
>> > is corrupt, and/or your disk is damaged.
>>

>> ..and/or it is some other hardware problem, maybe you also should test
>> your memory with memtest or something similiar? I'm using dump/restore
>> very frequently and I had never seen such problem. Neither on -RELAESE,
>> -STABLE, nor -CURRENT.
>> So I think you should make sure that your problem is not
>> hardware/filesystem dependent before you point dump/restore as a couse
>> of the problem. Peter Jeremy already gives you good hints to do that.
>
>One other thing would be to make absolutely sure that your version of dump &
>restore are in sync, the are very machine/version dependent.
>

>--

I've been watching this thread with some interest since we've had some
similar problems with dump/restore which we use every morning via cron
scripts on a number of servers to produce bootable clones as part of our
backup program. Have been doing this for years and also never saw a problem
as most of you say. We prefer dump/restore for backups.

However, last month upon upon upgrading those servers from FBSD-6.3px
(RELEASE) to 7.0px (RELEASE) we found that about one-half of the servers
had a similar problem as the original poster while the other half did not.
All of the servers (rackmounts) use the same (type) hardware. We spent many
hours trying to solve the problem with those that failed to dump/restore.
Also, searched for any others with the problem and only found a very few,
but without solutions to this issue. (Indeed, the only one was a reference
to any efforts to restore an older OS version which didn't apply here).

And, indeed we tried everything suggested here to fix the proble without
success. Sometimes the problem was dump which would reach 99% and never
finish -- it would stick there and would overlap with another cron start
the next day, and the next day, and the next day. (The servers that did
work fooled us and we found out about this issue on the others when the
overlaps appeared and drew our attention). That's when our work to try and
solve the issues started and went on for days.

Our script that has always worked contained this (after scraping and making
fresh FS):
/sbin/dump -D /root/dumpdates -0auL -f - / | /sbin/restore -rf -

Indeed, the first thing we did was to remove the pipe and tried to restore
from a file. However, because the dumps would not go past the 99%, no file
to restore from! There were some exceptions when the dump would complete,
but was not reliable. When these reached the restore level, restore would
go crazy with errors.

SOLUTION
The "clones" are a very important pasrt of our backup program. Since the
dump side of the problems simply stuck and provided no error message at all
and the errors from any restores were not useful, our only solution was to
revert back to FBSD-6.3 on those servers with this issue and dump/restore
went back to working again. We left those that were working on FBSD-7.0-R
and they continue to work okay.

We could only conclude that the problem was perhaps something with
hardeware, perhaps the way memory was handled in 7.0, but that is only a
guess.

Once again, every suggestion on this thread was tried during our long
efforts to fix the issue. Perhaps there is yet another suggestion? In the
meantime, we've decided to wait for 7.2R (7.1 did not fix the problems
either).

/Jack

(^_^)
Happy trails,
Jack L. Stone

System Admin
Sage-american

[Peggy Wilkins]

>>>>> Jack L Stone <ja...@sage-american.com> writes:

>> I've been watching this thread with some interest since we've had some
>> similar problems with dump/restore which we use every morning via cron
>> scripts on a number of servers to produce bootable clones as part of our
>> backup program. Have been doing this for years and also never saw a problem
>> as most of you say. We prefer dump/restore for backups.

>> However, last month upon upon upgrading those servers from FBSD-6.3px
>> (RELEASE) to 7.0px (RELEASE) we found that about one-half of the servers
>> had a similar problem as the original poster while the other half did not.
>> All of the servers (rackmounts) use the same (type) hardware. We spent many
>> hours trying to solve the problem with those that failed to dump/restore.
>> Also, searched for any others with the problem and only found a very few,
>> but without solutions to this issue. (Indeed, the only one was a reference
>> to any efforts to restore an older OS version which didn't apply here).

[snip]

>> SOLUTION
>> The "clones" are a very important pasrt of our backup program. Since the
>> dump side of the problems simply stuck and provided no error message at all
>> and the errors from any restores were not useful, our only solution was to
>> revert back to FBSD-6.3 on those servers with this issue and dump/restore
>> went back to working again. We left those that were working on FBSD-7.0-R
>> and they continue to work okay.

I was seeing this same problem on all my 64-bit systems: FreeBSD-7
dump would hang at a random point. Dump continues to work flawlessly
for me on FreeBSD-7/i386.

I ran across this which includes a patch:

http://www.freebsd.org/cgi/query-pr.cgi?pr=121684

The kernel patch linked to there solved the problem for me, but I am
running many production systems and am unwilling to apply this patch
to -RELEASE every time there is a kernel update (I just use the
standard GENERIC kernel which I get via freebsd-update). I now live
without dump on amd64. Apparently this fix is waiting on some related
issue; and I will be very happy when it makes it to the officially
released kernel.

plw

[Mikhail T.]

Greg Black написав(ла):

> On 2009-03-24, Mikhail T. wrote:
>
>> That's true. I just wanted to point out, that someone running dump only
>> (to make backups) is not going to know, whether his dumps are usable
>> (for whichever of the two reasons), until he needs them...
>>
>
> Such a person is not making backups and deserves what he gets.
>

But he *is* making backups -- kindly re-read my paragraph above... He
just is not routinely using them and thus does not know, that they
aren't usable... It is not unreasonable to expect the two utilities to
"just work", so I wouldn't be blaming such a person for not testing
restore.

That such a scenario is possible, despite the user making diligent
regular backups, is a very bad sign...

> I haven't got anything to say about dump/restore because I haven't
> bothered with them for years. I do know that dumps from mounted file
> systems will often appear to work, but will fail when it matters. This
> is not a bug and is expected behaviour to which the solution is obvious.
>

FS being mounted read-only should not be a problem. And even read-write
mounted filesystems will be Ok, although possibly inconsistent (a file
modified during backup may get to into dump "as of" any point). But an
idle FS is Ok to dump. Generally, when dumping read-write mounted
filesystems, one is supposed to use snapshots, but that is very buggy on
its own (leading to kernel crashes), so it is safer to rely on the FS
being "idle", if it can not be forced into read-only for some other reason.

-mi

[Dimitry Andric]

On 2009-03-24 07:30, Mikhail T. wrote:
> dump a0f - /old | restore -rf -
> [...]
> DUMP: 17.25% done, finished in 3:27 at Tue Mar 24 05:42:00 2009
> DUMP: 20.36% done, finished in 3:09 at Tue Mar 24 05:28:13 2009
> DUMP: 23.83% done, finished in 2:50 at Tue Mar 24 05:14:32 2009
> unknown tape header type -621260722
> abort? [yn]
>
> Looks like a junk value somewhere... Unitialized variable or some such.

Hmm, I can't reproduce this at all; I use dump and restore quite
regularly in this way, and I have never encountered this issue (except
for the occasional 'expected file NNN, got file MMM', which is usually
harmless).

Maybe the dump output gets corrupted in some way? (E.g. faulty RAM, or
disk?) If you are dumping a live filesystem, could it possibly help to
add the -L option?

[cpghost]

On Wed, Mar 25, 2009 at 12:04:18PM -0400, Mikhail T. wrote:
> Generally, when dumping read-write mounted filesystems, one is
> supposed to use snapshots, but that is very buggy on its own
> (leading to kernel crashes), so it is safer to rely on the FS being
> "idle", if it can not be forced into read-only for some other
> reason.

Perhaps your kernel/world is not recent enough? There used to be
problems with snapshots, including hangs or crashes, but IIRC, they
disappeared completly a couple of months ago, at least for me. (no,
sorry, I can't pinpoint the exact date, and much less the exact
revision nr).

Snapshots alone, and dump -L + restore (on RELENG_7/amd64) working
flawlessly here on 20+ rack-mounted servers; some to tape, some to
SAN, but I'm probably just lucky.

BTW, that's what makes this problem so intractable: those who could
debug it, can't reproduce it on their machines. It's scary to know
there's a bug lurking in there.

> -mi

-cpghost.

--
Cordula's Web. http://www.cordula.ws/

[Scott Lambert]

If you happen to have a different version of FreeBSD on another box of
boxes, older or newer, you might try a test restore of your problem dump
file on those boxes.

If you can make a dump file which shows this bug on being fed to
restore, which doesn't have sensitive information in it, I would be
happy to download it and see if I can reproduce the problem on a few
boxes with different versions of FreeBSD here. I had some restore
issues in May of 2008 which got resolved because I could create a dump
file which would fail to restore on a developer's system. But that was
a different issue than you are seeing.

--
Scott Lambert KC5MLE Unix SysAdmin
lam...@lambertfam.org

[Peter Jeremy]

On 2009-Mar-25 19:25:28 +1030, Daniel O'Connor <doco...@gsoft.com.au> wrote:
>One other thing would be to make absolutely sure that your version of dump &
>restore are in sync, the are very machine/version dependent.

Actually, they aren't - the archive format is very stable. (This is a
fairly important requirement - you don't want to suddenly be unable to
restore your backups after an upgrade). I can restore dumps made on
FreeBSD4.8/i386 and FreeBSD4.9/alpha on a FreeBSD-current/amd64 system
without problems.

It's possible that you might have problems with a backup made using a
very recent dump on a very old restore if you've used filesystem
features that didn't exist when that restore was built.

--
Peter Jeremy

[Peter Jeremy]

On 2009-Mar-25 07:21:37 -0500, "Jack L. Stone" <ja...@sage-american.com> wrote:
>However, last month upon upon upgrading those servers from FBSD-6.3px
>(RELEASE) to 7.0px (RELEASE) we found that about one-half of the servers
>had a similar problem as the original poster while the other half did not.

..

>the next day, and the next day, and the next day. (The servers that did
>work fooled us and we found out about this issue on the others when the
>overlaps appeared and drew our attention). That's when our work to try and
>solve the issues started and went on for days.

It's not clear to me whether all your servers have the problem and you
only initially noticed it on some of them or some of your servers work
and others dont. In the latter case, you are probably in a very good
position to identify the problem since it is related to some difference
between your servers.

>We could only conclude that the problem was perhaps something with
>hardeware, perhaps the way memory was handled in 7.0, but that is only a
>guess.

If you are talking about server-grade hardware (ECC RAM etc) then it's
unlikely to be RAM corruption. About the only thing I can think of
would be that if you have RAM above 4GB, you might be running foul of
an address being truncated somewhere (particularly in a device
driver). (The amd64 user memory map changed between 7.x and 8.x but I
don't think there was any change between 6.x & 7.x).

The pre-emption changes in 7.x and/or moving to an SMP host would seem
to increase the probability of hitting the problem fixed in the patch
mentioned later (kern/121684).

--
Peter Jeremy

[Daniel O'Connor]

On Thursday 26 March 2009 19:55:04 Peter Jeremy wrote:
> On 2009-Mar-25 19:25:28 +1030, Daniel O'Connor <doco...@gsoft.com.au>
wrote:
> >One other thing would be to make absolutely sure that your version of dump
> > & restore are in sync, the are very machine/version dependent.
>
> Actually, they aren't - the archive format is very stable. (This is a
> fairly important requirement - you don't want to suddenly be unable to
> restore your backups after an upgrade). I can restore dumps made on
> FreeBSD4.8/i386 and FreeBSD4.9/alpha on a FreeBSD-current/amd64 system
> without problems.

Hmm interesting.. I must confess I haven't had problems recently but I tend to
use tar these days anyway since it's more portable.

> It's possible that you might have problems with a backup made using a
> very recent dump on a very old restore if you've used filesystem
> features that didn't exist when that restore was built.

Could be, too long ago to remember :)

[Jack L. Stone]

At 09:45 AM 3.25.2009 -0500, Peggy Wilkins wrote:
>>>>>> Jack L Stone <ja...@sage-american.com> writes:
>

> >> I've been watching this thread with some interest since we've had some
> >> similar problems with dump/restore which we use every morning via cron
> >> scripts on a number of servers to produce bootable clones as part
of our
> >> backup program. Have been doing this for years and also never saw a
problem
> >> as most of you say. We prefer dump/restore for backups.
>
> >> However, last month upon upon upgrading those servers from FBSD-6.3px
> >> (RELEASE) to 7.0px (RELEASE) we found that about one-half of the
servers
> >> had a similar problem as the original poster while the other half
did not.
> >> All of the servers (rackmounts) use the same (type) hardware. We
spent many
> >> hours trying to solve the problem with those that failed to
dump/restore.
> >> Also, searched for any others with the problem and only found a
very few,
> >> but without solutions to this issue. (Indeed, the only one was a
reference
> >> to any efforts to restore an older OS version which didn't apply
here).

> [snip]

> >> SOLUTION
> >> The "clones" are a very important pasrt of our backup program.
Since the
> >> dump side of the problems simply stuck and provided no error
message at all
> >> and the errors from any restores were not useful, our only solution
was to
> >> revert back to FBSD-6.3 on those servers with this issue and
dump/restore
> >> went back to working again. We left those that were working on
FBSD-7.0-R
> >> and they continue to work okay.
>

>I was seeing this same problem on all my 64-bit systems: FreeBSD-7
>dump would hang at a random point. Dump continues to work flawlessly
>for me on FreeBSD-7/i386.
>
>I ran across this which includes a patch:
>
>http://www.freebsd.org/cgi/query-pr.cgi?pr=121684
>
>The kernel patch linked to there solved the problem for me, but I am
>running many production systems and am unwilling to apply this patch
>to -RELEASE every time there is a kernel update (I just use the
>standard GENERIC kernel which I get via freebsd-update). I now live
>without dump on amd64. Apparently this fix is waiting on some related
>issue; and I will be very happy when it makes it to the officially
>released kernel.
>
>plw
>

Thanks for the reply. Forgot to mention, our machines are all i386 with the
problem -- so are the ones without the problem.

Yes, I found that patch too and tried it on one of the servers -- no joy.

Guess we'll continue to wait also for now. Maybe 7.2/i386....or, until
someone finds the solution since we're out of ideas and stuck with 6.3 in
order to use dump that we have trusted.

[Mike Tancsa]

At 07:53 AM 3/26/2009, Jack L. Stone wrote:
> > >> SOLUTION
> > >> The "clones" are a very important pasrt of our backup program.
>Since the
> > >> dump side of the problems simply stuck and provided no error
>message at all
> > >> and the errors from any restores were not useful, our only solution
>was to
> > >> revert back to FBSD-6.3 on those servers with this issue and
>dump/restore
> > >> went back to working again. We left those that were working on
>FBSD-7.0-R
> > >> and they continue to work okay.
> >
>

>Thanks for the reply. Forgot to mention, our machines are all i386 with the
>problem -- so are the ones without the problem.
>
>Yes, I found that patch too and tried it on one of the servers -- no joy.
>
>Guess we'll continue to wait also for now. Maybe 7.2/i386....or, until
>someone finds the solution since we're out of ideas and stuck with 6.3 in
>order to use dump that we have trusted.

Hi,
I didnt see it in the thread clearly, but did you try
creating a dump without the -L... i.e. without snapshots enabled ?

---Mike

[Christian Walther]

Hi,

I followed this thread out of interest since I do not suffer from this error.
But I wonder if truss could shed some light into this issue. If for
example a dump hangs at 99%, it might be an idea to set up truss to
trace the dump process. Yes, this will produce lots of output, but
maybe it gives a hint as soon as dump hangs.

Christian Walther

[Jack L. Stone]

At 07:57 AM 3.26.2009 -0400, Mike Tancsa wrote:
>>
>>Thanks for the reply. Forgot to mention, our machines are all i386 with the
>>problem -- so are the ones without the problem.
>>
>>Yes, I found that patch too and tried it on one of the servers -- no joy.
>>
>>Guess we'll continue to wait also for now. Maybe 7.2/i386....or, until
>>someone finds the solution since we're out of ideas and stuck with 6.3 in
>>order to use dump that we have trusted.
>
>

>Hi,
> I didnt see it in the thread clearly, but did you try
>creating a dump without the -L... i.e. without snapshots enabled ?
>
> ---Mike
>

Yes, but it's for running a dump on a (L)ive FS and just spits out warnings
to that effect and has no effect on solving the problem(s).

[Mike Tancsa]

At 08:08 AM 3/26/2009, Jack L. Stone wrote:

>Yes, but it's for running a dump on a (L)ive FS and just spits out warnings
>to that effect and has no effect on solving the problem(s).

Unless the filesystem is very busy, you will get your data backed up.
If you have things like databases, I still would not trust
snapshots. Better to use pg_dump or mysqldump or the app that comes
with whatever DB you are using... When backing up things like / and
/usr, I would hazard a guess that most things are not changing while
the backup is running, at least they dont in my environments. I have
never had a problem with things like /home and even /var or /mail
which are changing quite a bit. We dont restore much in the course
of our daily routine, but we have always been able to restore
people's Maildir when they accidentally have deleted stuff and it all
worked without issue over the years.

---Mike

[Peter Schuller]

> >Yes, but it's for running a dump on a (L)ive FS and just spits out warnings
> >to that effect and has no effect on solving the problem(s).
>
> Unless the filesystem is very busy, you will get your data backed up.
> If you have things like databases, I still would not trust
> snapshots.

Uh. If backuping up a live database from a snapshot is not
trustworthy, either the snapshot facility is broken or the database is
broken (i.e., not crash-safe to begin with).

That said there are plenty of other reasos to use proper dump tools
(data portability, confirming the ability to actually read all rows
from a table, using a more often exercised code path and perhaps less
likely to have edge case bugs, etc).

--
/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <peter.s...@infidyne.com>'
Key retrieval: Send an E-Mail to getp...@scode.org
E-Mail: peter.s...@infidyne.com Web: http://www.scode.org

[Mike Tancsa]

At 10:01 AM 3/26/2009, Peter Schuller wrote:
> >
> > Unless the filesystem is very busy, you will get your data backed up.
> > If you have things like databases, I still would not trust
> > snapshots.
>
>Uh. If backuping up a live database from a snapshot is not
>trustworthy, either the snapshot facility is broken or the database is
>broken (i.e., not crash-safe to begin with).

.. or the database is configured in a risky way... But judging by
this thread, it seems dump with -L is indeed broken for some
people. Hence, I suggested dumping the database using the database's
backup tools and trying dump without -L

[Karl Denninger]

Peter Schuller wrote:
>>> Yes, but it's for running a dump on a (L)ive FS and just spits out warnings
>>> to that effect and has no effect on solving the problem(s).
>>>
>> Unless the filesystem is very busy, you will get your data backed up.
>> If you have things like databases, I still would not trust
>> snapshots.
>>
>
> Uh. If backuping up a live database from a snapshot is not
> trustworthy, either the snapshot facility is broken or the database is
> broken (i.e., not crash-safe to begin with).
>
> That said there are plenty of other reasos to use proper dump tools
> (data portability, confirming the ability to actually read all rows
> from a table, using a more often exercised code path and perhaps less
> likely to have edge case bugs, etc).
>

The issue with backing up a database live comes in when the filesystem
where the database and transaction log files are DIFFERS. You can get
into a pathological case in that instance.

If the transaction log and database itself are both on the same
snapshotted entity (that is, the snapshot is pulled at the same instant
in time for both) what you get BETTER be restorable or your database's
transaction log facility doesn't really do what it promises to do!

--
--
Karl Denninger
ka...@denninger.net

[Peter Schuller]

> ... or the database is configured in a risky way... But judging by

> this thread, it seems dump with -L is indeed broken for some
> people. Hence, I suggested dumping the database using the database's
> backup tools and trying dump without -L

Well, if dump -L is really broken I'd recommend just not using dump
unless you absolutely have to. That is assuming snapshots are not
broken so that you can still use mksnap_ffs with your favorite backup
tool.

[Peter Schuller]

> The issue with backing up a database live comes in when the filesystem
> where the database and transaction log files are DIFFERS. You can get
> into a pathological case in that instance.
>
> If the transaction log and database itself are both on the same
> snapshotted entity (that is, the snapshot is pulled at the same instant
> in time for both) what you get BETTER be restorable or your database's
> transaction log facility doesn't really do what it promises to do!

Absolutely. Doing things like snapshot based backups of databases
assumes you know what you're doing since it is not something which is
documented as an official procedure in your typical database
administrator guide.

Personally, while I would use such schemes, I would always use a plain
fully supported regular dump as a fallback position. I would only rely
on snapshot based processes to do fancy stuff (such as near-realtime
hot standby with zfs snaps + serialized incrementals).

[Karl Denninger]

Peter Schuller wrote:
>> The issue with backing up a database live comes in when the filesystem
>> where the database and transaction log files are DIFFERS. You can get
>> into a pathological case in that instance.
>>
>> If the transaction log and database itself are both on the same
>> snapshotted entity (that is, the snapshot is pulled at the same instant
>> in time for both) what you get BETTER be restorable or your database's
>> transaction log facility doesn't really do what it promises to do!
>>
>
> Absolutely. Doing things like snapshot based backups of databases
> assumes you know what you're doing since it is not something which is
> documented as an official procedure in your typical database
> administrator guide.
>
> Personally, while I would use such schemes, I would always use a plain
> fully supported regular dump as a fallback position. I would only rely
> on snapshot based processes to do fancy stuff (such as near-realtime
> hot standby with zfs snaps + serialized incrementals).
>

To add to this what SHOULD (ha!) work is to dump the database partition
FIRST and THEN dump the Transaction Log partition.

If you do it in the other order you WILL get screwed, as you will have
transactions committed in the database that are not in the XLOG. That
is essentially guaranteed to blow up in your face.

As always any backup scheme has to be TESTED so you can prove to your
own satisfaction that it is RESTORABLE. I can't tell you how many
business clients I have run into (and not only on Unix machines) that
have wind up with lots of backups and NONE of them can be restored -
because they never TESTED their backup strategy.

[Peter Schuller]

> To add to this what SHOULD (ha!) work is to dump the database partition
> FIRST and THEN dump the Transaction Log partition.

Depending on how the database works; specifically when old data in the
log may or may not be expunged. You could do this with PostgreSQL by
using it's PITR support for example. But to the extent that you have a
log which is only supposed to be used for internal reasons (such as
with pg by default), you'd likely be in trouble anyway unless you had
a specific reason to know that it is safe.

> As always any backup scheme has to be TESTED so you can prove to your
> own satisfaction that it is RESTORABLE. I can't tell you how many
> business clients I have run into (and not only on Unix machines) that
> have wind up with lots of backups and NONE of them can be restored -
> because they never TESTED their backup strategy.

Very true, but it is equally dangerous to rely on testing *only*; a
backup system can be very very broken yet appear to work during
testing, either because backups only break sometimes or because they
break in ways that do not obviously and immediately blow up in your
face.

--

[Karl Denninger]

Peter Schuller wrote:
>> To add to this what SHOULD (ha!) work is to dump the database partition
>> FIRST and THEN dump the Transaction Log partition.
>>
>
> Depending on how the database works; specifically when old data in the
> log may or may not be expunged. You could do this with PostgreSQL by
> using it's PITR support for example. But to the extent that you have a
> log which is only supposed to be used for internal reasons (such as
> with pg by default), you'd likely be in trouble anyway unless you had
> a specific reason to know that it is safe.
>

True. If the log rolls on you while you're taking the dumps you're DEAD.

Both design and verification are important :)

[Jake Scott]

Hi..

On Thu, 26 Mar 2009, Peter Schuller wrote:

>>> Yes, but it's for running a dump on a (L)ive FS and just spits out warnings
>>> to that effect and has no effect on solving the problem(s).
>>
>> Unless the filesystem is very busy, you will get your data backed up.
>> If you have things like databases, I still would not trust
>> snapshots.
>
> Uh. If backuping up a live database from a snapshot is not
> trustworthy, either the snapshot facility is broken or the database is
> broken (i.e., not crash-safe to begin with).

Exactly right - if you backup a database by relying on storage snapshots
then the best you will end up with is a database that needs to be
recovered when you restore those volumes (crash consistent). That's not a
good place to be in when your production DB has just blown up.

In addition, there are all sorts of complications which mean that you
might need to freeze IO on multiple volumes simultaneously in order for
this to have a chance of being successful (maintaining
write-order-fidelity).

I would strongly discourage anyone from using this method of backup for
anything that is considered production, thought it might do you for making
QA clones of a running database.

> That said there are plenty of other reasos to use proper dump tools
> (data portability, confirming the ability to actually read all rows
> from a table, using a more often exercised code path and perhaps less
> likely to have edge case bugs, etc).

Absolutely. You really must use a tool that interacts with the database
to perform the backup. Most commercial DBs have hooks that allow the
backup routines to call out to custom snapshot facilities. One would
usually request a backup through the database, which would then freeze IO
to its data files and maybe log files, deal with flushing caches etc and
then call your snapshot routine. I'm not aware that MySQL and Postgres do
though so the best you can do is a dump.

Jake

[Peter C. Lai]

On 2009-03-26 02:45:45PM +0000, Jake Scott wrote:
> Absolutely. You really must use a tool that interacts with the database to
> perform the backup. Most commercial DBs have hooks that allow the backup
> routines to call out to custom snapshot facilities. One would usually
> request a backup through the database, which would then freeze IO to its
> data files and maybe log files, deal with flushing caches etc and then call
> your snapshot routine. I'm not aware that MySQL and Postgres do though so
> the best you can do is a dump.

With MySQL at least, you can (ab)use the replication facilities so that
you can set up a "slave" and do the fs-level dump while the slave is in a
"frozen" state - the last time I played with MySQL, you could basically
desync your slave for a period of time (basically until transaction logs
are purged on the master), during which the slave will be consistent; do
the fs-level backup then kick the master to sync with the slave again.

--
===========================================================
Peter C. Lai | Bard College at Simon's Rock
Systems Administrator | 84 Alford Rd.
Information Technology Svcs. | Gt. Barrington, MA 01230 USA
peter AT simons-rock.edu | (413) 528-7428
===========================================================

[Peter Schuller]

> Exactly right - if you backup a database by relying on storage snapshots
> then the best you will end up with is a database that needs to be
> recovered when you restore those volumes (crash consistent). That's not a
> good place to be in when your production DB has just blown up.

If you have a production db that you care about, your database better
not have trouble recoverying from a crash consistent state. But again
I'm not suggesting that snapshot based backups be the primary method
of backing up.

In terms of time-to-recovery, having a crash consistent DB can be a
lot quicker to recover than grabbing a dump, whose restoration will
tend to be a lot slower than copying files.

> Absolutely. You really must use a tool that interacts with the database
> to perform the backup. Most commercial DBs have hooks that allow the
> backup routines to call out to custom snapshot facilities. One would
> usually request a backup through the database, which would then freeze IO
> to its data files and maybe log files, deal with flushing caches etc and
> then call your snapshot routine. I'm not aware that MySQL and Postgres do
> though so the best you can do is a dump.

I do not think "really must" is appropriate since clearly you can
recover without DB specific integration. There may be reasons why it's
better to have DB specific integration though (for example, limiting
the amount of log reply that will be needed at recovery). The
implication above that you cannot use snapshot based mechanisms with
PostgreSQL and MySQL is not true; it's just that if you do you have to
know what you're doing.

[Karl Denninger]

Jake Scott wrote:
>
>> That said there are plenty of other reasos to use proper dump tools
>> (data portability, confirming the ability to actually read all rows
>> from a table, using a more often exercised code path and perhaps less
>> likely to have edge case bugs, etc).
>
> Absolutely. You really must use a tool that interacts with the
> database to perform the backup. Most commercial DBs have hooks that
> allow the backup routines to call out to custom snapshot facilities.
> One would usually request a backup through the database, which would
> then freeze IO to its data files and maybe log files, deal with
> flushing caches etc and then call your snapshot routine. I'm not
> aware that MySQL and Postgres do though so the best you can do is a dump.
>
> Jake

VERY careful thought has to go into backup strategy with production
databases.

Hooks that call out and snapshot are not necessarily good enough
although they're "necessary" to get a dump that restores without the
database going into log-replay mode.

It is not difficult to do this with Postgresql; you can quiesce the
database, snapshot and then release it, then dump the snapshots. This
gives you transaction-complete dumps (as opposed to "crashed and rolled
forward" dumps). The latter ("crashed and rolled forward"), if its
sufficient, is trivially able to be done by having Postgresql (and most
other databases) keep a sufficient number of log segments that a
rollover cannot happen during the dump process itself, and either
snapshotting both filesystems at once, keeping both on the same
filesystem (undesirable for performance reasons) or dumping the database
first and XLOG second.

However, whether either of these approaches is sufficient is another
matter. One of the real problems with live transaction processing
systems is a means to know when there is a failure exactly what you
lost. This is not a trivial problem to solve and requires plenty of
thought before implementation, especially if you cannot afford the
outage time necessary to take the snapshots - in some cases even that
(relatively) short outage time is unacceptable.

[Peter Schuller]

> However, whether either of these approaches is sufficient is another
> matter. One of the real problems with live transaction processing
> systems is a means to know when there is a failure exactly what you
> lost. This is not a trivial problem to solve and requires plenty of
> thought before implementation, especially if you cannot afford the
> outage time necessary to take the snapshots - in some cases even that
> (relatively) short outage time is unacceptable.

I would like to point out that if the backup strategy is correct, a
COMMIT is guaranteed to be correctly honored, and the problem of
determining what was lost has more to do with the birds-eye view of to
what point in time the database was reverted as part of emergency
recovery, than any difficulty in understanding what actually happens
during snapshot recovery.

I completely understand that you have various requirements in
production that makes it a non-solution to just get a consistent
snapshot at some arbitrary point in time without synchronizing with
other software components somehow, but such issuse are into the realm
of application design and integration with the backup procedure, and
we are no longer talking about the viability of obtaining a consistent
backup of a single database through snapshotting.

[David Malone]

On Tue, Mar 24, 2009 at 02:03:12PM -0400, Mikhail T. wrote:
> Same problem:
>
> restore -rf ibmo.0.2009-03-24.dump
> load: 0.55 cmd: restore 11303 [nbufkv] 3.53u 3.91s 4% 27980k

> unknown tape header type 213474529
> abort? [yn]
>

> Please, advise. Thanks! Yours,

Hi Mikhail,

If you actually need to get a dump back that restore can't read,
you can try the "-D" option that I added a few years ago. Dump and
restore expect things to be in a block format, but if (say) dump
outputs a few bytes into the stream due to a bug, then the entire
end of the dump can become unreadable. The -D option to restore
tells it to try hard to get back in sync again.

I'd guess you've tripped over either a bug in dump or restore. If
you can file a PR, particularly with access to a sample dump, then
I can have a look and see if I can figure out what's going on.

Daid.

[Jack L. Stone]

At 09:32 AM 3.26.2009 -0400, Mike Tancsa wrote:
>At 08:08 AM 3/26/2009, Jack L. Stone wrote:
>

>>Yes, but it's for running a dump on a (L)ive FS and just spits out warnings
>>to that effect and has no effect on solving the problem(s).
>

>Unless the filesystem is very busy, you will get your data backed up.
>If you have things like databases, I still would not trust

>snapshots. Better to use pg_dump or mysqldump or the app that comes
>with whatever DB you are using... When backing up things like / and
>/usr, I would hazard a guess that most things are not changing while
>the backup is running, at least they dont in my environments. I have
>never had a problem with things like /home and even /var or /mail
>which are changing quite a bit. We dont restore much in the course
>of our daily routine, but we have always been able to restore
>people's Maildir when they accidentally have deleted stuff and it all
>worked without issue over the years.
>
> ---Mike

Yes, we have been using the "L" switch for as long as it has existed
because we have and always before that, backed up live FSes. And, as said
before, we do a dump/restore dump every morning to produce a bootable
clone. The clones have always worked for the many years we have done this.

Indeed, we have always used the MySQL's own dump separately to backup DBs.
But, that's OT.

[Jack L. Stone]

At 10:15 AM 3.26.2009 -0400, Mike Tancsa wrote:
>At 10:01 AM 3/26/2009, Peter Schuller wrote:
>> >

>> > Unless the filesystem is very busy, you will get your data backed up.
>> > If you have things like databases, I still would not trust
>> > snapshots.
>>
>>Uh. If backuping up a live database from a snapshot is not
>>trustworthy, either the snapshot facility is broken or the database is
>>broken (i.e., not crash-safe to begin with).
>

>... or the database is configured in a risky way... But judging by
>this thread, it seems dump with -L is indeed broken for some
>people. Hence, I suggested dumping the database using the database's
>backup tools and trying dump without -L
>

> ---Mike

Mike: The "L" for dumping is definitely NOT the problem.

Jack

(^_^)
Happy trails,
Jack L. Stone

System Admin
Sage-american

[Pete French]

> Absolutely. You really must use a tool that interacts with the database
> to perform the backup. Most commercial DBs have hooks that allow the
> backup routines to call out to custom snapshot facilities. One would
> usually request a backup through the database, which would then freeze IO
> to its data files and maybe log files, deal with flushing caches etc and
> then call your snapshot routine. I'm not aware that MySQL and Postgres do
> though so the best you can do is a dump.

mysql can do this - you can flush the tables and acuire a lock
simultaneously so that you can then snapshot the uderlying filesystem
and then release the lock to let everything continue. I use this for taking
database snapshots and it works fine. I stop my slaves before snapshotting
to avoid log files changing underneath me too .... like this...

#!/bin/sh
/usr/local/bin/mysql -usnapuser -psnapuser <<EOF
slave stop;
flush tables with read lock;
system /sbin/zfs snapshot archive/mysql@latest;
unlock tables;
slave start;
EOF

That appears to work fine. I do also do other dumps, but the above
works nicely for a quick and easy snapshot on a slave which can be
rolled back in the case of a crash (and will then update from the master
properly)

-pete.

[Jack L. Stone]

At 03:49 PM 3.26.2009 +0100, Peter Schuller wrote:
>> ... or the database is configured in a risky way... But judging by

>> this thread, it seems dump with -L is indeed broken for some
>> people. Hence, I suggested dumping the database using the database's
>> backup tools and trying dump without -L
>

>Well, if dump -L is really broken I'd recommend just not using dump
>unless you absolutely have to. That is assuming snapshots are not
>broken so that you can still use mksnap_ffs with your favorite backup
>tool.
>
>--
>/ Peter Schuller
>

No one has said the dump "L" is broken -- and is NOT but now may mislead
others just tuning into this thread.

Jack

(^_^)
Happy trails,
Jack L. Stone

System Admin
Sage-american

[Jack L. Stone]

At 10:46 AM 3.26.2009 -0500, Karl Denninger wrote:

>Jake Scott wrote:
>>
>>> That said there are plenty of other reasos to use proper dump tools
>>> (data portability, confirming the ability to actually read all rows
>>> from a table, using a more often exercised code path and perhaps less
>>> likely to have edge case bugs, etc).
>>

This thread has drifted off the main issue of using the FS dump/restore
problem.

Although, the DBs may be included as a part of this dump, it should NOT be
relied upon as a backup/restore of DBs. We use MySQL's dump/backup tools
strictly for the DBs. The dumps of the DBs (using MySQL's own dump tool)
here are run by cron jobs every hour or few hours at most. That keeps our
DB backups very recent and near current as possible is a restore is needed.

Jack

(^_^)
Happy trails,
Jack L. Stone

System Admin
Sage-american

[Tom Evans]

On Thu, 2009-03-26 at 14:45 +0000, Jake Scott wrote:
.
> Absolutely. You really must use a tool that interacts with the database
> to perform the backup. Most commercial DBs have hooks that allow the
> backup routines to call out to custom snapshot facilities. One would
> usually request a backup through the database, which would then freeze IO
> to its data files and maybe log files, deal with flushing caches etc and
> then call your snapshot routine. I'm not aware that MySQL and Postgres do
> though so the best you can do is a dump.
>
>
> Jake
>

Just to add, mysql has a utility (mysqlhotcopy) to allow you to directly
copy MyISAM databases with a guarentee of consistency (thus avoiding the
conversion from MyISAM data -> SQL, and no need to reimport when
recovering). It isn't exactly online though, any writes will be blocked
until the hotcopy finishes.

Still, it is only MyISAM, and not much call for that these days..

Cheers

Tom

[Mike Tancsa]

At 01:11 PM 3/26/2009, Jack L. Stone wrote:

>No one has said the dump "L" is broken -- and is NOT but now may mislead
>others just tuning into this thread.

OK, sorry I misunderstood that restore worked from dump files without -L....
I wonder if the original poster just needs to do an fsck on the disk.
The only time I had dump crap out on me (other than with -L) was on a
dirty filesystem.

I regularly do stuff like
cd /usr;dump 0f - /usr | (cd /tmp/usr; restore -rf - )
to duplicate partitions

---Mike

[Karl Denninger]

Mike Tancsa wrote:
> At 01:11 PM 3/26/2009, Jack L. Stone wrote:
>
>> No one has said the dump "L" is broken -- and is NOT but now may mislead
>> others just tuning into this thread.
>
>
> OK, sorry I misunderstood that restore worked from dump files without
> -L....
> I wonder if the original poster just needs to do an fsck on the disk.
> The only time I had dump crap out on me (other than with -L) was on a
> dirty filesystem.
>
> I regularly do stuff like
> cd /usr;dump 0f - /usr | (cd /tmp/usr; restore -rf - )
> to duplicate partitions
>
> ---Mike

I do as well but if the original user is trying to restore into a
filesystem that is either not clean or not EMPTY for a full restore he
could run into some trouble.

I've never had dump/restore fail by the way, other than due to bad
(tape) media (yes, it was that long ago :) )

[Peter Schuller]

> >> ... or the database is configured in a risky way... But judging by
> >> this thread, it seems dump with -L is indeed broken for some
> >> people. Hence, I suggested dumping the database using the database's
> >> backup tools and trying dump without -L
> >
> >Well, if dump -L is really broken I'd recommend just not using dump
> >unless you absolutely have to. That is assuming snapshots are not
> >broken so that you can still use mksnap_ffs with your favorite backup
> >tool.
> >
> >--
> >/ Peter Schuller
> >
>
> No one has said the dump "L" is broken -- and is NOT but now may mislead
> others just tuning into this thread.

I was commenting on the specific statement that I quoted above. I am
not claiming anything about dump being broken.

[Eugene Gladchenko]

Mikhail,

MT> I'm trying to migrate a filesystem from one disk to another using:
MT> dump a0hCf 0 32 - /old | restore -rf -
MT> (/old is already mounted read-only). The process runs for a while and
MT> then stops with:
MT> [...]
MT> DUMP: 22.85% done, finished in 3:57 at Tue Mar 24 01:03:21 2009
MT> DUMP: 24.66% done, finished in 3:50 at Tue Mar 24 01:00:58 2009
MT> DUMP: 26.44% done, finished in 3:43 at Tue Mar 24 00:59:14 2009
MT> unknown tape header type 1853384566
MT> abort? [yn]
MT> Any idea, what's going on? Why can't FreeBSD's restore read FreeBSD's
MT> dump's output?

Is your /tmp large enough to restore?

--
Eugene Gladchenko
EVG15-RIPE

[Rolf G Nielsen]

Mikhail T. wrote:
> Hello!

>
> I'm trying to migrate a filesystem from one disk to another using:
>

> dump a0hCf 0 32 - /old | restore -rf -

Try
dump -a0 -h 0 -C 32 -f - /old
instead.
Your command line should work, but I've had trouble with similar ones
and have been able to solve it by separating the options.

>
> (/old is already mounted read-only). The process runs for a while and

> then stops with:
>
> [...]

> DUMP: 22.85% done, finished in 3:57 at Tue Mar 24 01:03:21 2009

> DUMP: 24.66% done, finished in 3:50 at Tue Mar 24 01:00:58 2009

> DUMP: 26.44% done, finished in 3:43 at Tue Mar 24 00:59:14 2009

> unknown tape header type 1853384566

> abort? [yn]

>
> Any idea, what's going on? Why can't FreeBSD's restore read FreeBSD's

> dump's output?
>
> The system runs 7.0-STABLE from July 6th, i386. I just tried updating
> the dump and restore from source (latest 7.x) -- the error is the same...
>
> Please, advise. Thanks!
>
> -mi

>
> _______________________________________________
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"
>
>
>

--

Sincerly,

Rolf Nielsen

[Greg Black]

On 2009-03-25, Mikhail T. wrote:
> Greg Black ???????(??):
>> On 2009-03-24, Mikhail T. wrote:
>>
>>> That's true. I just wanted to point out, that someone running dump only
>>> (to make backups) is not going to know, whether his dumps are usable
>>> (for whichever of the two reasons), until he needs them...
>>
>> Such a person is not making backups and deserves what he gets.
>>
> But he *is* making backups -- kindly re-read my paragraph above... He
> just is not routinely using them and thus does not know, that they
> aren't usable... It is not unreasonable to expect the two utilities to
> "just work", so I wouldn't be blaming such a person for not testing
> restore.

Sorry, this person is *not* making backups in any meaningful fashion.
Unless you verify regularly (preferably every time you make a backup)
that you can restore both parts of the backup and the entire thing, you
are not making backups. You may be lucky, but you probably won't be
lucky when it really matters.

This is a long-standing issue with backups. But people who don't
understand the tools they are using tend to make these mistakes. The
result is bad. The solution is to employ competent people who both
understand the issues and have the authority to carry out their task.

[Doug Barton]

Greg Black wrote:
> Sorry, this person is *not* making backups in any meaningful fashion.
> Unless you verify regularly (preferably every time you make a backup)
> that you can restore both parts of the backup and the entire thing, you
> are not making backups. You may be lucky, but you probably won't be
> lucky when it really matters.

s/making backups/employing an effective data recovery strategy/

Please don't feed the "proper use of language" trolls. :)

Meanwhile, we may want to consider that this whole thread is veering
off topic ...

Doug

--

This .signature sanitized for your protection

[Mikhail T.]

Greg Black написав(ла):

> Sorry, this person is *not* making backups in any meaningful fashion.
> Unless you verify regularly (preferably every time you make a backup)
> that you can restore both parts of the backup and the entire thing, you
> are not making backups.

To qualify for your (and your kind's) recognition then, a person needs
to have at least as much extra storage capacity as the largest
filesystem they are backing up. They also need non-trivial scripting
abilities, because the OS doesn't include anything like what you are
describing (and I already do consider scheduling dumps via cron
"trivial", which may be a stretch). Yours may thus be an acceptable
requirement for a multi-computer shop with dedicated system
administration personnel, but for a private home user with only one
computer this simply is not reasonable.

Stating this as a requirement is ridiculous -- unless you are prepared
to say, that such people should not own a computer (with worthy data) at
all. And that's even more ridiculous... Make your pick.

I would agree with you, if the chosen backup method involved some
complex or third-party tools. But if the simple, OS-supplied orthogonal
dump/restore don't work together, then the OS is broken -- plain and
simple, and pointing a finger at the user: "Well, it is all your fault,
because you relied on us providing you with working utilities,
ha-ha-ha!" -- is the lamest excuse imaginable.

-mi

P.S. Some people have actually volunteered to help debug this problem
and I'm working on providing them with data (the troublesome partition
is, sadly, over 170Gb, so it takes a while). Any results/conclusions
will be posted under the original subject.
P.P.S. The data transferred fine using tar, but that is not the point --
the bug (confirmed by at least one more person) -- needs to be fixed
before a higher-profile embarrassment...

[Andrew Snow]

Mikhail T. wrote:
> To qualify for your (and your kind's) recognition then, a person
> needs to have at least as much extra storage capacity as the
> largest filesystem they are backing up. They also need
> non-trivial scripting abilities, because the OS doesn't
> include anything like what you are describing

Mikhail, users would be well advised to check their backups using this
option, without having to have the space to restore:

-N Do the extraction normally, but do not actually write any
changes to disk. This can be used to check the integrity of dump media
or other test purposes.

RESTORE(8) FreeBSD System Manager's Manual

[Mikhail T.]

Andrew Snow написав(ла):

> Mikhail T. wrote:
> > To qualify for your (and your kind's) recognition then, a person
> > needs to have at least as much extra storage capacity as the
> > largest filesystem they are backing up. They also need
> > non-trivial scripting abilities, because the OS doesn't
> > include anything like what you are describing
>
> Mikhail, users would be well advised to check their backups using this
> option, without having to have the space to restore:
>
> -N Do the extraction normally, but do not actually write any
> changes to disk. This can be used to check the integrity of dump media
> or other test purposes.

And then another "purist" will reject this as a fake and not
sufficiently conclusive test... Just ask around... Because, you see,
until you extract all data back and run a bit-by-bit comparison with the
original, you don't really know, do you?

-mi