fsck -r and -g errors: what to do next?

[Nico]

Hi!

First of all thank you for the software, i ave been using it for some time and its great :)

I run a backup every couple of days and fsck every week or month.

However I may have a problem since recently I am no longer able to pass "fsck". I am able to save new backups.

There seems to be a single corrupted pack. The output from quick fsck is:

    $ bup fsck --quick objects/pack/pack-4e15073a62edb135ce5183fdf42d80734af5d90d.pack -vvv
    fsck: checking pack-4e15073a62edb135ce5183fdf42d80734af5d90d (git)
    error: expected '0000000000000000000000000000000000000000', got '3d4c0e4b5dbaf62032a002278620fd91f60d1a95'

    pack-4e15073a62edb135ce5183fdf42d80734af5d90d git verify: failed (1)

    pack-4e15073a62edb135ce5183fdf42d80734af5d90d failed 

    fsck done.

Normal fsck says:

    $ bup fsck objects/pack/pack-4e15073a62edb135ce5183fdf42d80734af5d90d.pack 

    error: packfile ./objects/pack/pack-4e15073a62edb135ce5183fdf42d80734af5d90d.pack does not match index

--- prints the previous line many times ---

    fatal: pack is corrupted (SHA1 mismatch)
    pack-4e15073a62edb135ce5183fdf42d80734af5d90d git verify: failed (1)

In the objects directory i have:

• pack-4e15073a62edb135ce5183fdf42d80734af5d90d.pack
• pack-4e15073a62edb135ce5183fdf42d80734af5d90d.odx
  

But no .par2 file with that name.

I tried generatin the par2 file with fsck -g, which exited with the same error as plain fsck.

What should I do?

Thanks!

[Nico]

PD: i made a typo, the "odx" file is a "idx" file, .

[Rob Browning]

Nico <nmend...@gmail.com> writes:

> I run a backup every couple of days and fsck every week or month.

Are you running "fsck -g", i.e. you're normally generating par2 files,
but just don't have one for the problematic pack file?

> objects/pack/pack-4e15073a62edb135ce5183fdf42d80734af5d90d.pack

Hmm, what does

ls -l objects/pack/pack-4e15073a62edb135ce5183fdf42d80734af5d90d.pack

show?

--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Greg Troxel]

Nico <nmend...@gmail.com> writes:

> First of all thank you for the software, i ave been using it for some time
> and its great :)
>
> I run a backup every couple of days and fsck every week or month.
>
> However I may have a problem since recently I am no longer able to pass
> "fsck". I am able to save new backups.
>
> There seems to be a single corrupted pack. The output from quick fsck is:
>
> $ bup fsck --quick
> objects/pack/pack-4e15073a62edb135ce5183fdf42d80734af5d90d.pack -vvv
> fsck: checking pack-4e15073a62edb135ce5183fdf42d80734af5d90d (git)
> error: expected '0000000000000000000000000000000000000000', got
> '3d4c0e4b5dbaf62032a002278620fd91f60d1a95'
> pack-4e15073a62edb135ce5183fdf42d80734af5d90d git verify: failed (1)
> pack-4e15073a62edb135ce5183fdf42d80734af5d90d failed
> fsck done.
>
> Normal fsck says:
>
> $ bup fsck
> objects/pack/pack-4e15073a62edb135ce5183fdf42d80734af5d90d.pack
> error: packfile
> ./objects/pack/pack-4e15073a62edb135ce5183fdf42d80734af5d90d.pack does not
> match index
> --- prints the previous line many times ---
> fatal: pack is corrupted (SHA1 mismatch)
> pack-4e15073a62edb135ce5183fdf42d80734af5d90d git verify: failed (1)

So you have a corrupted pack. This is likely because there was a
problem in the filesystem or the disk.

I am not clear on recovery. Ideally there would be something that
attempts to iterate over a packfile and copy the valid objects to a new
one, so you'd lose as little as possible.

> But no .par2 file with that name.

Are you normally running fsck -g after each backup, to guard against
disk issues?

Did you run "fsck -r"? But, if not par2 files, that won't help.

I would first copy the repo to another dis,

Then, 'git fsck' might help.

I hope you are backing up to multiple independent bup repos. There
may be a way to fetch missing stuff via 'bup get'.

I expect that others have some wisdom.

[Nico]

Hi Rob and Greg. Thank you for your replies.

Rob:

The output of "ls -l objects/pack/pack-4e15073a62edb135ce5183fdf42d80734af5d90d.pack" is:

-rw------- 1 me me 13679919 ago  9  2020 objects/pack/pack-4e15073a62edb135ce5183fdf42d80734af5d90d.pack

I still save daily backups to this repo, and I used to be able to verify backups and generate recovery information every 30 days. All through KDE's "kup".

----

Greg:

It is possible that an unnoticed failed backup happened circa 9/august/2020, for whatever reason (most likely a power outage).

Perhaps i should point out that the repos reside in external USB HDDs.

I did run "fsck -r" but unfortunately it didn't help.

I backup slightly different sets of directories to three different bup repos (in different USB drives) at different times.

Would those be useful? I'm not sure what "missing stuff" means though.

[Nico]

Further questions:

What are the consequences of a corrupted packfile? Does the corruption affect a single backup date?

At the very least, I would like to know what was lost, and then be able to "fix" the repo (so that it passes the verification step, and i can save recovery information once again).

Is there a way of listing the "contents" of that packfile?

Is there a way of "fixing" the repo? Even if some information about that packfile is permanently lost.

Thank you.

[Rob Browning]

Nico <nmend...@gmail.com> writes:

> It is possible that an unnoticed failed backup happened circa
> 9/august/2020, for whatever reason (most likely a power outage).
>
> Perhaps i should point out that the repos reside in external USB HDDs.
>
> I did run "fsck -r" but unfortunately it didn't help.

Right. It looks like for whatever reason, you don't have a par2 file
for the unverfiable pack (i.e. no repair data). So all we have is the
packfile itself, which might or might not still be partially useful, but
there's no automatic way to know which parts of it might be ok.

In the worst case, I suspect we might still be able to recover some data
from it (depending on what's wrong) with "effort", but maybe it won't
come to that.

I suppose one unlikely possibility might be that you had a crash right
after bup finished that packfile, but before it was fully synced to
disk. Another option might be disk or fs corruption...

The current approach shouldn't move a "pack-..." into place until after
the data has been fully synced, so unless the platform or hardware is
lying (both have been known to in the past), the window for that
particular scenario really shouldn't exist.

(I'll reply with more details in a reply to your next message.)

[Rob Browning]

Nico <nmend...@gmail.com> writes:

> What are the consequences of a corrupted packfile? Does the corruption
> affect a single backup date?

Because bup deduplicates, each packfile only contains the fragments
(determined by the rolling checksum) that didn't already exist in some
other packfile in the repository that are needed by at the time of any
given "bup save".

That does mean that (assuming you haven't run "bup gc"), any problems
with that packfile should only affect saves run after it was created,
and the effect of the bad packfile is that future saves assume that all
of the objects bup thinks are in that packfile (all listed in the
corresponding .idx) don't need to be saved.

Now if the problem is that the bad packfile has missing objects, say it
was truncated, then I'd expect a future restore of any save that
depended on that packfile to crash when it tried to fetch one of the
missing objects. Alternately, the packfile might be corrupted, which
means the restore might work, but the results couldn't be entirely
trusted.

> At the very least, I would like to know what was lost, and then be
> able to "fix" the repo (so that it passes the verification step, and i
> can save recovery information once again).

With the current tools, there are a number of options, depending on how
far down the rabbit hole you want to go. I'll give an outline, and can
try to help with further detail, depending on how you feel you might
want to proceed.

In places where I refer to questionable.{pack,idx} below, I'm assuming
(and suggest) you work from a copy of the original
pack-4e15073a62edb135ce5183fdf42d80734af5d90d.idx etc., so we don't
accidentally make things worse. And you'll usually want both those
files, side-by-side in the same dir.

Also, I'd strongly recommend working on a copy of your repo, if you can
afford the space.

1) Examining the packfile and index

You can look at the contents of the broken pack/idx with various git
tools like this:

git show-index < questionable.idx

That will show all the hashes that the index claims are in the packfile,
which should be at least a subset (if not all) of the hashes that could
be missing or corrupted.

This should show a lot more information, and might give us some clue
about what's actually wrong:

git verify-pack -v questionable.idx

Another possibility

git index-pack -v -o dummy.idx questionable.pack

Of course you can delete the dummy.idx afterward -- just want to see
what the process says about the pack.

2) Examining the repo

You might want to try this first, but it's going to be a lot more
expensive than the commands above (i.e. it has to traverse *all* the
data in the repository):

git --git-dir=BUP_REPO fsck --name-objects

This might tell us more or less just what you wanted to know with
respect to which saves are broken.

I believe you can also pass final "ref" arguments to limit the check to
particular saves, i.e. you can give it branch names or individual save
commit hashes, which you can see those via "bup ... ls --hash ...".

3) Rolling back

One way to repair the repository would be to "roll back" all of your
save branches to just before that packfile's creation, then delete that
packfile and idx, and clobber the midx and bloom files.

To roll back, we'd need to figure out which save to roll back to, and
then use whatever git tools we like to reset the branch, e.g.

git --git-dir=BUP_REPO branch -f SAVE_BRANCH_NAME HASH_OF_SAVE_BEFORE_TROUBLE

You can see the save hashes via "bup ... ls --hash".

Then remove the packfile and index.

Next, while "bup gc" knows how to clobber midx and bloom, we don't yet
have top-level --clear commands, so instead you can do this:

rm REPO/objets/pack/midx-*
bup -d REPO bloom --force
bup -d REPO midx -a

Then just start making new saves again. You'd lose access to any saves
after that point, but it'd be "quick".

Afterward, all the other, newer packfiles would still be there, which
might speed the next save, and eventually, you could use bup rm/gc or
prune-older to drop any vestigial bits that were no longer needed.

I'd also suggest making sure "git verify" is happy with the repo (aside
from dangling blobs) at this point.

4) "Fixing" the repo more precisely than rolling back (riskier)

If the directories you're saving haven't changed a lot since the broken
packfile was created, then you might be able to move it (and its idx)
out of the objects/ dir, delete all the .midx files in the repo, run
"bup ... bloom --force", and then run new saves.

If those succeed, they may reintroduce some of the missing fragments
(objects) so that more of the older saves that were broken by the bad
packfile are now complete.

Basically, we'd have to find some way of re-saving all of the hashes
that a git-fsck still reports missing, for any saves you want to keep.

I'd also suggest making sure "git verify" is happy with the repo (aside
from dangling blobs) in the end.

5) Cleaning up

If you manage to get your repository in a state you're happy with, even
though there are still some broken saves. You can remove those via "bup
rm", so they won't affect future commands. This will likely just leave
some "dangling blobs" in your repository, but those are harmless, and
will nearly always exist if you've ever used "bup gc", since it doesn't
try to be precise by default.

Hope this helps