bup validate-ref-links blows up on empty repo

[Greg Troxel]

I know this is not really important. I've been running lots of
test-type invocations on all of my repos to patrol for midx corruption.
Mostly this is going well, and I'm nuking a few bad midx files.

I had a stray error on /b0/bup-test which is at the moment emtpy. I use
that as a destination in testing and regularly discard contents.

My impression is that there is no bug tracker and this is a feature :-)
so I am sending here. Feel free to decide this is not worth fixing.


A reproducer:

$ bup -d /tmp/bup init
Initialized empty Git repository in /tmp/bup/

$ bup -d /tmp/bup validate-ref-links
Traceback (most recent call last):
File "/usr/pkg/lib/bup/bup/main.py", line 221, in main
rc = run_subcmd(cmd_module, subcmd, opt['profile'])
File "/usr/pkg/lib/bup/bup/main.py", line 178, in run_subcmd
return module.main(args)
~~~~~~~~~~~^^^^^^
File "/usr/pkg/lib/bup/bup/cmd/validate_ref_links.py", line 20, in main
return validate_refs.main(args)
~~~~~~~~~~~~~~~~~~^^^^^^
File "/usr/pkg/lib/bup/bup/cmd/validate_refs.py", line 138, in main
find_live_objects(existing_count, cat_pipe,
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
refs=ref_info,
^^^^^^^^^^^^^^
idx_list=idxl,
^^^^^^^^^^^^^^
for_item=for_item,
^^^^^^^^^^^^^^^^^^
verbosity=verbosity)
^^^^^^^^^^^^^^^^^^^^
File "/usr/pkg/lib/bup/bup/gc.py", line 93, in find_live_objects
live_blobs = BloomWriter(bloom_filename, 'w+b', expected=existing_count, k=None)
File "/usr/pkg/lib/bup/bup/bloom.py", line 270, in __init__
assert expected > 0, expected
^^^^^^^^^^^^
AssertionError: 0
Exception ignored in: <function BloomWriter.__del__ at 0x7895433b3770>
Traceback (most recent call last):
File "/usr/pkg/lib/bup/bup/bloom.py", line 343, in __del__
def __del__(self): assert not self.map, self.path
AttributeError: 'bup.bloom.BloomWriter' object has no attribute 'map'

$ ls -l /tmp/bup/objects/pack/tmp-gc-i71b8dve.bloom
-rw------- 1 gdt wheel 0 Jul 13 10:24 /tmp/bup/objects/pack/tmp-gc-i71b8dve.bloom


My script "bup-check":

----------------------------------------
#!/bin/sh

repo=$1

if [ ! -f "$repo"/config ]; then
echo "$repo is not a bup repo"
exit 1
fi

repotag=`echo ${repo} | sed -e 's,/,_,g'`

file=CHECK.`date +%Y%m%d`.${repotag}

if [ -f "${file}" ]; then
echo "${file} exists"
exit 1
fi

(date
echo "midx check"
bup -d $repo midx -a --check --max-files 5
date
echo "midx auto (no force)"
bup -d $repo midx -a --max-files 5
date
echo "validate-ref-links"
bup -d $repo validate-ref-links
date
echo "validate-object-links"
bup -d $repo validate-object-links
date
echo "fsck"
bup -d $repo fsck
date
) < /dev/null > ${file} 2>&1

[Rob Browning]

Greg Troxel <g...@lexort.com> writes:

> I know this is not really important. I've been running lots of
> test-type invocations on all of my repos to patrol for midx corruption.
> Mostly this is going well, and I'm nuking a few bad midx files.

What's the indicator that they're bad, again?

Also of course I wonder how they got that way --- though we've improved
the midx handling a good bit in main (and 0.33.x) over the past year or
two, I think. Things that could have caused trouble.

For example, we've improved the sync/close handling (overall resource
management), and we should now detect and delete midxes with missing
idxes for --auto and --force.

> My impression is that there is no bug tracker and this is a feature :-)
> so I am sending here. Feel free to decide this is not worth fixing.
>
> A reproducer:
>
> $ bup -d /tmp/bup init
> Initialized empty Git repository in /tmp/bup/
>
> $ bup -d /tmp/bup validate-ref-links
> Traceback (most recent call last):
> File "/usr/pkg/lib/bup/bup/main.py", line 221, in main

Certainly a bug, and thanks for reporting it. Will fix.

--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Greg Troxel]

Rob Browning <r...@defaultvalue.org> writes:

> Greg Troxel <g...@lexort.com> writes:
>
>> I know this is not really important. I've been running lots of
>> test-type invocations on all of my repos to patrol for midx corruption.
>> Mostly this is going well, and I'm nuking a few bad midx files.
>
> What's the indicator that they're bad, again?

asserts when closing, probably because something bailed out.

> Also of course I wonder how they got that way --- though we've improved
> the midx handling a good bit in main (and 0.33.x) over the past year or
> two, I think. Things that could have caused trouble.

I think it's that and perhaps system crashes.

> For example, we've improved the sync/close handling (overall resource
> management), and we should now detect and delete midxes with missing
> idxes for --auto and --force.

Agreed it's much improved, which is most of why I am declining to try to
figure it out!

>> $ bup -d /tmp/bup init
>> Initialized empty Git repository in /tmp/bup/
>>
>> $ bup -d /tmp/bup validate-ref-links
>> Traceback (most recent call last):
>> File "/usr/pkg/lib/bup/bup/main.py", line 221, in main
>
> Certainly a bug, and thanks for reporting it. Will fix.

Grea, thanks

[Rob Browning]

Skip the --links tests when the object count is zero, and for now have
gc.find_live_objects reject an existing_count of zero, i.e. detecting
an empty repository is the caller's responsibility.

Thanks to Greg Troxel for reporting the problem.

Signed-off-by: Rob Browning <r...@defaultvalue.org>
Tested-by: Rob Browning <r...@defaultvalue.org>
---

Pushed to main.

lib/bup/cmd/validate_refs.py | 25 +++++++++++++------------
lib/bup/gc.py | 1 +
test/ext/test-validate-refs | 1 +
3 files changed, 15 insertions(+), 12 deletions(-)

diff --git a/lib/bup/cmd/validate_refs.py b/lib/bup/cmd/validate_refs.py
index 85f53365..f81b23f7 100644
--- a/lib/bup/cmd/validate_refs.py
+++ b/lib/bup/cmd/validate_refs.py
@@ -129,18 +129,19 @@ def main(argv):
existing_count = count_objects(git.repo(b'objects/pack'), verbosity)
if verbosity:
log(f'found {existing_count} objects\n')
- with ExitStack() as maybe_close_idxl:
- idxl = None
- if opt.links:
- idxl = git.PackIdxList(git.repo(b'objects/pack'))
- maybe_close_idxl.enter_context(idxl)
- live_objs, live_trees_ = \
- find_live_objects(existing_count, cat_pipe,
- refs=ref_info,
- idx_list=idxl,
- for_item=for_item,
- verbosity=verbosity)
- live_objs.close()
+ if existing_count:
+ with ExitStack() as maybe_close_idxl:
+ idxl = None
+ if opt.links:
+ idxl = git.PackIdxList(git.repo(b'objects/pack'))
+ maybe_close_idxl.enter_context(idxl)
+ live_objs, live_trees_ = \
+ find_live_objects(existing_count, cat_pipe,
+ refs=ref_info,
+ idx_list=idxl,
+ for_item=for_item,
+ verbosity=verbosity)
+ live_objs.close()
if bad_bupm:
return EXIT_FAILURE
if (ref_missing + found_missing + abridged_bupm):
diff --git a/lib/bup/gc.py b/lib/bup/gc.py
index 5ff5de3b..7ca2c280 100644
--- a/lib/bup/gc.py
+++ b/lib/bup/gc.py
@@ -85,6 +85,7 @@ def find_live_objects(existing_count, cat_pipe, refs=None, *,
# has been provided and does not return True, then a MissingObject
# will be raised. If idx_list is provided, then existence checks
# will be broad.
+ assert existing_count > 0, existing_count
pack_dir = git.repo(b'objects/pack')
ffd, bloom_filename = tempfile.mkstemp(b'.bloom', b'tmp-gc-', pack_dir)
os.close(ffd)
diff --git a/test/ext/test-validate-refs b/test/ext/test-validate-refs
index 41d15190..b8bc38ab 100755
--- a/test/ext/test-validate-refs
+++ b/test/ext/test-validate-refs
@@ -21,6 +21,7 @@ WVPASS cd "$tmpdir"
WVSTART 'handling of correct refs'
WVPASS rm -rf bup src
WVPASS bup init
+WVPASS bup validate-refs
WVPASS git config bup.split.trees true
WVPASS mkdir src
WVPASS mkdir -p src/a/b
--
2.47.3