Failure on remote save

[Nathaniel]

Hello all

I have an issue where bup is failing to save. It looks quite similar to this one: https://groups.google.com/g/bup-list/c/IrZaPTm0PZ8

This is with bup 0.33.7 on Debian Bookworm

I have several remote directories that I index and save in a cron job. I make no effort to guarantee that these directories' contents don't get updated while bup runs (does bup expect me to?). Two of the directories have begun failing with errors like so:

Reading index: 195, done.

Traceback (most recent call last):

 File "/usr/lib/bup/bup/metadata.py", line 875, in read

   result._load_posix1e_acl_rec(port, version=1)

 File "/usr/lib/bup/bup/metadata.py", line 575, in _load_posix1e_acl_rec

   acl_rep = vint.unpack('ssss', vint.read_bvec(port))

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

 File "/usr/lib/bup/bup/vint.py", line 185, in unpack

   return recv(port, types)

          ^^^^^^^^^^^^^^^^^

 File "/usr/lib/bup/bup/vint.py", line 161, in recv

   result.append(read_bvec(port))

                 ^^^^^^^^^^^^^^^

 File "/usr/lib/bup/bup/vint.py", line 129, in read_bvec

   n = read_vuint(port)

       ^^^^^^^^^^^^^^^^

 File "/usr/lib/bup/bup/vint.py", line 44, in read_vuint

   raise EOFError('encountered EOF while reading vuint')

EOFError: encountered EOF while reading vuint

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

 File "<frozen runpy>", line 198, in _run_module_as_main

 File "<frozen runpy>", line 88, in _run_code

 File "/usr/lib/bup/bup/main.py", line 396, in <module>

   main()

 File "/usr/lib/bup/bup/main.py", line 393, in main

   wrap_main(lambda : run_subcmd(cmd_module, subcmd))

 File "/usr/lib/bup/bup/compat.py", line 94, in wrap_main

   sys.exit(main())

            ^^^^^^

 File "/usr/lib/bup/bup/main.py", line 393, in <lambda>

   wrap_main(lambda : run_subcmd(cmd_module, subcmd))

                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

 File "/usr/lib/bup/bup/main.py", line 388, in run_subcmd

   run_module_cmd(module, args)

 File "/usr/lib/bup/bup/main.py", line 289, in run_module_cmd

   import_and_run_main(module, args)

 File "/usr/lib/bup/bup/main.py", line 284, in import_and_run_main

   module.main(args)

 File "/usr/lib/bup/bup/cmd/save.py", line 518, in main

   with msr, \

 File "/usr/lib/bup/bup/index.py", line 70, in __exit__

   with pending_raise(value, rethrow=True):

 File "/usr/lib/bup/bup/compat.py", line 41, in __exit__

   raise self.ex

 File "/usr/lib/bup/bup/cmd/save.py", line 521, in main

   tree = save_tree(opt, reader, hlink_db, msr, w)

          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

 File "/usr/lib/bup/bup/cmd/save.py", line 378, in save_tree

   meta = msr.metadata_at(ent.meta_ofs)

          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

 File "/usr/lib/bup/bup/index.py", line 75, in metadata_at

   return metadata.Metadata.read(self._file)

          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

 File "/usr/lib/bup/bup/metadata.py", line 888, in read

   raise Exception("EOF while reading Metadata")

Exception: EOF while reading Metadata

If I run it with BUP_DEBUG=11  and pass four verbose flags, I get the following:

pack-61c3e4cf39d9682ab2ab2138c7019f2008e831a5.idx

<About 83 gazillion more lines like the above>

client: removing extra indexes: set()

client: server requested load of: set()

midx: scanning /root/.bup/index-cache/oncata_com_

midx: 5 indexes; want no more than 5.

midx: nothing to do.

bloom: scanning /root/.bup/index-cache/oncata_com_

bup server: command: b'read-ref refs/heads/nextcloud'

bup server: command: b'quit'

bup server: done

bloom: nothing to do.

PackIdxList: using 5 indexes.

Reading index: 2710, done.

  /root/nextcloud/volumes/postgres/postmaster.pid

  /root/nextcloud/volumes/postgres/postmaster.opts

Traceback (most recent call last):

  File "/usr/lib/bup/bup/metadata.py", line 886, in read

    tag = vint.read_vuint(port)

          ^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/bup/bup/vint.py", line 44, in read_vuint

    raise EOFError('encountered EOF while reading vuint')

EOFError: encountered EOF while reading vuint

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "<frozen runpy>", line 198, in _run_module_as_main

  File "<frozen runpy>", line 88, in _run_code

  File "/usr/lib/bup/bup/main.py", line 396, in <module>

    main()

  File "/usr/lib/bup/bup/main.py", line 393, in main

    wrap_main(lambda : run_subcmd(cmd_module, subcmd))

  File "/usr/lib/bup/bup/compat.py", line 94, in wrap_main

    sys.exit(main())

             ^^^^^^

  File "/usr/lib/bup/bup/main.py", line 393, in <lambda>

    wrap_main(lambda : run_subcmd(cmd_module, subcmd))

                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/bup/bup/main.py", line 388, in run_subcmd

    run_module_cmd(module, args)

  File "/usr/lib/bup/bup/main.py", line 289, in run_module_cmd

    import_and_run_main(module, args)

  File "/usr/lib/bup/bup/main.py", line 284, in import_and_run_main

    module.main(args)

DemuxConn: marked closed

  File "/usr/lib/bup/bup/cmd/save.py", line 518, in main

    with msr, \

  File "/usr/lib/bup/bup/index.py", line 70, in __exit__

    with pending_raise(value, rethrow=True):

  File "/usr/lib/bup/bup/compat.py", line 41, in __exit__

    raise self.ex

  File "/usr/lib/bup/bup/cmd/save.py", line 521, in main

    tree = save_tree(opt, reader, hlink_db, msr, w)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/bup/bup/cmd/save.py", line 378, in save_tree

    meta = msr.metadata_at(ent.meta_ofs)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/bup/bup/index.py", line 75, in metadata_at

    return metadata.Metadata.read(self._file)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/bup/bup/metadata.py", line 888, in read

    raise Exception("EOF while reading Metadata")

Exception: EOF while reading Metadata

I took a copy of my BUP_DIR and did a `bup index --clear` but that did not help; the error still occurs just the same. Unfortunately, I don't seem to have any way to use bup to backup those directories right now.

Advice welcome. I can run whatever tests.

—
Nathaniel

[Greg Troxel]

You log output is mangled by line wrapping so it's very hard to read.

suggestions:
- check the integrity of the repository with bup fsck, git fsck
- remove all cached index files from the client
- remove all midx files (but see what others think, and set them aside
vs losing them)
- look at your ssh logs on the server and see if the connection ends
normally or if there is an error
- look at /var/log/messages or equivalent on both machines

[Rob Browning]

"'Nathaniel' via bup-list" <bup-...@googlegroups.com> writes:

> I have an issue where bup is failing to save. It looks quite similar
> to this one: https://groups.google.com/g/bup-list/c/IrZaPTm0PZ8 This
> is with bup 0.33.7 on Debian Bookworm

What command is failing, i.e. is this a "save -r" or "bup on save" or ...?

--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Nathaniel Hourt]

On 2025-11-18 08:28, Greg Troxel wrote:
> You log output is mangled by line wrapping so it's very hard to read.

How odd... I had some trouble with it, but I thought I got it all fixed.
Try here: https://groups.google.com/g/bup-list/c/8f43Sq3OBTI

> suggestions:
> - check the integrity of the repository with bup fsck, git fsck

Both return 0, though git warns of various dangling trees, commits, and
blobs.

> - remove all cached index files from the client
> - remove all midx files (but see what others think, and set them
> aside
> vs losing them)

I don't think I know enough about bup to know how to do what you're
suggesting. I tried moving all the idx and midx files from
$BUP_DIR/objects/pack away, and rerunning `bup index` on all of the
directories I back up. After that, not even `bup ls` worked. I restored
to my copy of BUP_DIR from before I started screwing with it and now it
at least can ls again. :-]

I can try removing these things, but I need a bit more indication of
what to go looking for.

> - look at your ssh logs on the server and see if the connection ends
> normally or if there is an error
> - look at /var/log/messages or equivalent on both machines

I had to reboot the server yesterday, and it looks like the errors began
right after that. The reboot probably happened while bup was running. I
don't have any logs of the original error, though.

—
Nathaniel

[Nathaniel Hourt]

The command is `bup on server save -n dirname dirname`

—
Nathaniel

[Johannes Berg]

On Mon, 2025-11-17 at 21:57 -0600, 'Nathaniel' via bup-list wrote:
>
> I took a copy of my BUP_DIR and did a `bup index --clear` but that did not help

which side did you do that on?

It _does_ look like a bad index, but you'd have to run that on 'server'
or 'bup on server index --clear'.

johannes

[Rob Browning]

"'Nathaniel Hourt' via bup-list" <bup-...@googlegroups.com> writes:

> The command is `bup on server save -n dirname dirname`

So there's some confusion because bup works with two important things
commonly referred to as an "index" (ignoring the midx). There are the
git pack indexes (*.idx) and bup's filesystem "index" (currently, by
default BUP_DIR/bupindex). You almost never want to delete the former,
thought they can be manually regenerated from the packfiles via
git-index-pack(1).

The bup index has some known issues (that we plan to try to finish
*fixing* after we release 0.34). It's also completely ephemeral, so it's
fine to --clear it, though it *is* expensive to rebuild from scratch.

Interrupting bup with a reboot might have caused trouble, though of
course it shouldn't, but if everything was fine before that, we should
be able to fix your repo one way or another.

[Greg Troxel]

"'Nathaniel Hourt' via bup-list" <bup-...@googlegroups.com> writes:

>> - remove all cached index files from the client
>> - remove all midx files (but see what others think, and set them
>> aside
>> vs losing them)
>
> I don't think I know enough about bup to know how to do what you're
> suggesting. I tried moving all the idx and midx files from
> $BUP_DIR/objects/pack away, and rerunning `bup index` on all of the
> directories I back up. After that, not even `bup ls` worked. I
> restored to my copy of BUP_DIR from before I started screwing with it
> and now it at least can ls again. :-]

No, do not remove idx in pack dir!!

By client, I meant in ~/.bup/index-cache.

[Nathaniel Hourt]

Ha, yeah OK. That did it. =)

So now my repo is in good order again, but I have bad copies of both
client and server's bup dirs in case there are any diagnostics I should
run.

Many thanks!

—
Nathaniel

[Rob Browning]

"'Nathaniel Hourt' via bup-list" <bup-...@googlegroups.com> writes:

> Ha, yeah OK. That did it. =)

Excellent.

> So now my repo is in good order again, but I have bad copies of both
> client and server's bup dirs in case there are any diagnostics I should
> run.

Appreciate the offer, though I'm pretty confident we know what the issue
is. (It's probably the potential for skew between the main (mmaped)
index file and the related state in the metadata "db" and/or hard link
"db".)

I expect that in your case the sudden halt left the main mmapped file
with references to entries that didn't make it into the metadata db.

The current plan is to replace all three with real transactions (most
likely via sqlite).

Thanks