Bup crashes on CIFS remote storage

[Martin van Es]

Hi,

I recently upgrdaded a test machine to ubuntu 24.04 and discovered that my kde kup-backup backups wouldn't backup anymore. Kup-backup is a graphical frontend to bup, when configured to create versioned backups.

After some testing, it turns out that the main difference with my stable machine that still works, was a local built bup version 0.32 vs 0.33.3-1build2 from packages in 24.04.

It turns out the problem only exists when I use a remote CIFS filesystem on my NAS, running samba version 4.15.13. And is reproducable on the stable machine when switching to 0.33.2-1 from packages on that machine (still 23.10).

So, long story short: 0.33.[2|3] doesn't play nice with my remote (LAN connected) CIFS mountpoint. The error I get in kup's logs is:

Kup is starting bup backup job at Tuesday, May 14, 2024 5:34:16 PM CEST

bup "-d" "/mnt/kronos/Martin/Backup/microvanes/Backup" "init"
bup "-d" "/mnt/kronos/Martin/Backup/microvanes/Backup" "index" "-u" "--exclude-rx-from" "/home/martin/.config/bup/exclude_patterns" "/home/martin/.config"
Traceback (most recent call last):
 File "<frozen runpy>", line 198, in _run_module_as_main
 File "<frozen runpy>", line 88, in _run_code
 File "/usr/lib/bup/bup/main.py", line 394, in <module>
   main()
 File "/usr/lib/bup/bup/main.py", line 391, in main
   wrap_main(lambda : run_subcmd(cmd_module, subcmd))
 File "/usr/lib/bup/bup/compat.py", line 82, in wrap_main
   sys.exit(main())
            ^^^^^^
 File "/usr/lib/bup/bup/main.py", line 391, in <lambda>
   wrap_main(lambda : run_subcmd(cmd_module, subcmd))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/lib/bup/bup/main.py", line 386, in run_subcmd
   run_module_cmd(module, args)
 File "/usr/lib/bup/bup/main.py", line 287, in run_module_cmd
   import_and_run_main(module, args)
 File "/usr/lib/bup/bup/main.py", line 282, in import_and_run_main
   module.main(args)
 File "/usr/lib/bup/bup/cmd/index.py", line 283, in main
   update_index(rp, excluded_paths, exclude_rxs, indexfile,
 File "/usr/lib/bup/bup/cmd/index.py", line 196, in update_index
   mi.close()
 File "/usr/lib/bup/bup/index.py", line 586, in close
   with self.cleanup:
 File "/usr/lib/python3.12/contextlib.py", line 610, in __exit__
   raise exc_details[1]
 File "/usr/lib/python3.12/contextlib.py", line 595, in __exit__
   if cb(*exc_details):
      ^^^^^^^^^^^^^^^^
 File "/usr/lib/bup/bup/helpers.py", line 772, in __exit__
   os.rename(self.tmp_path, self.path)
PermissionError: [Errno 13] Permission denied: b'/mnt/kronos/Martin/Backup/microvanes/Backup/bupindex-gc8hvrzu/pending' -> b'/mnt/kronos/Martin/Backup/microvanes/Backup/bupindex'

Exit code: 1

The error is triggered by adding a substantial number of files to backup (e.g. ~/.config/). It works fine if I only select one directory that contains only one test file.

So, I suspect some kind of race condition while renaming the tmp directory and cleanup that happens around that part of the code. The NAS is built on huge, but slow spinning disks, which may explain the encountered behaviour.

The mount options for the CIFS remote storage are:

//kronos/media/         /mnt/kronos     cifs    noauto,users,soft,intr,sec=none,noperm,file_mode=0644,dir_mode=0755,vers=3.0

Best regards,

Martin

[Greg Troxel]

I am chasing a bug that may be related.

NetBSD 9 and 10
bup 0.32 upgrade to 0.33
bup save to -r gdt@foo:/b0/bup-bar

Mostly things work but at last one combination fails.

I haven't posted because I haven't done adequate testing yet.

[Rob Browning]

Martin van Es <mrv...@gmail.com> writes:

> So, I suspect some kind of race condition while renaming the tmp
> directory and cleanup that happens around that part of the code. The
> NAS is built on huge, but slow spinning disks, which may explain the
> encountered behaviour.

One completely offhand (related) guess (without looking) -- maybe we're
not explicitly closing the file, gc isn't handling that for us soon
enough, and cifs cares (linux normally doesn't).

I suppose you might be able to test that with a small python program,
i.e. open a file on the mount, then try to os.rename() over it without
closing the stream.

--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Martin van Es]

I tried, but I can't reproduce with this little snippet (and, kup also doesn't crash on the single-dir/single-file testcase!)

So I expect a little more is going on during cleanup phase and many files?

#!/usr/bin/env python
import os

tmp_dir = b'/mnt/kronos/Martin/foo-123_456'
tmp_path = tmp_dir + b'pending'
path = b'/mnt/kronos/Martin/foo'

os.makedirs(tmp_dir)
f = open(tmp_path, mode='w', buffering=-1)
f.write('test')

os.rename(tmp_path, path)
os.rmdir(tmp_dir)
os.remove(path)

--

If 'but' was any useful, it would be a logic operator

[Rob Browning]

Martin van Es <mrv...@gmail.com> writes:

> I tried, but I can't reproduce with this little snippet (and, kup also
> doesn't crash on the single-dir/single-file testcase!)
> So I expect a little more is going on during cleanup phase and many files?
>
> #!/usr/bin/env python
> import os
>
> tmp_dir = b'/mnt/kronos/Martin/foo-123_456'
> tmp_path = tmp_dir + b'pending'
> path = b'/mnt/kronos/Martin/foo'
>
> os.makedirs(tmp_dir)
> f = open(tmp_path, mode='w', buffering=-1)
> f.write('test')
>
> os.rename(tmp_path, path)
> os.rmdir(tmp_dir)
> os.remove(path)

Yeah, worth a try, but of course, not that easy...

I think I had set up a cifs sandbox at some point, a while ago, but
don't have it right now. Of course, if you happen to come up with a way
we can reproduce the issue, that could well help resolve the problem
faster.