test failures on NetBSD

[Greg Troxel]

My last build that was ok:

==> CHECK.0.33-866-g2fc0ebb8.python3.13.UTC <==
========== 165 passed, 12 skipped, 1 deselected in 1691.87s (0:28:11) ==========

and just now a lot failed. Maybe this is a new test dependency not documented?


/usr/pkg/lib/python3.13/subprocess.py:577: CalledProcessError
----------------------------- Captured stderr call -----------------------------
Could not find platform independent libraries <prefix>
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x000078629b215800 (most recent call first):
<no Python frame>
--------------------------- Captured stderr teardown ---------------------------

Preserving: b'test/tmp/ext-test_ftp-py-test_ftpg7rrjnn1'

Preserving test HOME: b'/home/gdt/SOFTWARE/BUP/bup/test/tmp/home-g0iaiyek'
=========================== short test summary info ============================
FAILED test/ext/test-index:: - Exit status: 1
FAILED test/ext/test-index-check-device:: - Exit status: 1
FAILED test/ext/test-index-clear:: - Exit status: 1
FAILED test/ext/test-index-save-type-change:: - Exit status: 1
FAILED test/ext/test-init:: - Exit status: 1
FAILED test/ext/test-install:: - Exit status: 1
FAILED test/ext/test-list-idx:: - Exit status: 1
FAILED test/ext/test-ls:: - Exit status: 1
FAILED test/ext/test-ls-from-other-bup:: - Exit status: 1
FAILED test/ext/test-ls-remote:: - Exit status: 1
FAILED test/ext/test-main:: - Exit status: 1
FAILED test/ext/test-meta:: - Exit status: 1
FAILED test/ext/test-meta-acls:: - Exit status: 1
FAILED test/ext/test-misc:: - Exit status: 1
FAILED test/ext/test-on:: - Exit status: 1
FAILED test/ext/test-packsizelimit:: - Exit status: 1
FAILED test/ext/test-pyutil:: - Exit status: 1
FAILED test/ext/test-redundant-saves:: - Exit status: 1
FAILED test/ext/test-restore-map-owner:: - Exit status: 1
FAILED test/ext/test-restore-single-file:: - Exit status: 1
FAILED test/ext/test-rewrite:: - Exit status: 1
FAILED test/ext/test-rm:: - Exit status: 1
FAILED test/ext/test-rm-between-index-and-save:: - Exit status: 1
FAILED test/ext/test-save-creates-no-unrefs:: - Exit status: 1
FAILED test/ext/test-save-data-race:: - Exit status: 1
FAILED test/ext/test-save-errors:: - Exit status: 1
FAILED test/ext/test-save-restore:: - Exit status: 1
FAILED test/ext/test-save-restore-excludes:: - Exit status: 1
FAILED test/ext/test-save-smaller:: - Exit status: 1
FAILED test/ext/test-save-strip-graft:: - Exit status: 1
FAILED test/ext/test-save-symlink-race:: - Exit status: 1
FAILED test/ext/test-save-with-valid-parent:: - Exit status: 1
FAILED test/ext/test-sparse-files:: - Exit status: 1
FAILED test/ext/test-split-files-config:: - Exit status: 1
FAILED test/ext/test-split-join:: - Exit status: 1
FAILED test/ext/test-tag:: - Exit status: 1
FAILED test/ext/test-treesplit:: - Exit status: 1
FAILED test/ext/test-tz:: - Exit status: 1
FAILED test/ext/test-validate-object-links:: - Exit status: 1
FAILED test/ext/test-validate-ref-links:: - Exit status: 1
FAILED test/ext/test-validate-refs:: - Exit status: 138
FAILED test/ext/test-walk-object-order:: - Exit status: 138
FAILED test/ext/test-web:: - Exit status: 1
FAILED test/ext/test-xdev:: - Exit status: 1
FAILED test/ext/test_argv.py::test_argv - subprocess.CalledProcessError: Comm...
FAILED test/ext/test_ftp.py::test_ftp - subprocess.CalledProcessError: Comman...
===== 46 failed, 127 passed, 5 skipped, 1 deselected in 1401.71s (0:23:21) =====
gmake: *** [GNUmakefile:362: check] Error 1

[Rob Browning]

Greg Troxel <g...@lexort.com> writes:

> My last build that was ok:
>
> ==> CHECK.0.33-866-g2fc0ebb8.python3.13.UTC <==
> ========== 165 passed, 12 skipped, 1 deselected in 1691.87s (0:28:11) ==========
>
> and just now a lot failed. Maybe this is a new test dependency not documented?
>
>
> /usr/pkg/lib/python3.13/subprocess.py:577: CalledProcessError
> ----------------------------- Captured stderr call -----------------------------
> Could not find platform independent libraries <prefix>
> Fatal Python error: Failed to import encodings module
> Python runtime state: core initialized
> ModuleNotFoundError: No module named 'encodings'

Hmm, I don't see anywhere we import that directly. Wonder if
something's odd or changed with respect to the local system.

The other possibility might be that the changes I made when we raised
the minimum to 3.8 aren't OK on NetBSD.

If you want to test that, you could try before/after

cf087f523e7148df6d6771ca7c5b65d4e98db054
Raise minimum python from 3.7 to 3.8

Thanks
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Greg Troxel]

Rob Browning <r...@defaultvalue.org> writes:

> Greg Troxel <g...@lexort.com> writes:
>
>> My last build that was ok:
>>
>> ==> CHECK.0.33-866-g2fc0ebb8.python3.13.UTC <==
>> ========== 165 passed, 12 skipped, 1 deselected in 1691.87s (0:28:11) ==========
>>
>> and just now a lot failed. Maybe this is a new test dependency not documented?
>>
>>
>> /usr/pkg/lib/python3.13/subprocess.py:577: CalledProcessError
>> ----------------------------- Captured stderr call -----------------------------
>> Could not find platform independent libraries <prefix>
>> Fatal Python error: Failed to import encodings module
>> Python runtime state: core initialized
>> ModuleNotFoundError: No module named 'encodings'
>
> Hmm, I don't see anywhere we import that directly. Wonder if
> something's odd or changed with respect to the local system.

I have just upgraded a bunch of things, but bisecting is looking like
it's not my system. I'm just not 100% sure yet.

> The other possibility might be that the changes I made when we raised
> the minimum to 3.8 aren't OK on NetBSD.
>
> If you want to test that, you could try before/after
>
> cf087f523e7148df6d6771ca7c5b65d4e98db054
> Raise minimum python from 3.7 to 3.8

One commit later it's good

$ tail -1 CHECK.0.33-864-gbbc619a9.python3.13.UTC
========== 165 passed, 12 skipped, 1 deselected in 1618.29s (0:26:58) ==========

(just because that test is sitting there already).




I had been bisecting. Just finished:

$ git bisect good
1b4976f9fc100f12b8b1a34d1a7e96e5fcdd4116 is the first 'bad' commit
commit 1b4976f9fc100f12b8b1a34d1a7e96e5fcdd4116 (upstream/main, upstream/HEAD, github-upstream/master, github-upstream/main, github-upstream/HEAD, main)
Author: Rob Browning <r...@defaultvalue.org>
Date: Tue Jul 7 16:28:45 2026 -0500

RemoteRepo._config_get_fallback: remove unnecessary return (pylint)

Signed-off-by: Rob Browning <r...@defaultvalue.org>
Tested-by: Rob Browning <r...@defaultvalue.org>

lib/bup/repo/remote.py | 1 -
1 file changed, 1 deletion(-)


with status by hash

==> CHECK.0.33-855-g3d6869ea.python3.13.UTC <==
========== 165 passed, 12 skipped, 1 deselected in 1561.12s (0:26:01) ==========

==> CHECK.0.33-864-gbbc619a9.python3.13.UTC <==
========== 165 passed, 12 skipped, 1 deselected in 1618.29s (0:26:58) ==========

==> CHECK.0.33-866-g2fc0ebb8.python3.13.UTC <==
========== 165 passed, 12 skipped, 1 deselected in 1691.87s (0:28:11) ==========

==> CHECK.0.33-869-geef904c0.python3.13.UTC <==
========== 165 passed, 12 skipped, 1 deselected in 2088.47s (0:34:48) ==========

==> CHECK.0.33-871-gc507fab9.python3.13.UTC <==
===== 1 failed, 163 passed, 14 skipped, 1 deselected in 1967.92s (0:32:47) =====

==> CHECK.0.33-872-g1b4976f9.python3.13.UTC <==

===== 46 failed, 127 passed, 5 skipped, 1 deselected in 1401.71s (0:23:21) =====

I called 871 good, since it seemed a minor and likely separate issue.


This points to removing the unnecessary return. I am having a hard
time following.


I have reverted 1b4976f9fc100f12b8b1a34d1a7e96e5fcdd4116 locally and am
re-running tests. Yes, I realized that's like checking out
1b4976f9fc100f12b8b1a34d1a7e96e5fcdd4116^.

[Greg Troxel]

I don't understand what's going on, but
1b4976f9fc100f12b8b1a34d1a7e96e5fcdd4116 causes 45 test failures.

(The 1 failed before and after is from an earlier change, that I also
haven't tracked down.)


==> CHECK.0.33-871-gc507fab9.python3.13.UTC <==

FAILED test/ext/test-validate-refs:: - Exit status: 1

===== 1 failed, 163 passed, 14 skipped, 1 deselected in 1967.92s (0:32:47) =====

gmake: *** [GNUmakefile:362: check] Error 1

==> CHECK.0.33-872-g1b4976f9.python3.13.UTC <==

FAILED test/ext/test_ftp.py::test_ftp - subprocess.CalledProcessError: Comman...
===== 46 failed, 127 passed, 5 skipped, 1 deselected in 1401.71s (0:23:21) =====
gmake: *** [GNUmakefile:362: check] Error 1

==> CHECK.0.33-873-g49fbed18.python3.13.UTC <==

FAILED test/ext/test-validate-refs:: - Exit status: 1

===== 1 failed, 163 passed, 14 skipped, 1 deselected in 1747.65s (0:29:07) =====

gmake: *** [GNUmakefile:362: check] Error 1

$ git log -p -1
commit 49fbed183b5ffd593dbb06d5c3d0c97f1b4cac45 (HEAD -> main)
Author: Greg Troxel <g...@lexort.com>
Date: Thu Jul 9 20:19:24 2026 -0400

Revert "RemoteRepo._config_get_fallback: remove unnecessary return (pylint)"

This reverts commit 1b4976f9fc100f12b8b1a34d1a7e96e5fcdd4116.

diff --git a/lib/bup/repo/remote.py b/lib/bup/repo/remote.py
index f4594810..46ce34bb 100644
--- a/lib/bup/repo/remote.py
+++ b/lib/bup/repo/remote.py
@@ -62,6 +62,7 @@ class RemoteRepo(RepoProtocol):
assert opttype in ('int', 'bool', None)
if name not in valid_config_opts:
raise PermissionError(f'remote access to {name} is not allowed')
+ return None

def update_ref(self, refname, newval, oldval):
self.finish_writing()

[Rob Browning]

Greg Troxel <g...@lexort.com> writes:

> I don't understand what's going on, but
> 1b4976f9fc100f12b8b1a34d1a7e96e5fcdd4116 causes 45 test failures.
>
> (The 1 failed before and after is from an earlier change, that I also
> haven't tracked down.)

Is there any more output than what you had in your first message?
i.e. if you run one of the failing tests via

./pytest -s test/ext/...

Can you see anything else potentially interesting?

[Greg Troxel]

Rob Browning <r...@defaultvalue.org> writes:

> Greg Troxel <g...@lexort.com> writes:
>
>> I don't understand what's going on, but
>> 1b4976f9fc100f12b8b1a34d1a7e96e5fcdd4116 causes 45 test failures.
>>
>> (The 1 failed before and after is from an earlier change, that I also
>> haven't tracked down.)
>
> Is there any more output than what you had in your first message?
> i.e. if you run one of the failing tests via
>
> ./pytest -s test/ext/...
>
> Can you see anything else potentially interesting?

That seems to use installed bup, not git repo bup. I'll let another
run finish and look at the more detailed logs.

I'll also check a test run w/o bup installed. It may turn out the real
bug is the tests using system bup.

[Rob Browning]

Greg Troxel <g...@lexort.com> writes:

> That seems to use installed bup, not git repo bup. I'll let another
> run finish and look at the more detailed logs.

Depending on what you mean, the tests nearly always use ./bup,
i.e. anything invoked by "./pytest ..." or "make check", etc.

Though we do (now) have some testing against "some other bup"
(e.g. test-comparative-split-join) whenever BUP_TEST_OTHER_BUP is set:

BUP_TEST_OTHER_BUP=/usr/bin/bup ./pytest test/ext/test-comparative-split-join

[Greg Troxel]

Greg Troxel <g...@lexort.com> writes:

> I don't understand what's going on, but
> 1b4976f9fc100f12b8b1a34d1a7e96e5fcdd4116 causes 45 test failures.

The plot thickens. I ran make clean, and then built the commit that
failed earlier, and it passed. I wonder if there is a makefile dep
bug. Will keep trying to figure this out.

> (The 1 failed before and after is from an earlier change, that I also
> haven't tracked down.)

I figured this one out. It's a test bug, but hard to deal with.

$ touch foo
$ ls -l foo
-rw-r--r-- 1 gdt wheel 0 Jul 10 19:08 foo
$ du foo
1 foo

This is on zfs. On normal NetBSD filesystems, I get 0. du has an
ambiguous spec, and there's bits taken up with inodes.

Reading the test:

WVPASS err-to log bup validate-refs --links
WVPASS test "$(du log | cut -f1)" -eq 0

I think this is not about disk space at all. I think it's "Is this file
empty".

I wonder if one can use -s (file exists and is > 0) with !, or cmp
/dev/null, rather than relying on precise du output.

du is not specified by POSIX, and dates from Version 1! But it's "how
many blocks of storage does the filesystem say are being used", and I
don't think "0 bytes use 0 blocks" is axiomatic.


With this diff, I get a pass:

diff --git a/test/ext/test-validate-refs b/test/ext/test-validate-refs
index 11980c32..dd180d92 100755
--- a/test/ext/test-validate-refs
+++ b/test/ext/test-validate-refs
@@ -28,12 +28,13 @@ WVPASS touch src/a/{1,2,3}
WVPASS "$top/dev/make-splittable-tree" src/split-tree
WVPASS bup index src
WVPASS bup save --strip -n src src
+# Ensure output is empty.
WVPASS err-to log bup validate-refs --links
-WVPASS test "$(du log | cut -f1)" -eq 0
+WVPASS test ! -s log
WVPASS err-to log bup validate-refs --bupm
-WVPASS test "$(du log | cut -f1)" -eq 0
+WVPASS test ! -s log
WVPASS err-to log bup validate-refs
-WVPASS test "$(du log | cut -f1)" -eq 0
+WVPASS test ! -s log
WVPASS rm -rf src bup

WVSTART 'detection of abridged bupms'

[Rob Browning]

Greg Troxel <g...@lexort.com> writes:

> du is not specified by POSIX, and dates from Version 1! But it's "how
> many blocks of storage does the filesystem say are being used", and I
> don't think "0 bytes use 0 blocks" is axiomatic.

Right, I'd just been lazy. We'd need something like the non-standard
--apparent-size (or --bytes) option for du to work.

...and I'd also been hearing a vague bell ringing here for a bit, and
sure enough, there's dev/data-size. I'd also looked at test, but
overlooked the -s option, which as you say, would also work.

> With this diff, I get a pass:

OK, thanks. I'll fix it.

[Greg Troxel]

Greg Troxel <g...@lexort.com> writes:

> and just now a lot failed. Maybe this is a new test dependency not documented?
>
>
> /usr/pkg/lib/python3.13/subprocess.py:577: CalledProcessError
> ----------------------------- Captured stderr call -----------------------------
> Could not find platform independent libraries <prefix>
> Fatal Python error: Failed to import encodings module
> Python runtime state: core initialized
> ModuleNotFoundError: No module named 'encodings'
>
> Current thread 0x000078629b215800 (most recent call first):
> <no Python frame>
> --------------------------- Captured stderr teardown ---------------------------

After a bunch of bisecting and cleaning and rerunning I cannot reproduce
this. So either I'm confused (and there is or is not a test or real
bug, or I had bad state somehow (because of something about bup or about
my system), or something else!

But seriously, I cannot reproduce and I withdraw the report. Please
don't worry about my problem, and I'll repost if I come up with a
reproducible coherent problem

[Rob Browning]

Greg Troxel <g...@lexort.com> writes:

> After a bunch of bisecting and cleaning and rerunning I cannot reproduce
> this. So either I'm confused (and there is or is not a test or real
> bug, or I had bad state somehow (because of something about bup or about
> my system), or something else!
>
> But seriously, I cannot reproduce and I withdraw the report. Please
> don't worry about my problem, and I'll repost if I come up with a
> reproducible coherent problem

Sounds good, and thanks for the investigation.