Call for testing for a 0.33 release

[Rob Browning]

Quite a bit has changed since 0.32, and before we start merging some of
Johannes' even more substantial changes, let's release what we have.

https://github.com/bup/bup/tree/master
https://salsa.debian.org/rlb/bup/-/commits/master

Among other things, this release will be notable in that it's the first
one without support for python 2. At the moment, we don't enforce the
limit on the minor version (though I've been considering adding that),
but we do assume/require 3.7+.

Also be aware that the configuration process has changed (along with the
change of bup from a #! script to a binary) to require/respect
BUP_PYTHON_CONFIG rather than PYTHON, if you need to select a particular
version, e.g.:

BUP_PYTHON_CONFIG=/usr/bin/python3.9-config ./configure

instead of

PYTHON=/usr/bin/python3.9 ./configure

Thanks
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Nix]

On 9 Jul 2022, Rob Browning said:

> Quite a bit has changed since 0.32, and before we start merging some of
> Johannes' even more substantial changes, let's release what we have.
>
> https://github.com/bup/bup/tree/master
> https://salsa.debian.org/rlb/bup/-/commits/master
>
> Among other things, this release will be notable in that it's the first
> one without support for python 2. At the moment, we don't enforce the
> limit on the minor version (though I've been considering adding that),
> but we do assume/require 3.7+.
>
> Also be aware that the configuration process has changed (along with the
> change of bup from a #! script to a binary) to require/respect
> BUP_PYTHON_CONFIG rather than PYTHON, if you need to select a particular
> version, e.g.:
>
> BUP_PYTHON_CONFIG=/usr/bin/python3.9-config ./configure
>
> instead of
>
> PYTHON=/usr/bin/python3.9 ./configure

OK so I just tried this with commit
36fce270ed41330d1cf4ada5dda17c1b816f677d (HEAD of master right now). I
see pylint failures (with Python 3.10.4 and pylint 2.12.2, atop xfs:
upgrading pytest to 2.14 drops me into a nightmare of python packaging
problems involving an inability to find the packaging package, *even
when trying to install the packaging package*, which I don't have the
time or inclination to track down right now):

************* Module lib.bup.helpers
lib/bup/helpers.py:859:19: R1714: Consider merging these comparisons with "in" to 'ex.errno in (errno.EINVAL, errno.ENODEV)' (consider-using-in)
************* Module lib.bup.metadata
lib/bup/metadata.py:576:17: R1714: Consider merging these comparisons with "in" to 'e.errno in (errno.EPERM, errno.EOPNOTSUPP)' (consider-using-in)
lib/bup/metadata.py:698:23: R1714: Consider merging these comparisons with "in" to 'e.errno in (errno.EPERM, errno.EOPNOTSUPP)' (consider-using-in)

This halts make check. Doing as suggested lets the actual tests run.
Eventually test-save-restore-excludes fails:

Testing "index --exclude-rx '/foo/.' (dir content)" in /usr/src/bup/foo/test/ext/test-save-restore-excludes:
! /usr/src/bup/foo/test/ext/test-save-restore-excludes:144 rm -rf src /usr/src/bup/foo/test/tmp/test-save-restore-excludes-CmFe40C/bup buprestore.tmp ok
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
Initialized empty shared Git repository in /usr/src/bup/foo/test/tmp/test-save-restore-excludes-CmFe40C/bup/
! /usr/src/bup/foo/test/ext/test-save-restore-excludes:145 bup init ok
! /usr/src/bup/foo/test/ext/test-save-restore-excludes:146 mkdir src ok
! /usr/src/bup/foo/test/ext/test-save-restore-excludes:147 touch src/a ok
! /usr/src/bup/foo/test/ext/test-save-restore-excludes:148 touch src/b ok
! /usr/src/bup/foo/test/ext/test-save-restore-excludes:149 touch src/foo ok
! /usr/src/bup/foo/test/ext/test-save-restore-excludes:150 mkdir src/sub ok
! /usr/src/bup/foo/test/ext/test-save-restore-excludes:151 mkdir src/sub/foo ok
! /usr/src/bup/foo/test/ext/test-save-restore-excludes:152 touch src/sub/foo/a ok
! /usr/src/bup/foo/test/ext/test-save-restore-excludes:153 bup index -u src --exclude-rx /foo/. ok
! /usr/src/bup/foo/test/ext/test-save-restore-excludes:154 bup save --strip -n bupdir src ok
! /usr/src/bup/foo/test/ext/test-save-restore-excludes:155 bup restore -C buprestore.tmp /bupdir/latest/ ok
! /usr/src/bup/foo/test/ext/test-save-restore-excludes:156 cd buprestore.tmp ok
! /usr/src/bup/foo/test/ext/test-save-restore-excludes:156 find . ok
! /usr/src/bup/foo/test/ext/test-save-restore-excludes:156 sort ok
! /usr/src/bup/foo/test/ext/test-save-restore-excludes:157 exactly 2 arguments ok
Comparing:
.
--
.
./a
./b
./foo
./sub
./sub/foo
! /usr/src/bup/foo/test/ext/test-save-restore-excludes:157 '.' = '. ./a ./b ./foo ./sub ./sub/foo' FAILED
called from /usr/src/bup/foo/test/ext/test-save-restore-excludes:157 WVPASSEQ . .
./a
./b
./foo
./sub
./sub/foo
===================================================================================================== short test summary info =====================================================================================================
FAILED test/ext/test-save-restore-excludes::
=============================================================================== 1 failed, 135 passed, 7 skipped, 1 deselected in 1627.89s (0:27:07) ===============================================================================

My locale (except for LC_COLLATE) is en_GB, but forcing LANG and
LANGUAGE to C doesn't help.

[Rob Browning]

Nix <n...@esperi.org.uk> writes:

> OK so I just tried this with commit
> 36fce270ed41330d1cf4ada5dda17c1b816f677d (HEAD of master right now). I
> see pylint failures (with Python 3.10.4 and pylint 2.12.2, atop xfs:
> upgrading pytest to 2.14 drops me into a nightmare of python packaging
> problems involving an inability to find the packaging package, *even
> when trying to install the packaging package*, which I don't have the
> time or inclination to track down right now):

Thanks -- I'll see about the pylint issues (looks like I'm on (debian)
2.7.2-3 currently). If this is often an issue, imagine we might want to
add some (easier) way to suppress the pylint test.

> ! /usr/src/bup/foo/test/ext/test-save-restore-excludes:157 exactly 2 arguments ok
> Comparing:
> .
> --
> .
> ./a
> ./b
> ./foo
> ./sub
> ./sub/foo
> ! /usr/src/bup/foo/test/ext/test-save-restore-excludes:157 '.' = '. ./a ./b ./foo ./sub ./sub/foo' FAILED
> called from /usr/src/bup/foo/test/ext/test-save-restore-excludes:157 WVPASSEQ . .
> ./a
> ./b
> ./foo
> ./sub
> ./sub/foo

That's odd. May not be relevant, but what's the platform, etc.?

> My locale (except for LC_COLLATE) is en_GB, but forcing LANG and
> LANGUAGE to C doesn't help.

Thanks, though of course if that ended up mattering, it'd almost
certainly be a bug.

(I also noticed xfs -- that shouldn't matter either, but might be nice
to add that to our test suite, particularly if it's easy. I might take
a look later.)

[Nix]

On 24 Jul 2022, Rob Browning stated:

Homebrewed from-scratch box containing Python 3.10.4; Linux 5.16.19
(only because 5.18.12 is failing to boot for me right now); glibc 2.33;
GCC 11; fs is XFS atop libata->md-raid6->bcache (but if the blockdev
stack matters I'll be utterly astonished). XFS is a vaguely modernish v5
fs, with reflink support and 64-bit inodes (because it's a
multi-terabyte FS and you must use 64-bit inodes if you want that to
have non-awful performance with XFS).

--
NULL && (void)

[Rob Browning]

Nix <n...@esperi.org.uk> writes:

> Homebrewed from-scratch box containing Python 3.10.4; Linux 5.16.19
> (only because 5.18.12 is failing to boot for me right now); glibc 2.33;
> GCC 11; fs is XFS atop libata->md-raid6->bcache (but if the blockdev
> stack matters I'll be utterly astonished).

OK, thanks.

> XFS is a vaguely modernish v5 fs

Oh, I just meant that I noticed you mentioned xfs, and I was just
suggesting that since afaik, it's one of the other "main" linux
filesystems, it might be nice to have some CI testing there eventually.

[Rob Browning]

Nix <n...@esperi.org.uk> writes:

> ! /usr/src/bup/foo/test/ext/test-save-restore-excludes:155 bup restore -C buprestore.tmp /bupdir/latest/ ok
> ! /usr/src/bup/foo/test/ext/test-save-restore-excludes:156 cd buprestore.tmp ok
> ! /usr/src/bup/foo/test/ext/test-save-restore-excludes:156 find . ok
> ! /usr/src/bup/foo/test/ext/test-save-restore-excludes:156 sort ok
> ! /usr/src/bup/foo/test/ext/test-save-restore-excludes:157 exactly 2 arguments ok
> Comparing:
> .
> --
> .
> ./a
> ./b
> ./foo
> ./sub
> ./sub/foo
> ! /usr/src/bup/foo/test/ext/test-save-restore-excludes:157 '.' = '. ./a ./b ./foo ./sub ./sub/foo' FAILED
> called from /usr/src/bup/foo/test/ext/test-save-restore-excludes:157 WVPASSEQ . .
> ./a
> ./b
> ./foo
> ./sub
> ./sub/foo

Oh, and if you like, you should be able to experiment with that test
fairly easily. You can run it directly:

$ test/ext/test-svae-restore-excludes

or via pytest

$ ./pytest test/ext/test-svae-restore-excludes

and pytest's -s will show all the output if desired.

Stepping back, that's a surprising failure, i.e. find said there was
nothing in the restore, unless find is behaving unusually, or the rx
matching is somehow different for you, or...

You could also put a "bash" invocation just before the "actual=..." line
and poke around in buprestore.tmp, check the repo with "bup ls
/bupdir/latest" etc.

And while "bup ls ..." will show what's in the tree in the repository,
you can also find the exclusion in restore.py near the top of restore(),
e.g.:

if should_rx_exclude_path(fullname + (b'/' if treeish else b''),
exclude_rxs):
return

...if you want to add some print statements to see what it's doing or
similar.

Hope this helps

[Nix]

On 24 Jul 2022, Rob Browning outgrape:

> Nix <n...@esperi.org.uk> writes:
>
>> Homebrewed from-scratch box containing Python 3.10.4; Linux 5.16.19
>> (only because 5.18.12 is failing to boot for me right now); glibc 2.33;
>> GCC 11; fs is XFS atop libata->md-raid6->bcache (but if the blockdev
>> stack matters I'll be utterly astonished).
>
> OK, thanks.
>
>> XFS is a vaguely modernish v5 fs
>
> Oh, I just meant that I noticed you mentioned xfs, and I was just
> suggesting that since afaik, it's one of the other "main" linux
> filesystems, it might be nice to have some CI testing there eventually.

Oh definitely. If it breaks badly I'll definitely let you know, anyway :)

--
NULL && (void)

[Rob Browning]

Nix <n...@esperi.org.uk> writes:

> ************* Module lib.bup.helpers
> lib/bup/helpers.py:859:19: R1714: Consider merging these comparisons with "in" to 'ex.errno in (errno.EINVAL, errno.ENODEV)' (consider-using-in)
> ************* Module lib.bup.metadata
> lib/bup/metadata.py:576:17: R1714: Consider merging these comparisons with "in" to 'e.errno in (errno.EPERM, errno.EOPNOTSUPP)' (consider-using-in)
> lib/bup/metadata.py:698:23: R1714: Consider merging these comparisons with "in" to 'e.errno in (errno.EPERM, errno.EOPNOTSUPP)' (consider-using-in)

Just pushed a fix for this.

[Rob Browning]

Rob Browning <r...@defaultvalue.org> writes:

> Oh, and if you like, you should be able to experiment with that test
> fairly easily. You can run it directly:
>
> $ test/ext/test-svae-restore-excludes
>
> or via pytest
>
> $ ./pytest test/ext/test-svae-restore-excludes
>
> and pytest's -s will show all the output if desired.
>
> Stepping back, that's a surprising failure, i.e. find said there was
> nothing in the restore, unless find is behaving unusually, or the rx
> matching is somehow different for you, or...

Just checking back to see where this ended up. Are you still having
trouble? Just wondering whether this should be considered release
critical.

Thanks

[Rob Browning]

Rob Browning <r...@defaultvalue.org> writes:

> Quite a bit has changed since 0.32, and before we start merging some of
> Johannes' even more substantial changes, let's release what we have.

OK, I think we may have addressed all the more acute issues, but if not,
please reply.

If nothing else comes up, I'll proceed with the relesase.

[Nix]

On 12 Sep 2022, Rob Browning uttered the following:

> Just checking back to see where this ended up. Are you still having
> trouble? Just wondering whether this should be considered release
> critical.

Sorry, absence on summer holidays and LPC intervened. I'll check this
weekend when I'm back near my systems.

--
NULL && (void)

[Rob Browning]

Nix <n...@esperi.org.uk> writes:

> Sorry, absence on summer holidays and LPC intervened. I'll check this
> weekend when I'm back near my systems.

Sounds good, and there's no rush.