Fix issues exposed by additional platforms; improve manpages

[Rob Browning]

These patches address a number of issues revealed when testing on some
additional platforms. For example, Ubuntu systems that had a default
posix1e ACL set on the parent directory, and a newer macOS system
which has stopped responding to C.en_US (now requires just C).

The patches also include manpage improvements based on recent
suggestions by and discussions with Greg Troxel.

Pushed to main.

--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Rob Browning]

Previously I believe the test LANG=C.UTF-8 setting was sufficient, but
macos 15 now sorts case insensitively with that setting (14 did not).
Just use LC_ALL=C as we do elsewhere:

b7d094edf89cd34c14344cc37e413b4a1d6d64be
test.sh: set LC_ALL=C for git fsck to avoid localized output

b75d9969d1d5c1c5fe52ae690b632cb60a34495b
test-rm: sort compare-trees output with LC_ALL=C

...

Signed-off-by: Rob Browning <r...@defaultvalue.org>
Tested-by: Rob Browning <r...@defaultvalue.org>
---
test/ext/test-save-restore | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/ext/test-save-restore b/test/ext/test-save-restore
index a47f8284..4da39970 100755
--- a/test/ext/test-save-restore
+++ b/test/ext/test-save-restore
@@ -191,7 +191,7 @@ WVSTART "save disjoint top-level directories"

# For now, assume that "ls -a" and "sort" use the same order.
actual="$(WVPASS bup ls -AF src/latest)" || exit $?
- expected="$(echo -e "$pwd_top/\n$tmp_top/" | WVPASS sort)" || exit $?
+ expected="$(echo -e "$pwd_top/\n$tmp_top/" | LC_ALL=C sort)" || exit $?
WVPASSEQ "$actual" "$expected"
) || exit $?

--
2.47.3

[Rob Browning]

This test failed on systems where the test dir had a default posix1e
ACL because it didn't survive the round trip through the loopback
mounts we create.

Since this test is currently Linux only, for now, just explicitly
create ext4 filesystems with extended attributes enabled to avoid the
problem.

And invoke mount via WVEXPRC so we'll see the command in the test
output with file/line number, etc.

Signed-off-by: Rob Browning <r...@defaultvalue.org>
Tested-by: Rob Browning <r...@defaultvalue.org>
---

test/ext/test-xdev | 15 +++++++--------
1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/test/ext/test-xdev b/test/ext/test-xdev
index 52619e59..00812439 100755
--- a/test/ext/test-xdev
+++ b/test/ext/test-xdev
@@ -32,20 +32,19 @@ WVSTART 'drecurse'

WVPASS dd if=/dev/zero of=testfs-1.img bs=1M count=32
WVPASS dd if=/dev/zero of=testfs-2.img bs=1M count=32
-WVPASS mkfs -F testfs-1.img # Don't care what type (though must have symlinks)
-WVPASS mkfs -F testfs-2.img # Don't care what type (though must have symlinks)
+
+# Just assume ext4 for now
+WVPASS mkfs -t ext4 -F testfs-1.img
+WVPASS mkfs -t ext4 -F testfs-2.img
WVPASS mkdir -p src/mnt-1/hidden-1 src/mnt-2/hidden-2

-set -x
-mount -o loop testfs-1.img src/mnt-1
-rc=$?
-set +x
-if test "$rc" -ne 0; then
+WVEXPRC '*' mount -o loop,user_xattr testfs-1.img src/mnt-1
+if test "$?" -ne 0; then
WVSKIP 'Unable to mount via loopback'
exit 0
fi

-WVPASS mount -o loop testfs-2.img src/mnt-2
+WVPASS mount -o loop,ext_attr,user_xattr testfs-2.img src/mnt-2

WVPASS touch src/1

--
2.47.3

[Rob Browning]

Assuming the ordering is irrelevant, this avoids spurious differences
in, for example, the xstat output, which was causing some test
failures.

Signed-off-by: Rob Browning <r...@defaultvalue.org>
Tested-by: Rob Browning <r...@defaultvalue.org>
---

lib/bup/metadata.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/bup/metadata.py b/lib/bup/metadata.py
index 8903fc80..99d34101 100644
--- a/lib/bup/metadata.py
+++ b/lib/bup/metadata.py
@@ -1191,7 +1191,9 @@ def detailed_bytes(meta, fields = None):
if 'linux-attr' in fields and meta.linux_attr:
result.append(b'linux-attr: %x' % meta.linux_attr)
if 'linux-xattr' in fields and meta.linux_xattr:
- for name, value in meta.linux_xattr:
+ xats = meta.linux_xattr.copy()
+ xats.sort(key=lambda x: x[0])
+ for name, value in xats:
result.append(b'linux-xattr: %s -> %s' % (enc_sh(name), enc_sh(value)))
if 'posix1e-acl' in fields and meta.posix1e_acl:
acl = meta.posix1e_acl[0]
--
2.47.3

[Rob Browning]

Previous versions of bup did not fully respect the umask when creating
the bloom filter. Adjust WVEXPRC to return a matching exit status so
that we can use it to detect when compare-trees has found differences
while still including the compare-trees command in the test
output (with file and line number, etc.). Ignore the difference when
it's just the bup.bloom permissions. See also:

de397ec0fce03f7e01007fb8a4716d0eeac1a6b4
Respect umask when creating bloom filters

Signed-off-by: Rob Browning <r...@defaultvalue.org>
Tested-by: Rob Browning <r...@defaultvalue.org>
---

test/ext/test-comparative-split-join | 14 ++++++++++----
wvtest.sh | 12 +++++++++++-
2 files changed, 21 insertions(+), 5 deletions(-)

diff --git a/test/ext/test-comparative-split-join b/test/ext/test-comparative-split-join
index dbc6554d..b8b699cb 100755
--- a/test/ext/test-comparative-split-join
+++ b/test/ext/test-comparative-split-join
@@ -1,7 +1,8 @@
#!/usr/bin/env bash
-. wvtest.sh
-. wvtest-bup.sh
-. dev/lib.sh
+. wvtest.sh || exit $?
+. wvtest-bup.sh || exit $?
+. dev/lib.sh || exit $?
+. test/lib/btl.sh || exit $?

set -o pipefail

@@ -115,8 +116,13 @@ test-split-join()
# Exit should be 0 or 5 depending on whether that bup inits repo ids.
WVEXPRC '[05]' git config --file other-bup/config --unset bup.repo.id;
WVEXPRC '[05]' git config --file this-bup/config --unset bup.repo.id
- WVPASS "$top/dev/compare-trees" --no-times other-bup/ this-bup/

+ # Bup versions 0.33* and before may not fully respect the umask.
+ WVEXPRC '[01]' out-to out "$top/dev/compare-trees" --no-times \
+ other-bup/ this-bup/
+ if test "$?" -ne 0; then
+ WVPASSEQ '.f...p..... objects/pack/bup.bloom' "$(< out)"
+ fi
WVPASS cd "$orig_dir"
WVPASS rm -r split-join
}
diff --git a/wvtest.sh b/wvtest.sh
index 34ce9e69..c73736d7 100644
--- a/wvtest.sh
+++ b/wvtest.sh
@@ -52,6 +52,16 @@ _wvcheck()
}


+# Test for expected exit statuses:
+#
+# WVEXPRC '[01]' some command ...
+#
+# The first argument is a shell case pattern, and the test fails if
+# the command's exit status doesn't match it. If it does match, the
+# resulting $? will be that value. This also provides a way to ensure
+# a command ends up in the test output with line number regardless of
+# the exit status, i.e. WVEXPRC '*' some command ...
+#
WVEXPRC()
{
if test $# -lt 2; then
@@ -72,7 +82,7 @@ WVEXPRC()
$exp)
_wvcheck 0 "\$?=$rc matches $exp for $TEXT"
_wvpopcall
- return 0
+ return "$rc"
;;
*)
_wvcheck 1 "\$?=$rc matches $exp for $TEXT"
--
2.47.3

[Rob Browning]

Signed-off-by: Rob Browning <r...@defaultvalue.org>
Tested-by: Rob Browning <r...@defaultvalue.org>
---

lib/bup/_helpers.c | 37 +++++++++++++++++++------------------
1 file changed, 19 insertions(+), 18 deletions(-)

diff --git a/lib/bup/_helpers.c b/lib/bup/_helpers.c
index 9aa40101..c6a99785 100644
--- a/lib/bup/_helpers.c
+++ b/lib/bup/_helpers.c
@@ -1918,6 +1918,17 @@ static void test_integral_assignment_fits(void)
}
}

+static void setattr_or_die (PyObject *obj, const char *name, PyObject *val,
+ const char *msg)
+{
+ // Ensures reference to val is dropped
+ if (val == NULL || PyObject_SetAttrString(obj, name, val)) {
+ fputs(msg, stderr);
+ exit(BUP_EXIT_FAILURE);
+ }
+ Py_DECREF(val);
+}
+
static int setup_module(PyObject *m)
{
// FIXME: migrate these tests to configure, or at least don't
@@ -1950,17 +1961,6 @@ static int setup_module(PyObject *m)
}
}

- char *e;
- {
- PyObject *value;
- value = BUP_LONGISH_TO_PY(INT_MAX);
- PyObject_SetAttrString(m, "INT_MAX", value);
- Py_DECREF(value);
- value = BUP_LONGISH_TO_PY(UINT_MAX);
- PyObject_SetAttrString(m, "UINT_MAX", value);
- Py_DECREF(value);
- }
-
{
PyObject *math = PyImport_ImportModule("math");
if (!math) {
@@ -1979,16 +1979,17 @@ static int setup_module(PyObject *m)
}
}

+ setattr_or_die (m, "INT_MAX", BUP_LONGISH_TO_PY(INT_MAX),
+ "error: unable to define INT_MAX\n");
+ setattr_or_die (m, "UINT_MAX", BUP_LONGISH_TO_PY(UINT_MAX),
+ "error: unable to define UINT_MAX\n");
+
#ifdef BUP_HAVE_MINCORE_INCORE
- {
- PyObject *value;
- value = BUP_LONGISH_TO_PY(MINCORE_INCORE);
- PyObject_SetAttrString(m, "MINCORE_INCORE", value);
- Py_DECREF(value);
- }
+ setattr_or_die (m, "MINCORE_INCORE", BUP_LONGISH_TO_PY(MINCORE_INCORE),
+ "error: unable to define UINT_MAX\n");
#endif

- e = getenv("BUP_FORCE_TTY");
+ const char *e = getenv("BUP_FORCE_TTY");
get_state(m)->istty2 = isatty(2) || (atoi(e ? e : "0") & 2);
return 1;
}
--
2.47.3

[Rob Browning]

The errors were changed here

b60de4ac940f4a47d86cb441d185e7055601b46d
metadata: include path in all errors and switch to note_error

but apparently these tests weren't being properly exercised (e.g. in
the cirrus root tests).

Signed-off-by: Rob Browning <r...@defaultvalue.org>
Tested-by: Rob Browning <r...@defaultvalue.org>
---

test/ext/test-meta | 30 +++++++++++++++++-------------
1 file changed, 17 insertions(+), 13 deletions(-)

diff --git a/test/ext/test-meta b/test/ext/test-meta
index 8155858e..15df02ea 100755
--- a/test/ext/test-meta
+++ b/test/ext/test-meta
@@ -1,6 +1,7 @@
#!/usr/bin/env bash
. wvtest-bup.sh || exit $?

. dev/lib.sh || exit $?
+. test/lib/btl.sh || exit $?

set -o pipefail

@@ -738,10 +739,10 @@ if [ "$root_status" = root ]; then
WVPASS force-delete "$testfs_limited"/src-restore
WVPASS mkdir "$testfs_limited"/src-restore
WVPASS cd "$testfs_limited"/src-restore
- WVFAIL bup meta --extract --file "$testfs"/src.meta 2>&1 \
- | WVPASS grep -e '^Linux chattr:' \
- | WVPASS bup-cfg-py -c \
- 'import sys; exit(not len(sys.stdin.readlines()) == 3)'
+ WVFAIL err-to ../../err bup meta --extract --file "$testfs"/src.meta
+ WVPASS cd ../..
+ WVPASS grep -cE 'chattr.* failed' err > grep-count
+ WVPASSEQ 3 "$(< grep-count)"
) || exit $?
) || exit $?

@@ -761,11 +762,12 @@ if [ "$root_status" = root ]; then
WVPASS force-delete "$testfs_limited"/src-restore
WVPASS mkdir "$testfs_limited"/src-restore
WVPASS cd "$testfs_limited"/src-restore
- WVFAIL bup meta --extract --file "$testfs"/src.meta
- WVFAIL bup meta --extract --file "$testfs"/src.meta 2>&1 \
- | WVPASS grep -e "^xattr\.set u\?'" \
- | WVPASS bup-cfg-py -c \
- 'import sys; exit(not len(sys.stdin.readlines()) == 2)'
+ WVFAIL err-to ../../err bup meta --extract --file "$testfs"/src.meta
+ WVPASS cd ../../
+ WVPASS grep -cE 'xattr\.set .*src/(foo|bar/)$' err > grep-count
+ WVPASSEQ 2 "$(< grep-count)"
+ WVPASS grep -cF 'xattr.set ' err > grep-count
+ WVPASSEQ 2 "$(< grep-count)"
) || exit $?

WVSTART 'meta - POSIX.1e ACLs (as root)'
@@ -784,10 +786,12 @@ if [ "$root_status" = root ]; then
WVPASS force-delete "$testfs_limited"/src-restore
WVPASS mkdir "$testfs_limited"/src-restore
WVPASS cd "$testfs_limited"/src-restore
- WVFAIL bup meta --extract --file "$testfs"/src.meta 2>&1 \
- | WVPASS grep -e '^POSIX1e ACL applyto:' \
- | WVPASS bup-cfg-py -c \
- 'import sys; exit(not len(sys.stdin.readlines()) == 2)'
+ WVFAIL err-to ../../err bup meta --extract --file "$testfs"/src.meta
+ WVPASS cd ../../
+ WVPASS grep -cE 'POSIX1e ACL apply failed .*src/(foo|bar/)$' err > grep-count
+ WVPASSEQ 2 "$(< grep-count)"
+ WVPASS grep -cF 'POSIX1e ACL apply failed ' err > grep-count
+ WVPASSEQ 2 "$(< grep-count)"
) || exit $?

WVPASS umount "$testfs"
--
2.47.3

[Rob Browning]

Change setup_testfs and cleanup_testfs to require you to specify
"where", and adjust test_handling_of_incorrect_existing_linux_xattrs
accordingly, and to work in a tmpdir.

Add a finally clause to try to ensure we always cleanup the tempfs.

Signed-off-by: Rob Browning <r...@defaultvalue.org>
Tested-by: Rob Browning <r...@defaultvalue.org>
---

test/int/test_metadata.py | 69 ++++++++++++++++++++-------------------
1 file changed, 36 insertions(+), 33 deletions(-)

diff --git a/test/int/test_metadata.py b/test/int/test_metadata.py
index 6f958bfd..c076408c 100644
--- a/test/int/test_metadata.py
+++ b/test/int/test_metadata.py
@@ -1,5 +1,5 @@

-import errno, glob, stat, subprocess
+import errno, stat, subprocess
import os, sys
import pytest

@@ -25,15 +25,14 @@ def ex(*args, **kwargs):
return buptest.exc(args, **kwargs)


-def setup_testfs():
+def setup_testfs(img_path, mount_path, mb=32):
# Try to set up testfs with user_xattr, etc.
- assert(sys.platform.startswith('linux'))
- subprocess.call([b'umount', b'testfs'])
- ex(b'dd', b'if=/dev/zero', b'of=testfs.img', b'bs=1M', b'count=32')
- ex(b'mke2fs', b'-F', b'-j', b'-m', b'0', b'testfs.img')
- ex(b'rm', b'-rf', b'testfs')
- os.mkdir(b'testfs')
- exr = ex(b'mount', b'-o', b'loop,acl,user_xattr', b'testfs.img', b'testfs',
+ assert sys.platform.startswith('linux')
+ with open(img_path, 'xb') as img:
+ img.truncate(1024 * 1024 * mb)
+ ex(b'mke2fs', b'-F', b'-j', b'-m', b'0', img_path)
+ os.mkdir(mount_path)
+ exr = ex(b'mount', b'-o', b'loop,acl,user_xattr', img_path, mount_path,
check=False)
if exr.rc != 0:
return False
@@ -43,9 +42,9 @@ def setup_testfs():
return True


-def cleanup_testfs():
- subprocess.call([b'umount', b'testfs'])
- helpers.unlink(b'testfs.img')
+def cleanup_testfs(img_path, mount_path):
+ subprocess.call((b'umount', mount_path))
+ helpers.unlink(img_path)


def test_clean_up_archive_path():
@@ -255,29 +254,33 @@ if xattr:
return list(filter(lambda i: not i in (b'security.selinux', ),
attrs))

- def test_handling_of_incorrect_existing_linux_xattrs():
+ def test_handling_of_incorrect_existing_linux_xattrs(tmpdir):
+ if not sys.platform.startswith('linux'):
+ pytest.skip('skipping test -- not linux')
+ return
if not is_superuser() or detect_fakeroot():
pytest.skip('skipping test -- not superuser')
return
- if not setup_testfs():
+ os.chdir(tmpdir) # reverted by common_test_environment
+ if not setup_testfs(b'testfs.img', b'testfs'):
pytest.skip('unable to set up test fs; skipping dependent tests')
return
- for f in glob.glob(b'testfs/*'):
- ex(b'rm', b'-rf', f)
- path = b'testfs/foo'
- with open(path, 'wb'): pass
- xattr.set(path, b'foo', b'bar', namespace=xattr.NS_USER)
- m = metadata.from_path(path, archive_path=path, save_symlinks=True)
- xattr.set(path, b'baz', b'bax', namespace=xattr.NS_USER)
- m.apply_to_path(path, restore_numeric_ids=False)
- WVPASSEQ(remove_selinux(xattr.list(path)), [b'user.foo'])
- WVPASSEQ(xattr.get(path, b'user.foo'), b'bar')
- xattr.set(path, b'foo', b'baz', namespace=xattr.NS_USER)
- m.apply_to_path(path, restore_numeric_ids=False)
- WVPASSEQ(remove_selinux(xattr.list(path)), [b'user.foo'])
- WVPASSEQ(xattr.get(path, b'user.foo'), b'bar')
- xattr.remove(path, b'foo', namespace=xattr.NS_USER)
- m.apply_to_path(path, restore_numeric_ids=False)
- WVPASSEQ(remove_selinux(xattr.list(path)), [b'user.foo'])
- WVPASSEQ(xattr.get(path, b'user.foo'), b'bar')
- cleanup_testfs()
+ try:
+ path = b'testfs/foo'
+ with open(path, 'wb'): pass
+ xattr.set(path, b'foo', b'bar', namespace=xattr.NS_USER)
+ m = metadata.from_path(path, archive_path=path, save_symlinks=True)
+ xattr.set(path, b'baz', b'bax', namespace=xattr.NS_USER)
+ m.apply_to_path(path, restore_numeric_ids=False)
+ WVPASSEQ(remove_selinux(xattr.list(path)), [b'user.foo'])
+ WVPASSEQ(xattr.get(path, b'user.foo'), b'bar')
+ xattr.set(path, b'foo', b'baz', namespace=xattr.NS_USER)
+ m.apply_to_path(path, restore_numeric_ids=False)
+ WVPASSEQ(remove_selinux(xattr.list(path)), [b'user.foo'])
+ WVPASSEQ(xattr.get(path, b'user.foo'), b'bar')
+ xattr.remove(path, b'foo', namespace=xattr.NS_USER)
+ m.apply_to_path(path, restore_numeric_ids=False)
+ WVPASSEQ(remove_selinux(xattr.list(path)), [b'user.foo'])
+ WVPASSEQ(xattr.get(path, b'user.foo'), b'bar')
+ finally:
+ cleanup_testfs(b'testfs.img', b'testfs')
--
2.47.3