before-installation tests failing (search path)

4 views
Skip to first unread message

Greg Troxel

unread,
Jan 31, 2022, 11:05:37 AM1/31/22
to bup-...@googlegroups.com

System in NetBSD 9 amd64, with python3.9. Bup 0.32 built from pkgsrc
works (with 3.9), and two very complicated python things work, plus
certbot and other lesser things.

For a while, I have been unable to run the tests at all. I am just
building and running "gmake check" as I have been doing for a very long
time. I'm returning to figure this out.

The essence of my build script is

# In shell, before calling build.
export BUP_PYTHON_CONFIG=/usr/pkg/bin/python3.9-config

# Work around python3.8-config's lack of rpath
LDFLAGS="-L/usr/pkg/lib -Wl,-R/usr/pkg/lib"
export LDFLAGS

printenv > ENVIRONMENT.${VERSION}${PYVAR}${TZVAR}
gmake clean > CLEAN.${VERSION}${PYVAR}${TZVAR} 2>&1
./configure --with-pylint=no > CONFIGURE.${VERSION}${PYVAR}${TZVAR} 2>&1
gmake > BUILD.${VERSION}${PYVAR}${TZVAR} 2>&1
gmake check < /dev/null > CHECK.${VERSION}${PYVAR}${TZVAR} 2>&1

and a bunch of stuff to set the variables to keep my build logs
organized.

The check fails, and it's because the bup command can't find the python
libdir.

I used to get "No module named bup.main". This was from executing
(while in top-level of sources)

./bup

which was a symlink to cmd/lib/bup. I cleaned up and that symlink is
not generated by the build.

Before cleanup, I could do

./bup - failed
./cmd/lib/bup - failed
cd cmd && lib/bup - ok
cd cmd/lib && ./bup - ok

where failed is "no module" as above, and ok is either "version works"
or the expected help message.

Now, gmake check produces:

Warning: pandoc not found; skipping manpage generation
mkdir test/tmp
./pylint
./pylint: doing nothing given ./configure --with-pylint=no
./bup features
bash: line 1: ./bup: No such file or directory
gmake: *** [GNUmakefile:213: test] Error 127


It's as if the test production in the Makefile expects to be run from
src/cmd instead of top level.

If I run ./pytest from top level, I get

$ sh -x pytest
+ set -eu
+ dirname pytest
+ cd .
+ pwd -P
+ script_home=/home/gdt/SOFTWARE/BUP/bup
+ testlibdir=/home/gdt/SOFTWARE/BUP/bup/test/lib
+ export PYTHONPATH=/home/gdt/SOFTWARE/BUP/bup/test/lib
+ exec dev/bup-python -m pytest -v -m 'not release'
bup: unable find lib dir (No such file or directory): ./../lib

and sure enough:

$ dev/bup-python
bup: unable find lib dir (No such file or directory): ./../lib
$ cd dev
$ ./bup-python
Python 3.9.9 (main, Dec 22 2021, 13:31:36)
[GCC 7.4.0] on netbsd9
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D

so it looks like there are two problems:

make test production assumes there is a ./bup at top level when the
build doesn't make it

dev/bup-python expects to find ../lib, which if CWD is dev works, but
if CWD is top level does not


I can't find any "ln" in the Makefile except for the make install
target.
signature.asc

Johannes Berg

unread,
Jan 31, 2022, 11:48:17 AM1/31/22
to Greg Troxel, bup-...@googlegroups.com
On Mon, 2022-01-31 at 11:05 -0500, Greg Troxel wrote:
>
> so it looks like there are two problems:
>
> make test production assumes there is a ./bup at top level when the
> build doesn't make it

Looks like that's checked into the repo - are you not using git? Maybe
it's missing from some tarball?

> dev/bup-python expects to find ../lib, which if CWD is dev works, but
> if CWD is top level does not
>

what does

script_home="$(cd "$(dirname "$0")" && pwd -P)"

result in? Seems odd, because it wants to go from $0, maybe pwd -P is
behaving differently?

johannes

Greg Troxel

unread,
Jan 31, 2022, 8:32:09 PM1/31/22
to Johannes Berg, bup-...@googlegroups.com

Johannes Berg <joha...@sipsolutions.net> writes:

> On Mon, 2022-01-31 at 11:05 -0500, Greg Troxel wrote:
>>
>> so it looks like there are two problems:
>>
>> make test production assumes there is a ./bup at top level when the
>> build doesn't make it
>
> Looks like that's checked into the repo - are you not using git? Maybe
> it's missing from some tarball?

I am, but I was trying to clean, and it seemed really obvious that it
must be created by the build so I deleted it, and in running git status
I misssed it. After "git checkout bup":

$ gmake check
Refreshed lib/bup/checkout_info.py
Warning: pandoc not found; skipping manpage generation
./pylint
./pylint: doing nothing given ./configure --with-pylint=no
./bup features
/home/gdt/SOFTWARE/BUP/bup/bup: No module named bup.main
gmake: *** [GNUmakefile:213: test] Error 1

>> dev/bup-python expects to find ../lib, which if CWD is dev works, but
>> if CWD is top level does not
>
> what does
>
> script_home="$(cd "$(dirname "$0")" && pwd -P)"
>
> result in? Seems odd, because it wants to go from $0, maybe pwd -P is
> behaving differently?

I assume you mean in pytest. I added the obvious "echo $script-home"
right after assignment and then:

$ sh -x pytest
+ set -eu
+ dirname pytest
+ cd .
+ pwd -P
+ script_home=/home/gdt/SOFTWARE/BUP/bup
+ echo /home/gdt/SOFTWARE/BUP/bup
/home/gdt/SOFTWARE/BUP/bup
+ testlibdir=/home/gdt/SOFTWARE/BUP/bup/test/lib
+ export PYTHONPATH=/home/gdt/SOFTWARE/BUP/bup/test/lib
+ exec dev/bup-python -m pytest -v -m 'not release'
bup: unable find lib dir (No such file or directory): ./../lib

and tat looks ok

$ ls -l /home/gdt/SOFTWARE/BUP/bup/test/lib
total 2
-rw-r--r-- 1 gdt lexort 0 Dec 5 2020 __init__.py
drwxr-xr-x 2 gdt lexort 512 Jan 31 10:33 buptest
-rw-r--r-- 1 gdt lexort 738 Dec 5 2020 wvpytest.py

I think the problem is the bup command not finding
/home/gdt/SOFTWARE/BUP/bup/lib, which has the python code needed to run
commands.

On this system, pwd -P and pwd -L are the same and I don't think there
are symlinks.



Reading

I find

prepend_lib_to_pythonpath(const char * const exec_path,

and it seems to look for the "parent" and then look for lib within that.


I am wondering if the difference is that on my system, calling exec on
a symlink causes

char *parent = exe_parent_dir(exec_path);

to be the parent of the directory with the symlink, and on GNU/Linux, it
is the parent of the directory holding the file that the symlink points
to?

It seems to be the other way around (with the printf changed to print
variables):

$ dev/bup-python
bup: unable find lib dir (No such file or directory): parent=. bupmodpath=./../lib

Similarly perhaps the bup-as-symlink command fails to resolve path:

$ ./bup features
/home/gdt/SOFTWARE/BUP/bup/bup: No module named bup.main

$ PYTHONPATH=/home/gdt/SOFTWARE/BUP/bup/lib ./bup features
bup 0.32+
Source 37b8e2e3baf0b6b9900d7a60959d2da006ee3c61 2022-01-16 14:45:48 -0600
Python: 3.9.9
Command line editing (e.g. bup ftp): yes
Saving and restoring POSIX ACLs: no
Saving and restoring extended attributes (xattrs): no


So there's probably one problem.

It looks like exe_path, while ifdefed for NetBSD, is looking for

const int mib[] = {CTL_KERN, KERN_PROC, KERN_PROC_PATHNAME, -1};

and this is returning 0 as a result code but an empty string.
It appears this part of the sysctl mib has been withdrawn (and it's
certainly not specified by POSIX :-).


I commented out the NetBSD from the ifdef, resulting in the fallback
(searching in PATH) and tests are starting to run. However, I wonder if
I am testing with the bup lib that is installed (0.32) instead of git
master.

Overall this feels really fragile and non-portable, relying on things
not required by POSIX, and I wonder if it wouldn't be better to onfigure
in the path, and passing it in the env for tests. (I realize some
people think it's good to be able to move things around and drop them in
prefixes they weren't built for, but I view that as in general unsound
and not to be encouraged.)
signature.asc

Johannes Berg

unread,
Feb 3, 2022, 5:02:39 PM2/3/22
to Greg Troxel, bup-...@googlegroups.com
On Mon, 2022-01-31 at 20:32 -0500, Greg Troxel wrote:
>
> Reading
>
> I find
>
> prepend_lib_to_pythonpath(const char * const exec_path,
>
> and it seems to look for the "parent" and then look for lib within that.
>
>
> I am wondering if the difference is that on my system, calling exec on
> a symlink causes
>
> char *parent = exe_parent_dir(exec_path);
>
> to be the parent of the directory with the symlink, and on GNU/Linux, it
> is the parent of the directory holding the file that the symlink points
> to?

Hah.

Indeed on Linux the symlinks are resolved, apparently, in /proc/self/exe
which we read.

> It looks like exe_path, while ifdefed for NetBSD, is looking for
>
> const int mib[] = {CTL_KERN, KERN_PROC, KERN_PROC_PATHNAME, -1};
>
> and this is returning 0 as a result code but an empty string.
> It appears this part of the sysctl mib has been withdrawn (and it's
> certainly not specified by POSIX :-).

Oops :)

> I commented out the NetBSD from the ifdef, resulting in the fallback
> (searching in PATH) and tests are starting to run. However, I wonder if
> I am testing with the bup lib that is installed (0.32) instead of git
> master.

Quite possibly?

> Overall this feels really fragile and non-portable, relying on things
> not required by POSIX, and I wonder if it wouldn't be better to onfigure
> in the path, and passing it in the env for tests. (I realize some
> people think it's good to be able to move things around and drop them in
> prefixes they weren't built for, but I view that as in general unsound
> and not to be encouraged.)

Hmm. I guess it could be considered, but it'd also mean you can't
actually test then install without rebuilding, which is a bit ...
annoying?

However, I'm not sure why we don't use the standard bash trick of
realpath(argv[0])? That ought to be more or less portable, unless
somebody calls bup with some kind of execve() tricks?

johannes

Greg Troxel

unread,
Feb 3, 2022, 8:04:07 PM2/3/22
to Johannes Berg, bup-...@googlegroups.com

Johannes Berg <joha...@sipsolutions.net> writes:

>> Overall this feels really fragile and non-portable, relying on things
>> not required by POSIX, and I wonder if it wouldn't be better to onfigure
>> in the path, and passing it in the env for tests. (I realize some
>> people think it's good to be able to move things around and drop them in
>> prefixes they weren't built for, but I view that as in general unsound
>> and not to be encouraged.)
>
> Hmm. I guess it could be considered, but it'd also mean you can't
> actually test then install without rebuilding, which is a bit ...
> annoying?

What I meant is

compile in the right path

pass some BUP_TEST_PYTHONLIB_PATH (wrong name, but you get the point)
that has the uninstalled libs, and have the executable use that if
defined, or else the compiled in version

This is basically like using LD_LIBRARY_PATH in tests. It is slightly
icky, but it is devoid of magic and thus likely robust against things
the author does not know about.

> However, I'm not sure why we don't use the standard bash trick of
> realpath(argv[0])? That ought to be more or less portable, unless
> somebody calls bup with some kind of execve() tricks?

realpath is actually specified by POSIX:

https://pubs.opengroup.org/onlinepubs/9699919799/functions/realpath.html

As for argv[0], that doesn't seem standardized, but if the test harness
doesn't do anything crazy, which it doesn't seem to, then that ought to
work. It would certainly have fewer system-specific ifdefs, which would
be a huge improvement, tending to work on systems not yet debugged.




signature.asc
Reply all
Reply to author
Forward
0 new messages