Failing tests

[Nathaniel]

I'm having more failing tests.

Aside from the one I just sent a patch for, I'm also having three tests failing on a cut -b -40  line.

Looking at test-get-missing  as a representative example, the output is here: https://pastebin.com/5XKVmxRa

The line of the test that fails is:

WVPASS git ls-tree src:a | WVPASS cut -d' ' -f 3 \

   | WVPASS cut -b -40 | WVPASS head -1 > bupm-oid

I have worked this from every angle I can think of and I can't figure out what's going wrong... The only thing that jumps out to me as a clue is the fact that, in the output, it always shows the head  command running before  the cut  commands, and that seems wrong. How can head  run early in the pipeline before the output being piped to it as input is available?

If I move the head  command up in line and change the test to read like so, it passes:

WVPASS git ls-tree src:a | WVPASS head -1 | WVPASS cut -d' ' -f 3 \

   | WVPASS cut -b -40 > bupm-oid

But it makes no sense why it works when the other doesn't.

Does anyone else see something I'm missing?

—

Nathaniel

[Rob Browning]

"'Nathaniel' via bup-list" <bup-...@googlegroups.com> writes:

> Aside from the one I just sent a patch for, I'm also having three
> tests failing on a cut -b -40 line. Looking at test-get-missing as a
> representative example, the output is here:
> https://pastebin.com/5XKVmxRa

First, I'd say run ./pytest instead of pytest (then you also don't need
-m 'not release') so you'll have the right pytest arrangement (env,
etc.).

./pytest -v -m 'not release' test/ext/test-get-missing

Oh, and sometimes "-s" can be helpful. Though for the ext/ tests you
can generally just run them directly right now too (without pytest),
e.g.:

test/ext/test-get-missing

which can simplify things, avoiding pytest output
redirection/manipulation.

> WVPASS git ls-tree src:a | WVPASS cut -d' ' -f 3 \
> | WVPASS cut -b -40 | WVPASS head -1 > bupm-oid

From the test output, looks like this is roughly current main?

And if you run test/ext/test-get-missing directly, or via -s, I suspect
the output may improve, but in any case, another quick thing you can do
is to just hack up the test, e.g.:

...

WVPASS git ls-tree src:a
WVPASS git ls-tree src:a | WVPASS cut -d' ' -f 3

WVPASS git ls-tree src:a | WVPASS cut -d' ' -f 3 \
| WVPASS cut -b -40 | WVPASS head -1 > bupm-oid

exit 42

etc., and then you can see what's going on right when it exits -- what
exactly is being fed to the "cut -b -40"...

Hope this helps.
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Nathaniel Hourt]

On 2025-07-09 13:32, Rob Browning wrote:
> "'Nathaniel' via bup-list" <bup-...@googlegroups.com> writes:
>
>> Aside from the one I just sent a patch for, I'm also having three
>> tests failing on a cut -b -40 line. Looking at test-get-missing as a
>> representative example, the output is here:
>> https://pastebin.com/5XKVmxRa
>
> First, I'd say run ./pytest instead of pytest (then you also don't need
> -m 'not release') so you'll have the right pytest arrangement (env,
> etc.).
>
> ./pytest -v -m 'not release' test/ext/test-get-missing

I have been running `./pytest test/ext/test-get-missing` to run the test
through pytest. I guess it prints the long form at the top of its
output.

> Oh, and sometimes "-s" can be helpful. Though for the ext/ tests you
> can generally just run them directly right now too (without pytest),
> e.g.:
>
> test/ext/test-get-missing
>
> which can simplify things, avoiding pytest output
> redirection/manipulation.

Yep, been doing that too. =) Here's the output if I just run
`./test/ext/test-get-missing` directly: https://pastebin.com/FSq6kK4J

>> WVPASS git ls-tree src:a | WVPASS cut -d' ' -f 3 \
>> | WVPASS cut -b -40 | WVPASS head -1 > bupm-oid
>
> From the test output, looks like this is roughly current main?

Yeah, otherwise the fsck test fails. When the hardlink fix makes it into
a release, I'll target that instead, but right now it's main.

> And if you run test/ext/test-get-missing directly, or via -s, I suspect
> the output may improve, but in any case, another quick thing you can do
> is to just hack up the test, e.g.:
>
> ...
>
> WVPASS git ls-tree src:a
> WVPASS git ls-tree src:a | WVPASS cut -d' ' -f 3
>
> WVPASS git ls-tree src:a | WVPASS cut -d' ' -f 3 \
> | WVPASS cut -b -40 | WVPASS head -1 > bupm-oid
>
> exit 42
>
> etc., and then you can see what's going on right when it exits -- what
> exactly is being fed to the "cut -b -40"...

Alas, that's what I meant by "from every angle I can think of" — I've
been chewing the test up left and right. At every intermediate step, the
output looks right. Whatever it's doing, it seems to only do when it's
the complete pipeline. And that makes no sense, but it's what I'm
seeing.

Any idea why the output always shows the `head` step running before the
`cut` steps? That seems way off to me, and if it's actually executing
these stages out of order (somehow) then that would likely cause the
problem.

—
Nathaniel

[Greg Troxel]

"'Nathaniel Hourt' via bup-list" <bup-...@googlegroups.com> writes:

> Any idea why the output always shows the `head` step running before
> the `cut` steps? That seems way off to me, and if it's actually
> executing these stages out of order (somehow) then that would likely
> cause the problem.

I think with pipes they are set up and all N are executed and they are
nondeterministic about when started, but it shouldn't matter.

But obviously you've found something needing figuring out.

[Johannes Berg]

On Wed, 2025-07-09 at 15:00 -0400, Greg Troxel wrote:
> "'Nathaniel Hourt' via bup-list" <bup-...@googlegroups.com> writes:
>
> > Any idea why the output always shows the `head` step running before
> > the `cut` steps? That seems way off to me, and if it's actually
> > executing these stages out of order (somehow) then that would likely
> > cause the problem.
>
> I think with pipes they are set up and all N are executed and they are
> nondeterministic about when started, but it shouldn't matter.

And if you're going by the printout of the ok/FAIL, then I guess that's
determined by when the individual commands actually _exit_, so I
wouldn't worry about the order of that too much?

What tool _is_ your cut? Maybe you can print the exit status in WVPASS
or something, but my man page for it doesn't even document error
returns.

johannes

[Nathaniel Hourt]

The `cut` is coreutils, though busybox's `cut` also shows the issue.
Interestingly enough, busybox's `chmod` breaks `test-ls`... I think it
might be a bug in busybox; doesn't seem to handle the `chmod u=rwX`
correctly. Hmm...

Anyways, I decided to make a minimal environment to reproduce this issue
in. If anyone else wants to look at it, there's a docker image tarball
at https://nathaniel.land/fileshare/bup-test-failures.tar.gz (22MiB
download) that can be loaded with `docker load -i
bup-test-failures.tar.gz`. In the container, do `su - abuilder; cd bup;
abuild -rK` and watch it build and test. Bup source/build is in
`~abuilder/bup/src/bup-main` for poking around in.

Sadly, reproducing the problem in a minimal environment did not
immediately point me at the answer. Hahaha

—
Nathaniel

[Johannes Berg]

On Thu, 2025-07-10 at 17:06 -0500, 'Nathaniel Hourt' via bup-list wrote:
>
> Sadly, reproducing the problem in a minimal environment did not
> immediately point me at the answer. Hahaha

Well adding strace and running made it simple to figure out ... I don't
know what's special about this cut/head combination, but basically 'cut'
dies because it gets SIGPIPE, because 'head' exits after reading the
first line.

From the cut strace:

read(0, "09f8c6c0d5ad2df6fda3240aab881f2e"..., 1024) = 47
ioctl(1, TIOCGWINSZ, 0x7fff71fd0238) = -1 ENOTTY (Not a tty)
writev(1, [{iov_base="09f8c6c0d5ad2df6fda3240aab881f2e"..., iov_len=40}, {iov_base="\n", iov_len=1}], 2) = 41
read(0, "d00491fd7e5bb6fa28c517a0bb32b8b5"..., 1024) = 129
read(0, "", 1024) = 0
close(0) = 0
writev(1, [{iov_base="d00491fd7e5bb6fa28c517a0bb32b8b5"..., iov_len=123}, {iov_base=NULL, iov_len=0}], 2) = -1 EPIPE (Broken pipe)
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=3293, si_uid=1000} ---
+++ killed by SIGPIPE +++


You can play with something like

(for i in $(seq 1000) ; do echo asdfasdfasdfasdf ; done) | strace cut -b -1 | head -1

to reproduce it - and vary the '1000' number there.

In your container, we get the SIGPIPE with 2 instead of 1000. On my F42
system, I only get the SIGPIPE starting at a bit over 2000. Could be
something with the input buffer size, perhaps, on my system both cut and
head use 8k, but on your container they use 1k, though both are GNU
coreutils? Scheduling shouldn't be different between the container on my
system and my system itself, I'd think?

So mystery solved, but I really have no idea how to work around it for
the tests. We don't really want to interpose "WVPASS sponge /dev/stdout"
everywhere we use "head -1", I guess ... hrm.

johannes

[Johannes Berg]

On Mon, 2025-07-14 at 17:52 +0200, Johannes Berg wrote:
>
> So mystery solved, but I really have no idea how to work around it for
> the tests. We don't really want to interpose "WVPASS sponge /dev/stdout"
> everywhere we use "head -1", I guess ... hrm.

Well, actually, the right answer is probably to just remove the WVPASS
from the 'cut -b -40' in that case. Let it fail here, it doesn't matter.
It doesn't actually have a really useful exit status anyway.

johannes

[Nathaniel Hourt]

Well now that we know why it's happening, it might now make sense to
move the `head` call to the front of the line. Call it an optimization:
no need to `cut` those second, third, and fourth lines if we only really
want the first one!

Though I guess the point here is that it might not actually be cutting
up those lines even when it works? Oi...

Good call on using `strace` – I don't think of using it as often as I
should.

—
Nathaniel

[Johannes Berg]

On Mon, 2025-07-14 at 11:29 -0500, 'Nathaniel Hourt' via bup-list wrote:
> Well now that we know why it's happening, it might now make sense to
> move the `head` call to the front of the line. Call it an optimization:
> no need to `cut` those second, third, and fourth lines if we only really
> want the first one!

But it might just move the SIGPIPE to another command, and I think for
the 'git ls-tree' we might actually want the WVPASS, it might have a
useful exit status.

Dunno.

johannes

[Nathaniel Hourt]

Well, it seems that whereas `cut` is ornery if STDOUT breaks before it
finishes writing output, `git ls-tree` doesn't care (how would we verify
this?). And if the `cut` commands come after the `head`, they don't see
anything amiss.

—
Nathaniel

[Johannes Berg]

On Mon, 2025-07-14 at 11:50 -0500, 'Nathaniel Hourt' via bup-list wrote:
>
> Well, it seems that whereas `cut` is ornery if STDOUT breaks before it
> finishes writing output, `git ls-tree` doesn't care (how would we verify
> this?).

But it doesn't disable SIGPIPE either:

(in a large enough git repo, here the linux kernel because that's what I
have around)
$ strace git ls-tree -r HEAD|head -1
...
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=137553, si_uid=1000} ---

+++ killed by SIGPIPE +++

johannes

[Rob Browning]

Thanks for tracking this down -- I'd actually vaguely wondered about
sigpipe earlier, but hadn't pursued it.

I agree that ignoring the error there is likely "fine", though it's a
touch unsatisfying since in the limiting case I'd want to know if say, a
command segfaulted part way through (e.g. testing for "x" and a command
only prints the "x" of "xyz" and then crashes).

Of course, as you suggest, we can avoid the issue any number of ways,
via sponge, tempfiles, broader changes, etc. In any case I'll figure
something out.

Nathaniel, do we already know how pervasive this is on your system,
i.e. does this only affect the tests in a few places there right now, or
would we have to make more pervasive changes to get the tests to pass?

Thanks all

[Rob Browning]

"'Nathaniel Hourt' via bup-list" <bup-...@googlegroups.com> writes:

> Yeah, otherwise the fsck test fails. When the hardlink fix makes it into
> a release, I'll target that instead, but right now it's main.

If this is for "real use", I'd recommend 0.33.x for now, which should
also have the fix, rather than main. (I'd be cautious about relying on
main, particularly right now.)

[Nathaniel Hourt]

On 2025-07-14 12:23, Rob Browning wrote:
> "'Nathaniel Hourt' via bup-list" <bup-...@googlegroups.com> writes:
>
>> Yeah, otherwise the fsck test fails. When the hardlink fix makes it
>> into
>> a release, I'll target that instead, but right now it's main.
>
> If this is for "real use", I'd recommend 0.33.x for now, which should
> also have the fix, rather than main. (I'd be cautious about relying on
> main, particularly right now.)

OK, good call. Thanks for the heads up.

On 2025-07-14 12:19, Rob Browning wrote:
> Nathaniel, do we already know how pervasive this is on your system,
> i.e. does this only affect the tests in a few places there right now,
> or
> would we have to make more pervasive changes to get the tests to pass?

I believe this SIGPIPE issue is limited to those 4 tests, but I'll find
out soon. This issue seems closely related to the testing infrastructure
and not to bup itself, so I would be surprised if it crops up in other
places.

—
Nathaniel

[Rob Browning]

Nathaniel Hourt <i...@nathaniel.land> writes:

> I believe this SIGPIPE issue is limited to those 4 tests, but I'll find
> out soon. This issue seems closely related to the testing infrastructure
> and not to bup itself, so I would be surprised if it crops up in other
> places.

Oh, I meant "other places in bup's tests", i.e. wondering whether for
now we might be able to get away with more limited changes in a few
places, or we'd have to fix this more broadly for the tests to pass
reliably enough in your environment.

Thanks

[Rob Browning]

For this particular kind of failure, could we replace head -1 with
something that doesn't exit early, and if so, does 'sed -n 1p' do that
(and do that "everywhere")?

[Rob Browning]

Rob Browning <r...@defaultvalue.org> writes:

> For this particular kind of failure, could we replace head -1 with
> something that doesn't exit early, and if so, does 'sed -n 1p' do that
> (and do that "everywhere")?

OK, since I don't know offhand, I may just do something like this in
test-get-missing for now:

WVPASS readarray ls_tree < <(WVPASS git ls-tree src:a) || exit $?
bupm_oid="$(ent-oid "${ls_tree[0]}")" || exit $?

and side-step the whole issue.

That relies on ent-id, which I'd already written to shorten similar
pipelines in some forthcoming --rewrite --repair tests.

ent-oid() # git tree entry oid
{
local ls_tree_line="$1" # MODE TYPE OID\tname
oid="${ls_tree_line%%$'\t'*}"
echo "${oid##* }"

[Rob Browning]

Nathaniel Hourt <i...@nathaniel.land> writes:

> I believe this SIGPIPE issue is limited to those 4 tests, but I'll find
> out soon. This issue seems closely related to the testing infrastructure
> and not to bup itself, so I would be surprised if it crops up in other
> places.

Here's a preliminary, potential patch for all the "head -1"'s I
noticed. (Though I'm fairly certain there are other, related, potential
issues.)

[Nathaniel Hourt]

> --
> Rob Browning
> rlb @defaultvalue.org and @debian.org
> GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
> GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

The only test failures I ever had from the SIGPIPE issue were those
four. No other `head` commands did it; only the `cut | head` ones.

Your patch works for me on `main`. It obviously wasn't made for 0.33.x
and won't work there without adjustment.

Out of curiosity, what would be the downside to fixing this issue by
patching WVPASS to eat SIGPIPEs unless you tell it not to? It seems that
in the vast majority of cases, you don't want any test to fail simply
because a command in the pipeline which knows when it's got enough input
to do its job and closes the pipe? In those few cases you do, you
probably know it. Shouldn't the burden of extra syntax and complication
be put on those few cases (which you expect to be special and require
extra syntax) rather than all the ones where you don't want/expect to
care?

—
Nathaniel

[Rob Browning]

"'Nathaniel Hourt' via bup-list" <bup-...@googlegroups.com> writes:

> Your patch works for me on `main`. It obviously wasn't made for 0.33.x
> and won't work there without adjustment.

Thanks for testing, and right, not sure whether I'll backport it.

> Out of curiosity, what would be the downside to fixing this issue by
> patching WVPASS to eat SIGPIPEs unless you tell it not to?

Hmm, I'm not sure I see how that'd work -- I believe the issue is that
the command writing to head is getting a SIGPIPE during its write() call
when head exits "early".

Unless you mean something like what Johannes suggested, i.e. interposing
a "sponge /dev/stdout" to always exhaust stdin.

something | sponge /dev/stdout | head -1

[Rob Browning]

Rob Browning <r...@defaultvalue.org> writes:

> Hmm, I'm not sure I see how that'd work -- I believe the issue is that
> the command writing to head is getting a SIGPIPE during its write() call
> when head exits "early".
>
> Unless you mean something like what Johannes suggested, i.e. interposing
> a "sponge /dev/stdout" to always exhaust stdin.
>
> something | sponge /dev/stdout | head -1

Johannes also pointed out that you could also try to detect the
"probably a SIGPIPE" exit status, which for us, in bash, should be 128 +
SIGPIPE (the latter we can get via python signal.SIGPIPE).

Then if the application doesn't have some competing exit(THAT_VALUE) of
its own, and doesn't mess with SIGPIPE, then WVPASS (or similar) could
look for and ignore THAT_VALUE.

For now I think I may be fine with the readarrays, but in the relevant
cases so far, any of the approaches discussed would likely be fine,
including, practially speaking, perhaps even just dropping the WVPASS
from those head invocations.

[Rob Browning]

Rob Browning <r...@defaultvalue.org> writes:

> including, practially speaking, perhaps even just dropping the WVPASS
> from those head invocations.

Johannes mentioned to me that I conflated the direction (again) -- it'd
of course be the command piping *to* head.

[Nathaniel]

Yeah, I can't come up with an elegant way to do it. WVPASS controls the way the command gets input and could easily inject a helper into the pipeline to parrot stdin to stdout, and that helper could eat the SIGPIPE if whatever was on the other end of its stdout closed, but would still accept and ignore everything that came in on stdin... It would solve our SIGPIPE problem, but we'd still need to have some way to disable it so a pipeline from a program that never stops generating output (for instance, yes ) can still break as desired.

But injecting the parrot into the pipeline only makes sense if you know  WVPASS is downstream in a pipeline and will be processing input. For any invocation of WVPASS where the command won't be looking at stdin, adding the parrot and piping to the command causes it to hang waiting for input. So you can't do it unless you have some way to know whether the command is using stdin or not, and WVPASS gets used in both cases.

Anyways, Rob's given us a patch that works. It's no longer broke; I'll stop thinking about fixing it. =)

—
Nathaniel

[Nathaniel]

On Jul 18 2025, at 3:15 pm, Rob Browning <r...@defaultvalue.org> wrote:

[Rob Browning]

"'Nathaniel' via bup-list" <bup-...@googlegroups.com> writes:

> Anyways, Rob's given us a patch that works. It's no longer broke; I'll
> stop thinking about fixing it.

I actually just fixed it a slightly different way in main -- I kept the
readarray for one case, but replaced the others with a new WVPIPE that
works the way Johannes suggested, i.e. relies on bash subprocesses
exiting with 128 + SIGPIPE status for a SIGPIPE, e.g.:

WVPIPE something | WVPASS head -1 ...

Thanks