Call for testing for a 0.34 release

[Rob Browning]

It's time. Please test main in preparation for a 0.34 release. This is
a substantial release with a lot of change, so closer scrutiny is
warranted. See note/main.md for an overview, which may suggest changes
relevant to you that warrant closer scrutiny.

https://github.com/bup/bup/tree/main
https://codeberg.org/bup/bup/src/branch/main

And it's not critical, but if you have the space (and time) and you
happen to test repository conversions via say `bup get --rewrite`ing to
a new repository (and if you're comfortable sharing some information),
we'd be interested in the effect. For example, how did the object count
change:

# Assuming you have -c support
du -ch REPO/objects/pack/pack-*.idx | tail -n1

And how did the overall size of the repository change (du -sh REPO)?

Thanks
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Rob Browning]

Rob Browning <r...@defaultvalue.org> writes:

> we'd be interested in the effect. For example, how did the object count
> change:
>
> # Assuming you have -c support
> du -ch REPO/objects/pack/pack-*.idx | tail -n1

Oh, and this is of course the total idx size, but that's proportional to
the object count (plus or minus the per-idx fixed costs).

The "count:" in "git --git-dir REPO count-objects -v" would also work,
just a slightly different measurement. Happy to know both.

[Nathaniel Hourt]

> And it's not critical, but if you have the space (and time) and you
> happen to test repository conversions via say `bup get --rewrite`ing to
> a new repository (and if you're comfortable sharing some information),
> we'd be interested in the effect.

I was trying to get these numbers for you, but...

How exactly does one do the `bup get --rewrite` procedure to copy their
repository?

Making a guess from the bup-get manpage, I updated to 0.34 then tried
making a new repo and with my BUP_DIR pointing to the new repo, running
for each backup name in the old repo:
bup get --rewrite --repair -s /old/bup/dir --append <backup name>

The first backup I copied was small and went quickly, seemed to do the
right thing. The second one has been running for two days, has copied
89GiB, and staunchly refuses to copy a byte more. CPU cycles are going
to both the bup command and a `git cat-file --batch-command` it started.

Am I doing this wrong?

—
Nathaniel

[Rob Browning]

"'Nathaniel Hourt' via bup-list" <bup-...@googlegroups.com> writes:

> I was trying to get these numbers for you, but...
>
> How exactly does one do the `bup get --rewrite` procedure to copy their
> repository?

Effectively what I think you did:

- Pick source and dest repos (can even be the same repo, with
caveats[1]).

- Configure the destination repository as desired:

git --git-dir dest config bup.split.trees true
...

- Transfer/convert the data via bup get:

bup -d dest get -s source --rewrite --append some-branch

Add --repair if you want to allow/require repairs.

> The first backup I copied was small and went quickly, seemed to do the
> right thing. The second one has been running for two days, has copied
> 89GiB, and staunchly refuses to copy a byte more. CPU cycles are going
> to both the bup command and a `git cat-file --batch-command` it started.
>
> Am I doing this wrong?

It doesn't sound like it. So the repo size is no longer growing? And
is there any notable IO in say "iostat -mx 5" or htop's IO tab, or...?

Also, how big are the repository "*.idx" files relative to RAM?
e.g. the "total" line of say "du -sh REPO/objects/pack/*.idx, how big is
the source repository overall, and what's the source's "in-pack" value
for "git --git-dir source count-objects"?

[1] Rewrites will respect the destination configuration settings, and
when source and dest are the same you'll duplicate data in the repo
whenever the settings cause the results to change. Though you could
of course remove the old branches after the rewrite and run a
(potentially expensive) gc. And of course you'll also want
--append: rather than --append.

[Nathaniel Hourt]

On 2026-07-29 13:04, Rob Browning wrote:
> It doesn't sound like it. So the repo size is no longer growing? And
> is there any notable IO in say "iostat -mx 5" or htop's IO tab, or...?

No, at least not by so much as a byte in several hours. There was
definitely IO, mostly read (spikes up to ~5MB/s), from the git command
and a little from bup. Not much write that I could see.

I have canceled it by now to try with some smaller branches. I just
finished one that copied 2.4GB and that took many hours... so maybe I'm
just being too impatient? I wonder if I need to bring this over to a
faster computer to process... My backup server is not exactly renowned
for its speed...

> Also, how big are the repository "*.idx" files relative to RAM?
> e.g. the "total" line of say "du -sh REPO/objects/pack/*.idx, how big
> is
> the source repository overall, and what's the source's "in-pack" value
> for "git --git-dir source count-objects"?

System RAM is 8GB. Total size of *.idx files is 733MB. Normal memory
usage is around half a gig. Source repo is 169GB. count-objects says...
uhh... 0 objects, 0 kilobytes...?

Oh, and I just noticed there are several dozen of these sitting in
dmesg:
[3942947.573399] bup[14739]: unhandled signal 7 code 0x2 at
0x0000003f9b481087 in _helpers.so[3bc4,3f9f5c9000+9000]
[3942947.573436] CPU: 0 UID: 0 PID: 14739 Comm: bup Not tainted
6.19.0-friskybusiness #37 NONE
[3942947.573445] Hardware name: Milk-V Mars (DT)
[3942947.573450] epc : 0000003f9f5ccbc4 ra : 0000003f9f5ccbb4 sp :
0000003fe82774a0
[3942947.573456] gp : 0000002ae4cf6800 tp : 0000003fa0a03da8 t0 :
0000000000000038
[3942947.573462] t1 : 0000003f9f5cbaec t2 : 0000003fe82771d8 s0 :
0000003f9ad03ad4
[3942947.573467] s1 : 0000003fa0a01560 a0 : 0000003f9ac60ab8 a1 :
0000000000000004
[3942947.573472] a2 : 0000000003ffffff a3 : 0000003fe82774c7 a4 :
0000000000000000
[3942947.573478] a5 : 0000003f9b481087 a6 : 0000000000000000 a7 :
0000003fe8277450
[3942947.573483] s2 : 0000003f9b456000 s3 : 0000000000000014 s4 :
0000003fa08f31c8
[3942947.573489] s5 : 0000003fa054279a s6 : 0000003fa0869501 s7 :
0000003fa0873e81
[3942947.573494] s8 : 0000003fa08f31c8 s9 : 0000003fa02698e0 s10:
ffffffffffffffe8
[3942947.573500] s11: 0000003fa0873e80 t3 : 0000003fa09a5584 t4 :
0000003fe8277474
[3942947.573506] t5 : 0000003f9ac60ab8 t6 : 0000000000000069
[3942947.573510] status: 0000000200004020 badaddr: 0000003f9b481087
cause: 000000000000000d
[3942947.573521] Code: f0ef e1bf 57c2 4583 0271 6522 27c1 1782 9381 97ca
(c703) 0007

Looks like it's been popping one of those out every hour or two... Not
sure what to make of that...

Maybe I'll try doing this part in a slightly less exotic environment,
see what happens...

—
Nathaniel

[Rob Browning]

"'Nathaniel Hourt' via bup-list" <bup-...@googlegroups.com> writes:

> I have canceled it by now to try with some smaller branches. I just
> finished one that copied 2.4GB and that took many hours... so maybe I'm
> just being too impatient? I wonder if I need to bring this over to a
> faster computer to process... My backup server is not exactly renowned
> for its speed...

I can be expensive. But unless there's a lot of churn in the saves,
rewrite should at least be remembering things it's already rewritten
(i.e. pruning).

> System RAM is 8GB. Total size of *.idx files is 733MB. Normal memory
> usage is around half a gig. Source repo is 169GB. count-objects says...
> uhh... 0 objects, 0 kilobytes...?

Sorry, I forgot the -v (i.e. try count-objects -v). Though it sounds
like RAM may not be an issue. git can easily mmap all of those indexes
and have them fit in RAM.

> Oh, and I just noticed there are several dozen of these sitting in
> dmesg:
> [3942947.573399] bup[14739]: unhandled signal 7 code 0x2 at
> 0x0000003f9b481087 in _helpers.so[3bc4,3f9f5c9000+9000]

SIGBUS?

> Maybe I'll try doing this part in a slightly less exotic environment,
> see what happens...

Definitely be interesting to know if it behaves better elsewhere.

[Nathaniel Hourt]

On 2026-07-29 15:58, Rob Browning wrote:
> Definitely be interesting to know if it behaves better elsewhere.

Seems a lot happier on x86. No SIGBUS errors, and it got through the big
branch it was getting stuck on before. I'm beginning to suspect it
would've eventually gotten there, but honestly I'd rather just bring it
over to my laptop to do this part if it's going to take *that* long.

Those SIGBUS errors make me suspicious that something somewhere is
abusing RAM somehow, though... My servers are mostly RISC-V and use
MMIO, meaning there are wide swaths of address space which aren't
actually mapped to anything... Could something be poking one of those
dead spaces and generating those SIGBUSes? I don't know... Might be
worth spending some quality time with Valgrind.

—
Nathaniel

[Greg Troxel]

"'Nathaniel Hourt' via bup-list" <bup-...@googlegroups.com> writes:

I would expect that random pointers would be not mapped and result in
SIGSEGV.

I am guessing that on RISC-V unaligned data accesses are an error and
that they probably lead to SIGBUS. sparc is like this (and I think most
arm).

I have not tried rewrite on arm; probably I should.

[Greg Troxel]

Have you run bup's self tests, on x86_64, and on your RISC-V server?

[Johannes Berg]

On Wed, 2026-07-29 at 15:25 -0500, 'Nathaniel Hourt' via bup-list wrote:
>
> Oh, and I just noticed there are several dozen of these sitting in
> dmesg:
> [3942947.573399] bup[14739]: unhandled signal 7 code 0x2 at
> 0x0000003f9b481087 in _helpers.so[3bc4,3f9f5c9000+9000]
>

Do you have _helpers.so, ideally with debug info?

> [3942947.573521] Code: f0ef e1bf 57c2 4583 0271 6522 27c1 1782 9381 97ca (c703) 0007

But I guess we can decode this too if needed.

johannes

[Johannes Berg]

On Thu, 2026-07-30 at 20:06 +0200, Johannes Berg wrote:
>
>
> > [3942947.573521] Code: f0ef e1bf 57c2 4583 0271 6522 27c1 1782 9381 97ca (c703) 0007
>
> But I guess we can decode this too if needed.

Well, that didn't work at least in my attempt - maybe something's
different about your RISC-V, my objdump decoded nothing here.

johannes

[Mark Hewitt]

On 22/07/2026 22:48, Rob Browning wrote:
> It's time. Please test main in preparation for a 0.34 release.

I've tried this on FreeBSD 13.1p9, (with xfs, in a jail) and get two
failures in "long-check": test-rewrite and test-sparse-files.

I will find time in the next few days to look a little deeper - but I
attach the log file for this test in the meantime.

Mark

[Mark Hewitt]

On 31/07/2026 21:14, Mark Hewitt wrote:
> I've tried this on FreeBSD 13.1p9, (with xfs, in a jail) and get two
> failures in "long-check": test-rewrite and test-sparse-files.

Apologies for mistype - that's zfs of course, not xfs!!)

Mark

[Nathaniel Hourt]

On 2026-07-30 13:06, Johannes Berg wrote:
> On Wed, 2026-07-29 at 15:25 -0500, 'Nathaniel Hourt' via bup-list
> wrote:
>>
>> Oh, and I just noticed there are several dozen of these sitting in
>> dmesg:
>> [3942947.573399] bup[14739]: unhandled signal 7 code 0x2 at
>> 0x0000003f9b481087 in _helpers.so[3bc4,3f9f5c9000+9000]
>>
>
> Do you have _helpers.so, ideally with debug info?

I was unable to reproduce the issue in a debug build... But I've
attached the _helpers.so that was in use when I got those errors.

On 2026-07-30 09:09, Greg Troxel wrote:
> Have you run bup's self tests, on x86_64, and on your RISC-V server?

2 failures in long-check on RISC-V:
FAILED test/ext/test-ls:: - Exit status: 1
FAILED test/ext/test-ls-remote:: - Exit status: 1

6 failures on x86_64:
FAILED test/ext/test-fsck:: - Exit status: 1
FAILED test/ext/test-gc-removes-incomplete-trees:: - Exit status: 1
FAILED test/ext/test-index:: - Exit status: 1
FAILED test/ext/test-meta:: - Exit status: 1
FAILED test/ext/test-rm:: - Exit status: 1
FAILED test/ext/test-save-restore-excludes:: - Exit status: 1

[Nathaniel Hourt]

Before the rewrite, with 0.33.10:

# du -ch bup/objects/pack/pack-*.idx | tail -n1
733M total
# du -hs bup
169G
# git --git-dir bup/ count-objects -v
warning: garbage found:
bup//objects/pack/midx-bbff2d04869d40c8a8548dc74843f3e0af3b6c1b.midx
warning: garbage found: bup//objects/pack/bup.bloom
warning: garbage found:
bup//objects/pack/midx-58e7b6768faa2e63341acd499cd37be083aa7100.midx
count: 0
size: 0
in-pack: 25671029
packs: 13585
size-pack: 176150871
prune-packable: 0
garbage: 3
size-garbage: 669036



After the rewrite, with 0.34~89ddbc64a7f2430c1c99b8cd17212c77e380f5ad:

# du -ch bup-34/objects/pack/*.idx | tail -n1
686M total
# du -hs bup-34
169G bup-34
# git --git-dir bup-34 count-objects -v
warning: garbage found:
bup-34/objects/pack/midx-4f1da8bc127be3e17d188547af9564fb3ac1c7b1.midx
warning: garbage found:
bup-34/objects/pack/midx-60e8534fb096becead8c8c5a31b65730874f1127.midx
warning: garbage found: bup-34/objects/pack/bup.bloom
count: 0
size: 0
in-pack: 25667686
packs: 192
size-pack: 176115083
prune-packable: 0
garbage: 3
size-garbage: 599421

[Rob Browning]

Mark Hewitt <mjh.br...@gmail.com> writes:

> ! /usr/home/mjh/Sandbox/bup-test/bup/test/ext/test-rewrite:100 bup+ -d /usr/home/mjh/Sandbox/bup-test/bup/test/tmp/test-rewrite-NkTe3QQ/bup ls -l save/latest/y/testfile1 FAILED
> ! /usr/home/mjh/Sandbox/bup-test/bup/test/ext/test-rewrite:101 grep -- -1122334455 FAILED
> ! /usr/home/mjh/Sandbox/bup-test/bup/test/ext/test-rewrite:103 bup -d /usr/home/mjh/Sandbox/bup-test/bup/test/tmp/test-rewrite-NkTe3QQ/bup ls -l save/latest/y/testfile1 FAILED
> ! /usr/home/mjh/Sandbox/bup-test/bup/test/ext/test-rewrite:104 grep -- 158664 FAILED

If you get test-rewrite to fail again (e.g. "./pytest
test/ext/test--rewrite"), and you want to pursue it, it'd be interesting
to see what's going wrong there. You could change say the test starting
on line 103 like this:

WVPASS out-to log bup -d "$BUP_DIR" ls -l save/latest/y/testfile1
WVPASS WVPASS grep -- 158664 log

and add this to the top of the file:

. test/lib/btl.sh

Then we should be able to see the ls output when it fails.

I'm also not surprised test-sparse-files isn't behaving as expected on
zfs. Offhand, I'd guess those failures are spurious, but I'll plan to
look more closely.

[Rob Browning]

Rob Browning <r...@defaultvalue.org> writes:

> If you get test-rewrite to fail again (e.g. "./pytest
> test/ext/test--rewrite"), and you want to pursue it, it'd be interesting
> to see what's going wrong there. You could change say the test starting
> on line 103 like this:
>
> WVPASS out-to log bup -d "$BUP_DIR" ls -l save/latest/y/testfile1
> WVPASS WVPASS grep -- 158664 log
>
> and add this to the top of the file:
>
> . test/lib/btl.sh
>
> Then we should be able to see the ls output when it fails.

Wait, nevermind.

I was looking at the pytest summary information while thinking I was
looking at the real output. That failure is presumably just due to the
"cannot access" error, and so I'd expect the ls output to be either
empty or truncated, meaning that we probably won't learn anything from
the changes I suggested.

[Johannes Berg]

On Fri, 2026-07-31 at 21:14 +0100, Mark Hewitt wrote:
> On 22/07/2026 22:48, Rob Browning wrote:
> > It's time. Please test main in preparation for a 0.34 release.
>
> I've tried this on FreeBSD 13.1p9, (with xfs, in a jail) and get two
> failures in "long-check": test-rewrite and test-sparse-files.

The sparse one just seems like it has behaviour differences between
lseek() variants or something. I wouldn't worry about it, and anyway
writing sparse files is an optimisation you can't rely on.


The rewrite ones ... I was thinking if you had a symlink in your parent
path by any chance, with a symlink I do get a failure, but a different
one?!

Does this reproduce? You can also run just this test script with

$ ./pytest test/ext/test-rewrite

The reason I'm thinking about symlinks etc. is that you had

> ! /usr/home/mjh/Sandbox/bup-test/bup/test/ext/test-rewrite:97 bup+ -d /usr/home/mjh/Sandbox/bup-test/bup/test/tmp/test-rewrite-NkTe3QQ/bup save -n save --strip /home/mjh/Sandbox/bup-test/bup/test/sampledata ok
> error: cannot access "''/save/2026-07-31-120546/y" in 'save/latest/y/testfile1'

> ! /usr/home/mjh/Sandbox/bup-test/bup/test/ext/test-rewrite:100 bup+ -d /usr/home/mjh/Sandbox/bup-test/bup/test/tmp/test-rewrite-NkTe3QQ/bup ls -l save/latest/y/testfile1 FAILED

so it saved, with --strip, the "/home/mjh/Sandbox/bup-
test/bup/test/sampledata" dir, and the testfile1 really should exist
there unless your checkout was somehow corrupted (seems unlikely).

But the later bup ls (note the command is printed after its own output
... yes this is confusing) can't access it, so my best guess right now
is the --strip is doing the wrong thing.

I think the output of this might help with figuring out things:

--- a/test/ext/test-rewrite
+++ b/test/ext/test-rewrite
@@ -94,7 +94,11 @@ bup+()

# force a re-save of the testfile1 to get it w/o size
WVPASS bup index --fake-invalid "$top/test/sampledata/y/testfile1"
+WVPASS bup index -s /
WVPASS bup+ -d "$BUP_DIR" save -n save --strip "$top/test/sampledata"
+WVPASS bup index -s /
+GIT_DIR="$BUP_DIR" WVPASS git ls-tree -r save
+exit 1

# check that we get the "unknown" size out
WVPASS bup+ -d "$BUP_DIR" ls -l save/latest/y/testfile1 |



johannes

[Mark Hewitt]

On 01/08/2026 19:25, Johannes Berg wrote:

The sparse one just seems like it has behaviour differences between
lseek() variants or something. I wouldn't worry about it, and anyway
writing sparse files is an optimisation you can't rely on.

Yes, I've seen this failure before and for exactly those reasons, I have not been concerned (but included here, just to be complete!)

The rewrite ones ... I was thinking if you had a symlink in your parent
path by any chance, with a symlink I do get a failure, but a different
one?!

Yes - but difficult to avoid because FreeBSD has a symlink home -> usr/home. This is because the FreeBSD default was /usr/home and the /home symlink is a later addition for Linux compatibility.

If I change the user's home directory to be in /usr/home, there is no symlink, and the test then succeeds - so thanks for spotting the cause here. However any Linux (or other) tool that first looks for /home will find the symlink - so this may not be a general solution, and may not match common usage for home directories on FreeBSD anyway.

Does this reproduce? You can also run just this test script with

  $ ./pytest test/ext/test-rewrite

Yes, easily reproduced.

I think the output of this might help with figuring out things:

Thanks for the suggested change here - but now this is clearly due to the "standard" symlink, I think the issue is identified and a solution is possible. I should have re-read the README.

Mark.

[Johannes Berg]

Hi,

> > The sparse one just seems like it has behaviour differences between
> > lseek() variants or something. I wouldn't worry about it, and anyway
> > writing sparse files is an optimisation you can't rely on.
> >
>  Yes, I've seen this failure before and for exactly those reasons, I have not been concerned (but included here, just to be complete!)

Makes sense!

> but now this is clearly due to the "standard" symlink, I think the issue is identified and a solution is possible. I should have re-read the README.

It's strange that the failure mode is different for me, but still, good.

The test was meant to handle it by changing dir to the real path (at the
top of the file), but then 'realpath' wasn't available everywhere so it
got changed to 'cd "$(pwd)"', but that doesn't do anything - perhaps
then 'cd "$(pwd -P)"' is available enough (it _is_ in posix ...) and
fixes it ...

I think it should be fixable, but given the apparent (and surprising)
lack of standard ways to resolve symlinks ... I guess you could try that
pwd -P, it does seem to work on Linux.

johannes

[Greg Troxel]

Johannes Berg <joha...@sipsolutions.net> writes:

> The test was meant to handle it by changing dir to the real path (at the
> top of the file), but then 'realpath' wasn't available everywhere so it
> got changed to 'cd "$(pwd)"', but that doesn't do anything - perhaps
> then 'cd "$(pwd -P)"' is available enough (it _is_ in posix ...) and
> fixes it ...

I see it as pretty normal to use paths that seem right and to put bits
where you can and symlink.

I tend strongly to "ok to rely on POSIX", absent evidence of trouble.

FWIW, pwd -P is fine on NetBSD.

-P was added in Issue 6, released in 2004:

https://pubs.opengroup.org/onlinepubs/9799919799/utilities/pwd.html

[Rob Browning]

Mark Hewitt <mjh.br...@gmail.com> writes:

> ! /usr/home/mjh/Sandbox/bup-test/bup/test/ext/test-sparse-files:45 /home/mjh/Sandbox/bup-test/bup/dev/sparse-size restore/src/foo ok
> ! /usr/home/mjh/Sandbox/bup-test/bup/test/ext/test-sparse-files:46 [ 2621441 = 0 ] FAILED
> called from /usr/home/mjh/Sandbox/bup-test/bup/test/ext/test-sparse-files:46 WVPASS [ 2621441 = 0 ]

I'm wondering what we want to do here. I suppose we could just skip the
sparse file tests for some filesystem types. So far, we've tried to
adapt.

If you wanted to pursue this, you should be able to run

./pytest test/ext/test-sparse-files

or

test/ext/test-sparse-files

directly, and dev/sparse-size already has some accommodations for btrfs
and zfs's previously observed "laziness", but this looks like something
else.

And of course, as mentioned before, there's no specification for how
sparseness is "suppposed" to behave. The main question would be whether
or not this is just a normal possibility for zfs. i.e. as Johannes
suggested, it could be that zfs lseek() just sometimes doesn't create
sparseness in situations where we expect it (where ext4 does).

[Greg Troxel]

I had a dim memory of this, so I rewrote a separable test case. This
run is on NetBSD 10.

$ sh -x _test
+ rm -rf DIR
+ mkdir DIR
+ du -k DIR
1 DIR
+ dd if=/dev/zero of=DIR/SPARSE bs=1k count=8 seek=1024
8+0 records in
8+0 records out
8192 bytes transferred in 0.001 secs (8192000 bytes/sec)
+ du -k DIR
1 DIR
+ sync
+ sleep 30
+ du -k DIR
131 DIR

$ ls -l DIR
total 131
-rw-r--r-- 1 gdt users 1056768 Aug 4 19:37 SPARSE


The file is 1032 * 1024 bytes long. The number of 1k blocks would be
1032.

The number of blocks to store written zeros is 8 * 1024.

There are 122 blocks somehow, as part of representing the file plus the
one that the empty dir takes.

So there is some sparseness.

zfs surely has allocated blocks, but I think they are in the intent log,
not yet committed, and thus do not show up in du until after a bit.

I believe we have a "if zfs, wait" stanza someplace else already for
maybe this very reason.

running with 8192 instead of 1024 seek:

$ sh -x _test
+ rm -rf DIR
+ mkdir DIR
+ du -k DIR
1 DIR
+ dd if=/dev/zero of=DIR/SPARSE bs=1k count=8 seek=8192
8+0 records in
8+0 records out
8192 bytes transferred in 0.001 secs (8192000 bytes/sec)
+ du -k DIR
1 DIR
+ sync
+ sleep 30
+ du -k DIR
131 DIR
+ ls -l DIR
total 131
-rw-r--r-- 1 gdt users 8396800 Aug 4 19:41 SPARSE

same storage used. Proof of sparse I call it.

[Mark Hewitt]

On 04/08/2026 21:01, Rob Browning wrote:
> test/ext/test-sparse-files

The FreeBSD manpage for lseek says:

> If whence is SEEK_HOLE, the offset is set to the start of the next
>  hole greater than or equal to the supplied offset.  The definition
>  of a hole is provided below.
> ...
> ...
> A "hole" is defined as a contiguous range of bytes in a file, all having
> the value of zero, but not all zeros in a file are guaranteed to be
> represented as holes returned with SEEK_HOLE.  File systems are allowed
> to expose ranges of zeros with SEEK_HOLE, but not required to.
> Applications can use SEEK_HOLE to optimise their behavior for ranges of
> zeros, but must not depend on it to find all such ranges in a file.  Each
> file is presented as having a zero-size virtual hole at the very end of
> the file.  The existence of a hole at the end of every data region allows
> for easy programming and also provides compatibility to the original
> implementation in Solaris.  It also causes the current file size (i.e.,
> end-of-file offset) to be returned to indicate that there are no more
> holes past the supplied offset.  Applications should use
> fpathconf(_PC_MIN_HOLE_SIZE) or pathconf(_PC_MIN_HOLE_SIZE) to determine
> if a file system supports SEEK_HOLE.  See pathconf(2).
>
> For file systems that do not supply information about holes, the file
> will be represented as one entire data region.
>
So the trivial test:

> #include <stdio.h>
> #include <unistd.h>
> #include <fcntl.h>
> #include <errno.h>
>
> int main(int argc, char **argv) {
>   long phole = pathconf("hole.c", _PC_MIN_HOLE_SIZE);
>   printf("%ld\n", phole);
>   int fd = open("testfile", O_CREAT|O_RDWR, 0666);
>   char buf[1024] = {0};
>   ssize_t rw = write(fd, buf, 2*phole);
>   off_t rs = lseek(fd, 4*phole, SEEK_CUR);
>   printf("SEEK %ld (%d)\n", rs, errno);
>   rw =  write(fd, "X", 1);
>   printf("%ld\n", rw);
>   close(fd);
> }
>
> mjh@backup:~/Sandbox/bup-test2/t $ ls -la testfile
> -rw-r--r--  1 mjh  mjh  3073 Aug  5 01:09 testfile
> mjh@backup:~/Sandbox/bup-test2/t $ du -s testfile
> 1    testfile
>
Running the equivalent trivia on Linux:

> mjh@Voyager:~/Downloads$ ls -la testfile
> -rw-rw-r-- 1 mjh mjh 3073 Aug  5 01:07 testfile
> mjh@Voyager:~/Downloads$ du -s testfile
> 4    testfile
>

And I believe dev/sparse-size depends on the latter behaviour

Mark

[Mark Hewitt]

On 05/08/2026 01:19, Mark Hewitt wrote:
>
> And I believe dev/sparse-size depends on the latter behaviour
>

My mistake - it used to rely on du -s, now uses SEEK_DATA and SEEK_HOLE
when it can find support - so this yields '0' on both systems.

Mark.

[Rob Browning]

Greg Troxel <g...@lexort.com> writes:

> I had a dim memory of this, so I rewrote a separable test case. This
> run is on NetBSD 10.
>
> $ sh -x _test
> + rm -rf DIR
> + mkdir DIR
> + du -k DIR
> 1 DIR
> + dd if=/dev/zero of=DIR/SPARSE bs=1k count=8 seek=1024
> 8+0 records in
> 8+0 records out
> 8192 bytes transferred in 0.001 secs (8192000 bytes/sec)
> + du -k DIR
> 1 DIR
> + sync
> + sleep 30
> + du -k DIR
> 131 DIR

Hmm, why check the size of DIR and not just SPARSE? To date when
working on sparseness, I've only been concerned with individual files'
"apparent size" (for GNU du, --apparent-size, and presumably also ls -l
or stat's value) vs "device usage" (du's normal result).

> I believe we have a "if zfs, wait" stanza someplace else already for
> maybe this very reason.

We do, after earlier experiences with zfs, in an attempt to accommodate
the behavior there.

And overall, were you just saying that you saw sparseness with your
system and zfs?

[Rob Browning]

Mark Hewitt <mjh.br...@gmail.com> writes:

> And I believe dev/sparse-size depends on the latter behaviour

Hmm, I'm not sure I completely understood what you're saying you've
determined yet.

Oh, and for what it's worth, unless you're saying there's something we
might just need to fix, I've been leaning toward limiting the "must
pass" sparse tests to arrangements that behave the way the current code
expects (e.g. ext4 and maybe xfs? ufs?), leaving out btrfs and zfs for
now.

[Greg Troxel]

Rob Browning <r...@defaultvalue.org> writes:

> Oh, and for what it's worth, unless you're saying there's something we
> might just need to fix, I've been leaning toward limiting the "must
> pass" sparse tests to arrangements that behave the way the current code
> expects (e.g. ext4 and maybe xfs? ufs?), leaving out btrfs and zfs for
> now.

I would prefer to loosen the test per fs to at least detect "file is
full size" vs "file is much smaller", if that's doable.

[Greg Troxel]

Rob Browning <r...@defaultvalue.org> writes:

> Greg Troxel <g...@lexort.com> writes:
>
>> I had a dim memory of this, so I rewrote a separable test case. This
>> run is on NetBSD 10.
>>
>> $ sh -x _test
>> + rm -rf DIR
>> + mkdir DIR
>> + du -k DIR
>> 1 DIR
>> + dd if=/dev/zero of=DIR/SPARSE bs=1k count=8 seek=1024
>> 8+0 records in
>> 8+0 records out
>> 8192 bytes transferred in 0.001 secs (8192000 bytes/sec)
>> + du -k DIR
>> 1 DIR
>> + sync
>> + sleep 30
>> + du -k DIR
>> 131 DIR
>
> Hmm, why check the size of DIR and not just SPARSE?

No good reason, and I don't think it will matter because it's just
adding the dir size to the file size.

> To date when working on sparseness, I've only been concerned with
> individual files' "apparent size" (for GNU du, --apparent-size, and
> presumably also ls -l or stat's value) vs "device usage" (du's normal
> result).

I suggest avoiding assuming du is GNU du. But ls -l and stat should
match.

>> I believe we have a "if zfs, wait" stanza someplace else already for
>> maybe this very reason.
>
> We do, after earlier experiences with zfs, in an attempt to accommodate
> the behavior there.

OK, great.

> And overall, were you just saying that you saw sparseness with your
> system and zfs?

Yes, I am saying that. I fixed my test program to run du directly on
the sparse file.

I am writing 8 1k blocks, 8192 blocks into the file. So there are 8
blocks used of actual data, plus blocks used to store pointers and
information about the hole, in a FS-dependent, can't say it's wrong from
outside, kind of way.

~/SPARSE > sh -x _test

+ rm -rf DIR
+ mkdir DIR

+ cd DIR
+ dd if=/dev/zero of=SPARSE bs=1k count=8 seek=8192

8+0 records in
8+0 records out
8192 bytes transferred in 0.001 secs (8192000 bytes/sec)

+ ls -l
total 1
-rw-r--r-- 1 gdt users 8396800 Aug 13 19:16 SPARSE
+ du -k SPARSE
1 SPARSE

+ sync
+ sleep 30

+ ls -l
total 131
-rw-r--r-- 1 gdt users 8396800 Aug 13 19:16 SPARSE
+ du -k SPARSE
131 SPARSE
~/SPARSE > dc
10k
8396800 1024/p
8200.0000000000


Size: The file takes up 8200 * 1024 bytes, but uses 131 1K blocks of
storage, says du.

Timeliness: Somehow, after writing the file, but before sync, du says it
takes 1 block. That's probably because there is an uncommitted journal
entry (well, committed to the journal) for the new bits, and du may
operate on the committed fs, and then when a new root pointer is written
to commit the journal to the fs, du sees that. A deep zfs mystery
lurks, but the bottom line for bup testing is:

need to wait after writing, before du reflects the values that it will
have long term. (minutes later, it remains 131)

du on UFS or extN (?) may only reflect data blocks and not indirect
blocks, and du on zfs may be different. We could either have a "< 150
is pass" for now, or we could have a table of fs type and values.


NetBSD's tmpfs does not do sparse. Same test, du -k says 8200 blocks.

NetBSD's UFS2 (ffs) does sparse. Same test, 48, so 8 for the manifest
zeros and 40 for ?, probably indirect blocks and lists of block numbers
that code for fictitious-zero-block.

$ sh ~/SPARSE/_test

8+0 records in
8+0 records out
8192 bytes transferred in 0.001 secs (8192000 bytes/sec)

total 48
-rw-r--r-- 1 gdt wheel 8396800 Aug 13 19:24 SPARSE
48 SPARSE
total 48
-rw-r--r-- 1 gdt wheel 8396800 Aug 13 19:24 SPARSE
48 SPARSE