bup split -q produces output

[Brian Minton]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I'm trying to use bup split to back up a block device. Wanting to
see a progress bar, I used pv on the input. However, the stderr from
pv mixed with the stderr from bup split, and made it hard to read.

So, the command I used was the following:
pv /dev/sdb | bup split -q -n sda

I expected to see only the progress bar from pv, but no output from
bup. I actually saw bup display how many bytes it was splitting. The
output (on stderr) of both commands was jumbled together.

4464 kbytes
Splitting: 8504320
kbytes8.11GiB 0:23:12 [16.7MiB/s] [=====================>
Splitting: 8532992 kbytes

Splitting: 8555520 kbytes8.16GiB 0:23:17
[2.94MiB/s] [=====================>
Splitting: 8626176 kbytes8.23GiB 0:23:33 [6.32MiB/s]
[=====================>
Splitting: 9027584 kbytes8.61GiB 0:23:57 [15.7MiB/s]
[=======================>
Splitting: 9340928 kbytes8.91GiB 0:24:26 [4.28MiB/s]
[=======================>
Splitting: 10946560 kbytes10.4GiB 0:26:06 [20.8MiB/s]
[===========================>
Splitting: 11877376 kbytes
Splitting:
13900800 kbytes Splitting: 14541824
kbytes13.9GiB 0:29:52 [21.5MiB/s]
[=====================================>
Splitting: 15621120 kbytes
15.6GiB
0:31:37 [19.8MiB/s] [==========================================>
] 54% ETA 0:26:17Splitting: 17204224
kbytes16.4GiB 0:32:34 [17.0MiB/s]
[============================================>
Splitting: 17561600 kbytes16.7GiB 0:32:58 [4.70MiB/s]
[=============================================>
Splitting: 17592320 kbytes16.8GiB 0:33:04 [4.94MiB/s]
[=============================================>
Splitting: 17789952 kbytes17.0GiB 0:33:19 [17.6MiB/s]
[==============================================>
Splitting: 17859584 kbytes17.0GiB 0:33:23 [16.9MiB/s]
[==============================================>
Splitting: 18150400 kbytes
Splitting:
18617344 kbytes
Splitting: 23196672
kbytes Splitting: 23306240 kbytes

Splitting: 23384064 kbytes

Splitting: 25569280 kbytes Splitting: 25883648
kbytes
Splitting: 28786688 kbytes

Splitting: 29483008 kbytes
28.6GiB 0:39:18 [12.4MiB/s]
[=====================================================================
=========>] 100%


Thanks,
Brian Minton
-----BEGIN PGP SIGNATURE-----

iHUEAREIAB0WIQT5xLt2Dng/DewQpoprjrOgZc+6qQUCYZaZYAAKCRBrjrOgZc+6
qddqAP9o46DPbnLM6D2+fiqlNtjI3QDl67bl8y0oPld8YMUxhwD/cclr790IL10r
UMtLX0uG/8REbDwpiBh8gqhjCGZ/3kw=
=dDYL
-----END PGP SIGNATURE-----

[Stefan Monnier]

> So, the command I used was the following:
> pv /dev/sdb | bup split -q -n sda
>
> I expected to see only the progress bar from pv, but no output from
> bup.

If you don't want bup's output then you can do:

pv /dev/sdb | bup split -q -n sda >/dev/null 2>&1

And of course, you can use an actual file instead of /dev/null so that
you can review bup's output later on.


-- Stefan

[Brian Minton]

On 11/18/21 2:02 PM, Stefan Monnier wrote:
>> So, the command I used was the following:
>> pv /dev/sdb | bup split -q -n sda
>>
>> I expected to see only the progress bar from pv, but no output from
>> bup.
>
> If you don't want bup's output then you can do:
>
> pv /dev/sdb | bup split -q -n sda >/dev/null 2>&1

That is certainly a workaround, but the documentation definitely says
that the -q option should disable progress messages.


-q, --quiet
disable progress messages.


(from bup split --help)

[Rob Browning]

Brian Minton <br...@minton.name> writes:

> That is certainly a workaround, but the documentation definitely says
> that the -q option should disable progress messages.
>
> -q, --quiet
> disable progress messages.

Indeed - while it says that, the code actually calls qprogress()

def qprogress(s):
"""Calls progress() only if we haven't printed progress in a while.

This avoids overloading the stderr buffer with excess junk.
"""

Offhand, I'm leaning in favor of agreeing that "disable" doesn't really
align well with "only if we haven't ... in a while".

So if no one else disagrees, I might rework that to omit those messages
whenever -q has been specified.

Thanks
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Rob Browning]

Thanks to Brian Minton for reporting the issue.

Signed-off-by: Rob Browning <r...@defaultvalue.org>
---

Proposed.

lib/bup/cmd/split.py | 25 +++++++++++++++----------
1 file changed, 15 insertions(+), 10 deletions(-)

diff --git a/lib/bup/cmd/split.py b/lib/bup/cmd/split.py
index 11392e20..f1699b53 100755
--- a/lib/bup/cmd/split.py
+++ b/lib/bup/cmd/split.py
@@ -82,15 +82,19 @@ def main(argv):
else:
date = time.time()

- # Hack around lack of nonlocal vars in python 2
- total_bytes = [0]
- def prog(filenum, nbytes):
- total_bytes[0] += nbytes
- if filenum > 0:
- qprogress('Splitting: file #%d, %d kbytes\r'
- % (filenum+1, total_bytes[0] // 1024))
- else:
- qprogress('Splitting: %d kbytes\r' % (total_bytes[0] // 1024))
+ if opt.quiet:
+ def prog(filenum, nbytes):
+ pass
+ else:
+ # Hack around lack of nonlocal vars in python 2
+ total_bytes = [0]
+ def prog(filenum, nbytes):
+ total_bytes[0] += nbytes
+ if filenum > 0:
+ qprogress('Splitting: file #%d, %d kbytes\r'
+ % (filenum+1, total_bytes[0] // 1024))
+ else:
+ qprogress('Splitting: %d kbytes\r' % (total_bytes[0] // 1024))


is_reverse = environ.get(b'BUP_SERVER_REVERSE')
@@ -173,7 +177,8 @@ def main(argv):
progress=prog)
for (sha, size, level) in shalist:
out.write(hexlify(sha) + b'\n')
- reprogress()
+ if not opt.quiet:
+ reprogress()
elif opt.tree or opt.commit or opt.name:
if opt.name: # insert dummy_name which may be used as a restore target
mode, sha = \
--
2.30.2

[Rob Browning]

Thanks to Brian Minton for reporting the issue.

Signed-off-by: Rob Browning <r...@defaultvalue.org>

Tested-by: Rob Browning <r...@defaultvalue.org>
---

Rebased.

lib/bup/cmd/split.py | 26 ++++++++++++++++----------
1 file changed, 16 insertions(+), 10 deletions(-)

diff --git a/lib/bup/cmd/split.py b/lib/bup/cmd/split.py
index 87da1c28..34d9ca16 100755
--- a/lib/bup/cmd/split.py
+++ b/lib/bup/cmd/split.py
@@ -104,15 +104,20 @@ def opts_from_cmdline(argv):
return opt

def split(opt, files, parent, out, pack_writer):

- # Hack around lack of nonlocal vars in python 2
- total_bytes = [0]
- def prog(filenum, nbytes):
- total_bytes[0] += nbytes
- if filenum > 0:
- qprogress('Splitting: file #%d, %d kbytes\r'
- % (filenum+1, total_bytes[0] // 1024))
- else:
- qprogress('Splitting: %d kbytes\r' % (total_bytes[0] // 1024))
+
+ if opt.quiet:
+ def prog(filenum, nbytes):
+ pass
+ else:
+ # Hack around lack of nonlocal vars in python 2
+ total_bytes = [0]
+ def prog(filenum, nbytes):
+ total_bytes[0] += nbytes
+ if filenum > 0:
+ qprogress('Splitting: file #%d, %d kbytes\r'
+ % (filenum+1, total_bytes[0] // 1024))
+ else:
+ qprogress('Splitting: %d kbytes\r' % (total_bytes[0] // 1024))

new_blob = pack_writer.new_blob
new_tree = pack_writer.new_tree
@@ -122,7 +127,8 @@ def split(opt, files, parent, out, pack_writer):
progress=prog)
for sha, size, level in shalist:

[Brian Minton]

I've got a patch that I'm testing, but I haven't gotten the test to pass yet.

[Rob Browning]

Brian Minton <br...@minton.name> writes:

> I've got a patch that I'm testing, but I haven't gotten the test to pass yet.

Oh, did you see the v2 I posted yesterday?

[Brian Minton]

No, I didn't see it.  I'll take a look. 

[Rob Browning]

Brian Minton <br...@minton.name> writes:

> No, I didn't see it. I'll take a look.

I haven't looked closely, and I'm not sure, but I'd imagine the patches
might be similar. In any case, I'm happy to proceed with yours if you
like.

Thanks much