[PATCH] main/options: fix output string confusion
2 views
Skip to first unread message
Johannes Berg
Nov 22, 2021, 3:38:16 PM11/22/21
to bup-...@googlegroups.com
There are two bugs:
1) We assume that we can obtain the tty_width() from the
sys.stderr.fileno(), but that's only true if we haven't
ourselves redirected stderr to a pipe, as we do in
run_module_cmd() if fix_stderr is true.
Fix this by adding an (optional, defaulting to stderr)
argument for the file descriptor to tty_width(), and
using that in the places where we're running under the
redirection.
2) A simple typo/oversight meant that if the line was too
long (though due to the first bug assumed 70 chars) the
'width' chars would be removed at the beginning, rather
than actually showing 'width' chars. Fix that too.
These issues manifest themselves in e.g. 'bup save' showing
only the last few characters of the progress line if it was
longer than 70 characters.
Signed-off-by: Johannes Berg <joha...@sipsolutions.net>
---
lib/bup/main.py | 6 +++---
lib/bup/options.py | 6 +++---
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/lib/bup/main.py b/lib/bup/main.py
index 0c21c88f5703..226e463fd029 100755
--- a/lib/bup/main.py
+++ b/lib/bup/main.py
@@ -224,7 +224,7 @@ def print_clean_line(dest, content, width, sep=None):
assert not sep_rx.match(x)
content = b''.join(content)
if sep == b'\r' and len(content) > width:
- content = content[width:]
+ content = content[:width]
os.write(dest, content)
if len(content) < width:
os.write(dest, b' ' * (width - len(content)))
@@ -247,10 +247,10 @@ def filter_output(srcs, dests):
try:
while srcs:
ready_fds, _, _ = select.select(srcs, [], [])
- width = tty_width()
for fd in ready_fds:
buf = os.read(fd, 4096)
dest = dest_for[fd]
+ width = tty_width(dest)
if not buf:
srcs = tuple([x for x in srcs if x is not fd])
print_clean_line(dest, pending.pop(fd, []), width)
@@ -272,7 +272,7 @@ def filter_output(srcs, dests):
# Try to finish each of the streams
for fd, pending_items in compat.items(pending):
dest = dest_for[fd]
- width = tty_width()
+ width = tty_width(dest)
try:
print_clean_line(dest, pending_items, width)
except (EnvironmentError, EOFError) as ex:
diff --git a/lib/bup/options.py b/lib/bup/options.py
index 6f1f162d2b26..cfdc687b74ee 100644
--- a/lib/bup/options.py
+++ b/lib/bup/options.py
@@ -130,16 +130,16 @@ def _intify(v):
if not fcntl and termios:
- def _tty_width():
+ def _tty_width(fd=0):
return 70
else:
- def _tty_width():
+ def _tty_width(fd=sys.stderr.fileno()):
forced = os.environ.get('BUP_TTY_WIDTH', None)
if forced:
return int(forced)
s = struct.pack("HHHH", 0, 0, 0, 0)
try:
- s = fcntl.ioctl(sys.stderr.fileno(), termios.TIOCGWINSZ, s)
+ s = fcntl.ioctl(fd, termios.TIOCGWINSZ, s)
except IOError:
return 70
ysize, xsize, ypix, xpix = struct.unpack('HHHH', s)
--
2.33.1
1) We assume that we can obtain the tty_width() from the
sys.stderr.fileno(), but that's only true if we haven't
ourselves redirected stderr to a pipe, as we do in
run_module_cmd() if fix_stderr is true.
Fix this by adding an (optional, defaulting to stderr)
argument for the file descriptor to tty_width(), and
using that in the places where we're running under the
redirection.
2) A simple typo/oversight meant that if the line was too
long (though due to the first bug assumed 70 chars) the
'width' chars would be removed at the beginning, rather
than actually showing 'width' chars. Fix that too.
These issues manifest themselves in e.g. 'bup save' showing
only the last few characters of the progress line if it was
longer than 70 characters.
Signed-off-by: Johannes Berg <joha...@sipsolutions.net>
---
lib/bup/main.py | 6 +++---
lib/bup/options.py | 6 +++---
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/lib/bup/main.py b/lib/bup/main.py
index 0c21c88f5703..226e463fd029 100755
--- a/lib/bup/main.py
+++ b/lib/bup/main.py
@@ -224,7 +224,7 @@ def print_clean_line(dest, content, width, sep=None):
assert not sep_rx.match(x)
content = b''.join(content)
if sep == b'\r' and len(content) > width:
- content = content[width:]
+ content = content[:width]
os.write(dest, content)
if len(content) < width:
os.write(dest, b' ' * (width - len(content)))
@@ -247,10 +247,10 @@ def filter_output(srcs, dests):
try:
while srcs:
ready_fds, _, _ = select.select(srcs, [], [])
- width = tty_width()
for fd in ready_fds:
buf = os.read(fd, 4096)
dest = dest_for[fd]
+ width = tty_width(dest)
if not buf:
srcs = tuple([x for x in srcs if x is not fd])
print_clean_line(dest, pending.pop(fd, []), width)
@@ -272,7 +272,7 @@ def filter_output(srcs, dests):
# Try to finish each of the streams
for fd, pending_items in compat.items(pending):
dest = dest_for[fd]
- width = tty_width()
+ width = tty_width(dest)
try:
print_clean_line(dest, pending_items, width)
except (EnvironmentError, EOFError) as ex:
diff --git a/lib/bup/options.py b/lib/bup/options.py
index 6f1f162d2b26..cfdc687b74ee 100644
--- a/lib/bup/options.py
+++ b/lib/bup/options.py
@@ -130,16 +130,16 @@ def _intify(v):
if not fcntl and termios:
- def _tty_width():
+ def _tty_width(fd=0):
return 70
else:
- def _tty_width():
+ def _tty_width(fd=sys.stderr.fileno()):
forced = os.environ.get('BUP_TTY_WIDTH', None)
if forced:
return int(forced)
s = struct.pack("HHHH", 0, 0, 0, 0)
try:
- s = fcntl.ioctl(sys.stderr.fileno(), termios.TIOCGWINSZ, s)
+ s = fcntl.ioctl(fd, termios.TIOCGWINSZ, s)
except IOError:
return 70
ysize, xsize, ypix, xpix = struct.unpack('HHHH', s)
--
2.33.1
Nix
Nov 23, 2021, 2:27:05 PM11/23/21
to Johannes Berg, bup-...@googlegroups.com
On 22 Nov 2021, Johannes Berg outgrape:
> There are two bugs:
> 1) We assume that we can obtain the tty_width() from the
> sys.stderr.fileno(), but that's only true if we haven't
> ourselves redirected stderr to a pipe, as we do in
> run_module_cmd() if fix_stderr is true.
>
> Fix this by adding an (optional, defaulting to stderr)
> argument for the file descriptor to tty_width(), and
> using that in the places where we're running under the
> redirection.
>
> 2) A simple typo/oversight meant that if the line was too
> long (though due to the first bug assumed 70 chars) the
> 'width' chars would be removed at the beginning, rather
> than actually showing 'width' chars. Fix that too.
>
> These issues manifest themselves in e.g. 'bup save' showing
> only the last few characters of the progress line if it was
> longer than 70 characters.
As the reporter: I can confirm that this fixes things enough that I can
no longer observe anything wrong (in that, on my very wide terminals,
there is no longer any truncation at all :) ).
> There are two bugs:
> 1) We assume that we can obtain the tty_width() from the
> sys.stderr.fileno(), but that's only true if we haven't
> ourselves redirected stderr to a pipe, as we do in
> run_module_cmd() if fix_stderr is true.
>
> Fix this by adding an (optional, defaulting to stderr)
> argument for the file descriptor to tty_width(), and
> using that in the places where we're running under the
> redirection.
>
> 2) A simple typo/oversight meant that if the line was too
> long (though due to the first bug assumed 70 chars) the
> 'width' chars would be removed at the beginning, rather
> than actually showing 'width' chars. Fix that too.
>
> These issues manifest themselves in e.g. 'bup save' showing
> only the last few characters of the progress line if it was
> longer than 70 characters.
no longer observe anything wrong (in that, on my very wide terminals,
there is no longer any truncation at all :) ).
Nix
Nov 23, 2021, 2:30:22 PM11/23/21
to Johannes Berg, bup-...@googlegroups.com
On 22 Nov 2021, Johannes Berg said:
> There are two bugs:
> 1) We assume that we can obtain the tty_width() from the
> sys.stderr.fileno(), but that's only true if we haven't
> ourselves redirected stderr to a pipe, as we do in
> run_module_cmd() if fix_stderr is true.
>
> Fix this by adding an (optional, defaulting to stderr)
> argument for the file descriptor to tty_width(), and
> using that in the places where we're running under the
> redirection.
>
> 2) A simple typo/oversight meant that if the line was too
> long (though due to the first bug assumed 70 chars) the
> 'width' chars would be removed at the beginning, rather
> than actually showing 'width' chars. Fix that too.
>
> These issues manifest themselves in e.g. 'bup save' showing
> only the last few characters of the progress line if it was
> longer than 70 characters.
Hm. Not quite, actually -- while this fixes output from bup itself, the
> There are two bugs:
> 1) We assume that we can obtain the tty_width() from the
> sys.stderr.fileno(), but that's only true if we haven't
> ourselves redirected stderr to a pipe, as we do in
> run_module_cmd() if fix_stderr is true.
>
> Fix this by adding an (optional, defaulting to stderr)
> argument for the file descriptor to tty_width(), and
> using that in the places where we're running under the
> redirection.
>
> 2) A simple typo/oversight meant that if the line was too
> long (though due to the first bug assumed 70 chars) the
> 'width' chars would be removed at the beginning, rather
> than actually showing 'width' chars. Fix that too.
>
> These issues manifest themselves in e.g. 'bup save' showing
> only the last few characters of the progress line if it was
> longer than 70 characters.
output from bup bloom and bup midx, which used to be displayed, is now
invisible (the progress just... freezes while a bloom or midx is going
on). I suppose this should be fixed as well...
Johannes Berg
Nov 23, 2021, 2:38:07 PM11/23/21
to Nix, bup-...@googlegroups.com
On Tue, 2021-11-23 at 19:27 +0000, Nix wrote:
>
> As the reporter: I can confirm that this fixes things enough that I can
> no longer observe anything wrong (in that, on my very wide terminals,
> there is no longer any truncation at all :) ).
>
Oh, sorry, I hadn't realised you reported this too - my wife and myself
>
> As the reporter: I can confirm that this fixes things enough that I can
> no longer observe anything wrong (in that, on my very wide terminals,
> there is no longer any truncation at all :) ).
>
observed it the other day too. :)
johannes
Johannes Berg
Nov 23, 2021, 2:42:00 PM11/23/21
to Nix, bup-...@googlegroups.com
On Tue, 2021-11-23 at 19:30 +0000, Nix wrote:
>
> Hm. Not quite, actually -- while this fixes output from bup itself, the
> output from bup bloom and bup midx, which used to be displayed, is now
> invisible (the progress just... freezes while a bloom or midx is going
> on). I suppose this should be fixed as well...
>
I think there's a separate bug with this - I noticed something like this
>
> Hm. Not quite, actually -- while this fixes output from bup itself, the
> output from bup bloom and bup midx, which used to be displayed, is now
> invisible (the progress just... freezes while a bloom or midx is going
> on). I suppose this should be fixed as well...
>
too.
(and found your thread reporting this now, I guess I forgot about it)
The issue here seems to be interleaving from multiple processes not
creating a newline to get it to stick? At least that's what had been my
impression? Or maybe it shouldn't stick?
I'm honestly no longer sure what the behaviour was, or what it even
should be ... I did see the 'bup bloom' output while save was running
though (and that wasn't truncated).
Maybe it's time for me to actually get out my progressbar patch again
that attempted to clarify, unify, and prettify all this:
https://github.com/jmberg/bup/tree/progressbar
johannes
Johannes Berg
Nov 23, 2021, 2:47:25 PM11/23/21
to bup-...@googlegroups.com
On Mon, 2021-11-22 at 21:38 +0100, Johannes Berg wrote:
>
> while srcs:
> ready_fds, _, _ = select.select(srcs, [], [])
> - width = tty_width()
> for fd in ready_fds:
> buf = os.read(fd, 4096)
> dest = dest_for[fd]
> + width = tty_width(dest)
> if not buf:
> srcs = tuple([x for x in srcs if x is not fd])
> print_clean_line(dest, pending.pop(fd, []), width)
That should perhaps move into the "if not buf", it's probably not needed
>
> while srcs:
> ready_fds, _, _ = select.select(srcs, [], [])
> - width = tty_width()
> for fd in ready_fds:
> buf = os.read(fd, 4096)
> dest = dest_for[fd]
> + width = tty_width(dest)
> if not buf:
> srcs = tuple([x for x in srcs if x is not fd])
> print_clean_line(dest, pending.pop(fd, []), width)
outside. And I had to move it into the loop since it's a different fd -
obviously the terminal size won't change, but I need at least one fd ...
johannes
Nix
Nov 23, 2021, 2:55:24 PM11/23/21
to Johannes Berg, bup-...@googlegroups.com
On 23 Nov 2021, Johannes Berg stated:
and bup bloom was sent to stderr as usual: this too is a different sort
of progress bar, which given that both operations can take quite a long
time is fairly useful.
Your patch also hid the "... recieved from server" stuff which is
usually printed when getting indexes into the local cache.
Maybe all this has something to do with whether bup-server is in use? (I
always use it, even for local backups.)
--
NULL && (void)
> I'm honestly no longer sure what the behaviour was, or what it even
> should be ... I did see the 'bup bloom' output while save was running
> though (and that wasn't truncated).
Interesting. The behaviour in the past was that the output from bup midx
> should be ... I did see the 'bup bloom' output while save was running
> though (and that wasn't truncated).
and bup bloom was sent to stderr as usual: this too is a different sort
of progress bar, which given that both operations can take quite a long
time is fairly useful.
Your patch also hid the "... recieved from server" stuff which is
usually printed when getting indexes into the local cache.
Maybe all this has something to do with whether bup-server is in use? (I
always use it, even for local backups.)
--
NULL && (void)
Johannes Berg
Nov 23, 2021, 3:50:32 PM11/23/21
to Nix, bup-...@googlegroups.com
On Tue, 2021-11-23 at 19:55 +0000, Nix wrote:
> On 23 Nov 2021, Johannes Berg stated:
> > I'm honestly no longer sure what the behaviour was, or what it even
> > should be ... I did see the 'bup bloom' output while save was running
> > though (and that wasn't truncated).
>
> Interesting. The behaviour in the past was that the output from bup midx
> and bup bloom was sent to stderr as usual: this too is a different sort
> of progress bar, which given that both operations can take quite a long
> time is fairly useful.
Yes. I'm just not sure it was a temporary progress that went away when
> On 23 Nov 2021, Johannes Berg stated:
> > I'm honestly no longer sure what the behaviour was, or what it even
> > should be ... I did see the 'bup bloom' output while save was running
> > though (and that wasn't truncated).
>
> Interesting. The behaviour in the past was that the output from bup midx
> and bup bloom was sent to stderr as usual: this too is a different sort
> of progress bar, which given that both operations can take quite a long
> time is fairly useful.
the save resumed, or if both were somehow shown, or something else
entirely?
> Your patch also hid the "... recieved from server" stuff which is
> usually printed when getting indexes into the local cache.
> Maybe all this has something to do with whether bup-server is in use? (I
> always use it, even for local backups.)
>
server ...
johannes
Nix
Nov 26, 2021, 10:00:21 AM11/26/21
to Johannes Berg, bup-...@googlegroups.com
On 23 Nov 2021, Johannes Berg told this:
> On Tue, 2021-11-23 at 19:55 +0000, Nix wrote:
>> On 23 Nov 2021, Johannes Berg stated:
>> > I'm honestly no longer sure what the behaviour was, or what it even
>> > should be ... I did see the 'bup bloom' output while save was running
>> > though (and that wasn't truncated).
>>
>> Interesting. The behaviour in the past was that the output from bup midx
>> and bup bloom was sent to stderr as usual: this too is a different sort
>> of progress bar, which given that both operations can take quite a long
>> time is fairly useful.
>
> Yes. I'm just not sure it was a temporary progress that went away when
> the save resumed, or if both were somehow shown, or something else
> entirely?
It's temporary: it appears while the bloom/midx is in use, then vanishes
and is replaced by the usual progress bar.
>> Your patch also hid the "... recieved from server" stuff which is
>> usually printed when getting indexes into the local cache.
>
> Which my patch?
[PATCH] main/options: fix output string confusion
--
NULL && (void)
> On Tue, 2021-11-23 at 19:55 +0000, Nix wrote:
>> On 23 Nov 2021, Johannes Berg stated:
>> > I'm honestly no longer sure what the behaviour was, or what it even
>> > should be ... I did see the 'bup bloom' output while save was running
>> > though (and that wasn't truncated).
>>
>> Interesting. The behaviour in the past was that the output from bup midx
>> and bup bloom was sent to stderr as usual: this too is a different sort
>> of progress bar, which given that both operations can take quite a long
>> time is fairly useful.
>
> Yes. I'm just not sure it was a temporary progress that went away when
> the save resumed, or if both were somehow shown, or something else
> entirely?
and is replaced by the usual progress bar.
>> Your patch also hid the "... recieved from server" stuff which is
>> usually printed when getting indexes into the local cache.
>
> Which my patch?
--
NULL && (void)
Johannes Berg
Nov 28, 2021, 11:19:52 AM11/28/21
to Nix, bup-...@googlegroups.com
On Fri, 2021-11-26 at 15:00 +0000, Nix wrote:
> >
> > Yes. I'm just not sure it was a temporary progress that went away when
> > the save resumed, or if both were somehow shown, or something else
> > entirely?
>
> It's temporary: it appears while the bloom/midx is in use, then vanishes
> and is replaced by the usual progress bar.
Right, that's kind of what I thought. But that's not the case now?
> >
> > Yes. I'm just not sure it was a temporary progress that went away when
> > the save resumed, or if both were somehow shown, or something else
> > entirely?
>
> It's temporary: it appears while the bloom/midx is in use, then vanishes
> and is replaced by the usual progress bar.
I'm not sure I see how it wouldn't be. Hmm.
> > > Your patch also hid the "... recieved from server" stuff which is
> > > usually printed when getting indexes into the local cache.
> >
> > Which my patch?
>
> [PATCH] main/options: fix output string confusion
>
- perhaps the "client: received index suggestion: ..." message?
Or maybe "Receiving index from server ..."? But also that seems OK?
Unless somehow it doesn't get printed at all now? At least in my test:
https://p.sipsolutions.net/ac8f42895a3c43fa.txt
(store as lib/bup/cmd/test.py and run 'bup test')
So not sure what you're seeing? Hmm.
I do see at the end the message when it finishes is kind of left over
there, but I believe that was always the case, and it's hard to fix with
the current setup.
johannes
Nix
Nov 29, 2021, 9:48:26 AM11/29/21
to Johannes Berg, bup-...@googlegroups.com
On 28 Nov 2021, Johannes Berg verbalised:
> On Fri, 2021-11-26 at 15:00 +0000, Nix wrote:
>> >
>> > Yes. I'm just not sure it was a temporary progress that went away when
>> > the save resumed, or if both were somehow shown, or something else
>> > entirely?
>>
>> It's temporary: it appears while the bloom/midx is in use, then vanishes
>> and is replaced by the usual progress bar.
>
> Right, that's kind of what I thought. But that's not the case now?
Indeed not :( not here, anyway.
>> > > Your patch also hid the "... recieved from server" stuff which is
>> > > usually printed when getting indexes into the local cache.
>> >
>> > Which my patch?
>>
>> [PATCH] main/options: fix output string confusion
>>
> Hm. I'm not sure what "received from server" you're referring to though
> - perhaps the "client: received index suggestion: ..." message?
>
> Or maybe "Receiving index from server ..."? But also that seems OK?
> Unless somehow it doesn't get printed at all now? At least in my test:
Yes that :) sorry, braino.
> https://p.sipsolutions.net/ac8f42895a3c43fa.txt
>
> (store as lib/bup/cmd/test.py and run 'bup test')
>
>
> So not sure what you're seeing? Hmm.
A test that replicates this needs to be long enough that it does index
receipt in the middle as a consequence of doing a midx or midx/bloom
run: the index receipt that happens at the end of a run is fine. The
others never appear.
> I do see at the end the message when it finishes is kind of left over
> there, but I believe that was always the case, and it's hard to fix with
> the current setup.
That's the *intended case* for all the others :)
--
NULL && (void)
> On Fri, 2021-11-26 at 15:00 +0000, Nix wrote:
>> >
>> > Yes. I'm just not sure it was a temporary progress that went away when
>> > the save resumed, or if both were somehow shown, or something else
>> > entirely?
>>
>> It's temporary: it appears while the bloom/midx is in use, then vanishes
>> and is replaced by the usual progress bar.
>
> Right, that's kind of what I thought. But that's not the case now?
>> > > Your patch also hid the "... recieved from server" stuff which is
>> > > usually printed when getting indexes into the local cache.
>> >
>> > Which my patch?
>>
>> [PATCH] main/options: fix output string confusion
>>
> Hm. I'm not sure what "received from server" you're referring to though
> - perhaps the "client: received index suggestion: ..." message?
>
> Or maybe "Receiving index from server ..."? But also that seems OK?
> Unless somehow it doesn't get printed at all now? At least in my test:
> https://p.sipsolutions.net/ac8f42895a3c43fa.txt
>
> (store as lib/bup/cmd/test.py and run 'bup test')
>
>
> So not sure what you're seeing? Hmm.
receipt in the middle as a consequence of doing a midx or midx/bloom
run: the index receipt that happens at the end of a run is fine. The
others never appear.
> I do see at the end the message when it finishes is kind of left over
> there, but I believe that was always the case, and it's hard to fix with
> the current setup.
--
NULL && (void)
Johannes Berg
Nov 29, 2021, 1:05:32 PM11/29/21
to Nix, bup-...@googlegroups.com
On Mon, 2021-11-29 at 14:48 +0000, Nix wrote:
> On 28 Nov 2021, Johannes Berg verbalised:
>
> > On Fri, 2021-11-26 at 15:00 +0000, Nix wrote:
> > > >
> > > > Yes. I'm just not sure it was a temporary progress that went away when
> > > > the save resumed, or if both were somehow shown, or something else
> > > > entirely?
> > >
> > > It's temporary: it appears while the bloom/midx is in use, then vanishes
> > > and is replaced by the usual progress bar.
> >
> > Right, that's kind of what I thought. But that's not the case now?
>
> Indeed not :( not here, anyway.
Hm. I think it looked OK here.
> On 28 Nov 2021, Johannes Berg verbalised:
>
> > On Fri, 2021-11-26 at 15:00 +0000, Nix wrote:
> > > >
> > > > Yes. I'm just not sure it was a temporary progress that went away when
> > > > the save resumed, or if both were somehow shown, or something else
> > > > entirely?
> > >
> > > It's temporary: it appears while the bloom/midx is in use, then vanishes
> > > and is replaced by the usual progress bar.
> >
> > Right, that's kind of what I thought. But that's not the case now?
>
> Indeed not :( not here, anyway.
What does it look like for you?
>
> > Or maybe "Receiving index from server ..."? But also that seems OK?
> > Unless somehow it doesn't get printed at all now? At least in my test:
>
> Yes that :) sorry, braino.
>
> > https://p.sipsolutions.net/ac8f42895a3c43fa.txt
> >
> > (store as lib/bup/cmd/test.py and run 'bup test')
> >
> >
> > So not sure what you're seeing? Hmm.
>
> A test that replicates this needs to be long enough that it does index
> receipt in the middle as a consequence of doing a midx or midx/bloom
> run: the index receipt that happens at the end of a run is fine. The
> others never appear.
You can't do a index receipt as a consequence of midx/bloom - it's the
> > Or maybe "Receiving index from server ..."? But also that seems OK?
> > Unless somehow it doesn't get printed at all now? At least in my test:
>
> Yes that :) sorry, braino.
>
> > https://p.sipsolutions.net/ac8f42895a3c43fa.txt
> >
> > (store as lib/bup/cmd/test.py and run 'bup test')
> >
> >
> > So not sure what you're seeing? Hmm.
>
> A test that replicates this needs to be long enough that it does index
> receipt in the middle as a consequence of doing a midx or midx/bloom
> run: the index receipt that happens at the end of a run is fine. The
> others never appear.
other way around, after index receipt midx/bloom are done.
> > I do see at the end the message when it finishes is kind of left over
> > there, but I believe that was always the case, and it's hard to fix with
> > the current setup.
>
> That's the *intended case* for all the others :)
progress message?
I guess I'm still confused as to what you *are* seeing and what you
*expect* to see. I thought you were expecting to see all the messages of
"Receiving index ...", save progress, and midx/bloom process inbetween -
but all as intermediate until the end?
johannes
Nix
Nov 30, 2021, 9:25:11 AM11/30/21
to Johannes Berg, bup-...@googlegroups.com
On 29 Nov 2021, Johannes Berg stated:
> On Mon, 2021-11-29 at 14:48 +0000, Nix wrote:
>> On 28 Nov 2021, Johannes Berg verbalised:
>>
>> > On Fri, 2021-11-26 at 15:00 +0000, Nix wrote:
>> > > >
>> > > > Yes. I'm just not sure it was a temporary progress that went away when
>> > > > the save resumed, or if both were somehow shown, or something else
>> > > > entirely?
>> > >
>> > > It's temporary: it appears while the bloom/midx is in use, then vanishes
>> > > and is replaced by the usual progress bar.
>> >
>> > Right, that's kind of what I thought. But that's not the case now?
>>
>> Indeed not :( not here, anyway.
>
> Hm. I think it looked OK here.
>
> What does it look like for you?
Rather than the flow of directory names in verbose mode pausing
periodically and being replaced by output from bup bloom and bup midx
(and "Receiving index from server" afterwards), the output simply pauses
periodically and nothing is displayed (but top etc show that bup bloom
and bup midx are running as usual), following which the bup progress
output resumes as usual.
The only index reception message displayed is at the very end.
>> A test that replicates this needs to be long enough that it does index
>> receipt in the middle as a consequence of doing a midx or midx/bloom
>> run: the index receipt that happens at the end of a run is fine. The
>> others never appear.
>
> You can't do a index receipt as a consequence of midx/bloom - it's the
> other way around, after index receipt midx/bloom are done.
Er yeah obviously that's what I mean to say, yes! :) um.
> I guess I'm still confused as to what you *are* seeing and what you
> *expect* to see. I thought you were expecting to see all the messages of
> "Receiving index ...", save progress, and midx/bloom process inbetween -
> but all as intermediate until the end?
Yep. The (one-line) bup midx / bup bloom progress output should appear
and then vanish; the only thing which survives in the flow of directory
names is the index receipt messages (which also, uh, appear to no longer
appear). None of this happens any more: there is just silence.
(Is all this really important? Probably not, but it *is* nice to know
what these sometimes-very-long-running jobs are doing. I've had single
bup bloom runs take hours, sometimes.)
--
NULL && (void)
> On Mon, 2021-11-29 at 14:48 +0000, Nix wrote:
>> On 28 Nov 2021, Johannes Berg verbalised:
>>
>> > On Fri, 2021-11-26 at 15:00 +0000, Nix wrote:
>> > > >
>> > > > Yes. I'm just not sure it was a temporary progress that went away when
>> > > > the save resumed, or if both were somehow shown, or something else
>> > > > entirely?
>> > >
>> > > It's temporary: it appears while the bloom/midx is in use, then vanishes
>> > > and is replaced by the usual progress bar.
>> >
>> > Right, that's kind of what I thought. But that's not the case now?
>>
>> Indeed not :( not here, anyway.
>
> Hm. I think it looked OK here.
>
> What does it look like for you?
periodically and being replaced by output from bup bloom and bup midx
(and "Receiving index from server" afterwards), the output simply pauses
periodically and nothing is displayed (but top etc show that bup bloom
and bup midx are running as usual), following which the bup progress
output resumes as usual.
The only index reception message displayed is at the very end.
>> A test that replicates this needs to be long enough that it does index
>> receipt in the middle as a consequence of doing a midx or midx/bloom
>> run: the index receipt that happens at the end of a run is fine. The
>> others never appear.
>
> You can't do a index receipt as a consequence of midx/bloom - it's the
> other way around, after index receipt midx/bloom are done.
> I guess I'm still confused as to what you *are* seeing and what you
> *expect* to see. I thought you were expecting to see all the messages of
> "Receiving index ...", save progress, and midx/bloom process inbetween -
> but all as intermediate until the end?
and then vanish; the only thing which survives in the flow of directory
names is the index receipt messages (which also, uh, appear to no longer
appear). None of this happens any more: there is just silence.
(Is all this really important? Probably not, but it *is* nice to know
what these sometimes-very-long-running jobs are doing. I've had single
bup bloom runs take hours, sometimes.)
--
NULL && (void)
Johannes Berg
Nov 30, 2021, 11:29:19 AM11/30/21
to Nix, bup-...@googlegroups.com
On Tue, 2021-11-30 at 14:25 +0000, Nix wrote:
>
> Rather than the flow of directory names in verbose mode
>
Wait, flow of directory names? Hmm.
>
> Rather than the flow of directory names in verbose mode
>
> pausing
> periodically and being replaced by output from bup bloom and bup midx
> (and "Receiving index from server" afterwards),
>
> the output simply pauses
> periodically and nothing is displayed (but top etc show that bup bloom
> and bup midx are running as usual), following which the bup progress
> output resumes as usual.
> The only index reception message displayed is at the very end.
> > > A test that replicates this needs to be long enough that it does index
> > > receipt in the middle as a consequence of doing a midx or midx/bloom
> > > run: the index receipt that happens at the end of a run is fine. The
> > > others never appear.
> >
> > You can't do a index receipt as a consequence of midx/bloom - it's the
> > other way around, after index receipt midx/bloom are done.
>
> Er yeah obviously that's what I mean to say, yes! :) um.
>
> > I guess I'm still confused as to what you *are* seeing and what you
> > *expect* to see. I thought you were expecting to see all the messages of
> > "Receiving index ...", save progress, and midx/bloom process inbetween -
> > but all as intermediate until the end?
>
> Yep. The (one-line) bup midx / bup bloom progress output should appear
> and then vanish; the only thing which survives in the flow of directory
> names is the index receipt messages (which also, uh, appear to no longer
> appear). None of this happens any more: there is just silence.
How do you control the verbosity? Are you using "bup save -vvv" or so?
Or are you using "BUP_DEBUG=3 bup save"?
johannes
Johannes Berg
Nov 30, 2021, 11:35:17 AM11/30/21
to Nix, bup-...@googlegroups.com
On Tue, 2021-11-30 at 17:29 +0100, Johannes Berg wrote:
>
> How do you control the verbosity? Are you using "bup save -vvv" or so?
> Or are you using "BUP_DEBUG=3 bup save"?
>
Oh. Maybe you're using "bup --debug save ..."? But even that seems to
>
> How do you control the verbosity? Are you using "bup save -vvv" or so?
> Or are you using "BUP_DEBUG=3 bup save"?
>
work, I was thinking that might not pass it on through the environment
properly with the new binary setup.
johannes
Johannes Berg
Nov 30, 2021, 11:46:27 AM11/30/21
to Nix, bup-...@googlegroups.com
at all? I can kind of see similar behaviour in that there's no output
while it's running, and then all the output appears but is immediately
pushed out by the next file/directory messages ...
But I wasn't seeing this earlier, so it's confusing.
johannes
Johannes Berg
Nov 30, 2021, 12:09:15 PM11/30/21
to Nix, bup-...@googlegroups.com
OK, I tracked this down to the new binary/pipe setup.
Basically, it works in commit b969967873d0bf51238181f62382d3883e23eae1.
And then it's broken in commit 815bb1e6e36126de106331121e27e2f73883e649.
The reason seems to be the way the pipes are set up, and if save is an
external command then both midx and save redirect to the same filtering
machinery running in the main bup binary.
If, however, save is a module, then midx appears to redirect to save,
and then somehow stuff doesn't get out properly.
No time to dig further right now, but maybe that gives somebody some
thoughts.
johannes
Basically, it works in commit b969967873d0bf51238181f62382d3883e23eae1.
And then it's broken in commit 815bb1e6e36126de106331121e27e2f73883e649.
The reason seems to be the way the pipes are set up, and if save is an
external command then both midx and save redirect to the same filtering
machinery running in the main bup binary.
If, however, save is a module, then midx appears to redirect to save,
and then somehow stuff doesn't get out properly.
No time to dig further right now, but maybe that gives somebody some
thoughts.
johannes
Johannes Berg
Nov 30, 2021, 12:59:16 PM11/30/21
to Nix, bup-...@googlegroups.com
here causes the subprocess to no longer see the stderr as a tty, and so
the C code in _helpers doesn't even output anything.
So we need to export the variable, which is a bit tricky.
https://p.sipsolutions.net/86c86b192b69f4ad.txt
or even just
https://p.sipsolutions.net/d096cc8e808a8b57.txt
That seems to work for me.
johannes
Nix
Dec 2, 2021, 3:47:56 PM12/2/21
to Johannes Berg, bup-...@googlegroups.com
On 30 Nov 2021, Johannes Berg stated:
> On Tue, 2021-11-30 at 18:09 +0100, Johannes Berg wrote:
>> OK, I tracked this down to the new binary/pipe setup.
>>
>> Basically, it works in commit b969967873d0bf51238181f62382d3883e23eae1.
>> And then it's broken in commit 815bb1e6e36126de106331121e27e2f73883e649.
>>
>> The reason seems to be the way the pipes are set up, and if save is an
>> external command then both midx and save redirect to the same filtering
>> machinery running in the main bup binary.
>>
>> If, however, save is a module, then midx appears to redirect to save,
>> and then somehow stuff doesn't get out properly.
>>
>> No time to dig further right now, but maybe that gives somebody some
>> thoughts.
>
> OK I found the issue - the problem is that the top-level redirection
> here causes the subprocess to no longer see the stderr as a tty, and so
> the C code in _helpers doesn't even output anything.
Ugh! I never thought of looking *there* at all!
> So we need to export the variable, which is a bit tricky.
>
> https://p.sipsolutions.net/86c86b192b69f4ad.txt
>
> or even just
>
>
> https://p.sipsolutions.net/d096cc8e808a8b57.txt
>
> That seems to work for me.
I'll try it out tomorrow.
--
NULL && (void)
> On Tue, 2021-11-30 at 18:09 +0100, Johannes Berg wrote:
>> OK, I tracked this down to the new binary/pipe setup.
>>
>> Basically, it works in commit b969967873d0bf51238181f62382d3883e23eae1.
>> And then it's broken in commit 815bb1e6e36126de106331121e27e2f73883e649.
>>
>> The reason seems to be the way the pipes are set up, and if save is an
>> external command then both midx and save redirect to the same filtering
>> machinery running in the main bup binary.
>>
>> If, however, save is a module, then midx appears to redirect to save,
>> and then somehow stuff doesn't get out properly.
>>
>> No time to dig further right now, but maybe that gives somebody some
>> thoughts.
>
> OK I found the issue - the problem is that the top-level redirection
> here causes the subprocess to no longer see the stderr as a tty, and so
> the C code in _helpers doesn't even output anything.
> So we need to export the variable, which is a bit tricky.
>
> https://p.sipsolutions.net/86c86b192b69f4ad.txt
>
> or even just
>
>
> https://p.sipsolutions.net/d096cc8e808a8b57.txt
>
> That seems to work for me.
--
NULL && (void)
Johannes Berg
Dec 2, 2021, 3:53:32 PM12/2/21
to Nix, bup-...@googlegroups.com
On Thu, 2021-12-02 at 20:47 +0000, Nix wrote:
> >
> > OK I found the issue - the problem is that the top-level redirection
> > here causes the subprocess to no longer see the stderr as a tty, and so
> > the C code in _helpers doesn't even output anything.
>
> Ugh! I never thought of looking *there* at all!
Me neither! But then I went back in the git history, saw the messages
> >
> > OK I found the issue - the problem is that the top-level redirection
> > here causes the subprocess to no longer see the stderr as a tty, and so
> > the C code in _helpers doesn't even output anything.
>
> Ugh! I never thought of looking *there* at all!
and went looking ...
> > So we need to export the variable, which is a bit tricky.
> >
> > https://p.sipsolutions.net/86c86b192b69f4ad.txt
> >
> > or even just
> >
> >
> > https://p.sipsolutions.net/d096cc8e808a8b57.txt
> >
> > That seems to work for me.
>
> I'll try it out tomorrow.
>
way thanks!
johannes
Reply all
Reply to author
Forward
0 new messages