info
Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
[gentoo-user] Tailing compressed build logs
7 views
Skip to first unread message
Bryan Gardiner
Mar 6, 2023, 11:40:04 PM3/6/23
to
Hi folks,
How can I follow Portage's compressed build logs in real time as they
are generated?
I keep build logs and use FEATURES=compress-build-logs so that they
don't get too large. I can peek at how a build is going with zless on
build.log.gz, which doesn't update (understandably), but I would
really like to be able to watch a log with some "tail -f" equivalent.
I get streaming output with
tail -c +1 -f build.log.gz | od -t x1
but the following hangs with no output:
tail -c +1 -f build.log.gz | gunzip
even with a build log that is 72KB compressed (2.4MB uncompressed),
which should be larger than any pipe buffers... Any idea why gunzip
can't handle this, or what I should I should be doing instead?
Thanks,
Bryan
How can I follow Portage's compressed build logs in real time as they
are generated?
I keep build logs and use FEATURES=compress-build-logs so that they
don't get too large. I can peek at how a build is going with zless on
build.log.gz, which doesn't update (understandably), but I would
really like to be able to watch a log with some "tail -f" equivalent.
I get streaming output with
tail -c +1 -f build.log.gz | od -t x1
but the following hangs with no output:
tail -c +1 -f build.log.gz | gunzip
even with a build log that is 72KB compressed (2.4MB uncompressed),
which should be larger than any pipe buffers... Any idea why gunzip
can't handle this, or what I should I should be doing instead?
Thanks,
Bryan
Mickaël Bucas
Mar 7, 2023, 3:00:04 AM3/7/23
to
Reading the man page, "zless" is just a wrapper around "less".
You can check with:
$ file $(which zless)
/usr/bin/zless: POSIX shell script, ASCII text executable
$ less $(which zless)
So it should support the same options including typing "F" at the end
of a file to keep trying to read when the end of file is reached.
I made a small test, but it didn't work:
# Create a growing file
$ yes | nl | gzip > zless-test.gz &
# Try to follow at the end
$ zless zless-test.gz
With ">" to go to the end and "F" to continue, I didn't get the
expected behavior, it stood still at the point I was viewing.
I don't know if it's really a bug or if I made a mistake...
(Don't forget to stop the growing file :) )
Best regards
Mickaël Bucas
Andreas Stiasny
Mar 8, 2023, 8:20:05 AM3/8/23
to
On 07.03.23 05:35, Bryan Gardiner wrote:
> but the following hangs with no output:
>
> tail -c +1 -f build.log.gz | gunzip
I think you should either use gunzip -f --stdout instead of just gunzip
> but the following hangs with no output:
>
> tail -c +1 -f build.log.gz | gunzip
or use zcat.
Andreas
Bryan Gardiner
Mar 9, 2023, 2:50:04 AM3/9/23
to
- Bryan
Bryan Gardiner
Mar 9, 2023, 2:50:06 AM3/9/23
to
Hi,
On Tue, 07 Mar 2023 09:01:34 +0000
Michael <confa...@kintzios.com> wrote:
> On Tuesday, 7 March 2023 07:52:01 GMT Mickaël Bucas wrote:
> > Le mar. 7 mars 2023 à 05:36, Bryan Gardiner <b...@khumba.net> a
> > écrit :
> > > Hi folks,
> > >
> > > How can I follow Portage's compressed build logs in real time as
> > > they are generated?
> > >
> > > ...
> > So it should support the same options including typing "F" at the
> > end of a file to keep trying to read when the end of file is
> > reached.
> >
> > I made a small test, but it didn't work:
> > # Create a growing file
> > $ yes | nl | gzip > zless-test.gz &
> > # Try to follow at the end
> > $ zless zless-test.gz
> >
> > With ">" to go to the end and "F" to continue, I didn't get the
> > expected behavior, it stood still at the point I was viewing.
> > I don't know if it's really a bug or if I made a mistake...
> > (Don't forget to stop the growing file :) )
I suppose it makes sense that a simple "gunzip -c file.gz | less"
doesn't work, assuming zless is equivalent to that. gunzip sees the
end of the file and exits rather than waiting for additional content.
Then less sees its pipe close.
So I think the "tail -c +1 -f build.log.gz" bit is needed, and appears
to work since od shows data appended every few seconds.
> You could try:
>
> tail -c +1 -f build.log.gz | gunzip | less
>
> I think it should work, but I haven't tried it.
Thanks, but I suspect that less is still at the whim of gunzip here,
and gunzip isn't producing anything.
Maybe gunzip just has a massive output buffer. If I do something a bit
more complicated:
(N=0; while true; do
(( N++ ))
echo "$N" | gzip
if (( N % 4000 == 0 )); then sleep 1; fi
done) | gunzip | od -A d -t x1
then nothing is shown for 15 seconds or so, then od quickly dumps
51952 bytes of hex output, then another long pause before dumping
another 60480 bytes of hex. If I try this with "pigz -cd" instead of
"gunzip" then output starts flowing after just ~400 bytes, but sadly
that still doesn't work for build.log.gz.
Anyway, I've had another thought, that since portage is flushing lines
to the gzipped log fairly frequently, it's probably not getting the
best compression ratio. So I went and tested recompressing some logs,
and got a decent improvement:
- nodejs-18.9.1: build.log.gz is 112KB, recompressed it's 68.9KB
(38.5% savings).
- qtwebengine-5.15.8_p20230112: build.log.gz is 1.44MB, recompressed
it's 1.01MB (29.9% savings).
- gtk+-3.24.37: build.log.gz is 157KB, recompressed it's 120KB
(23.6% savings).
So maybe I'm better off abandoning this feature, and setting something
up to compress build logs after the fact. Then good ol' tail -f will
work just fine.
Thanks, cheers,
Bryan
On Tue, 07 Mar 2023 09:01:34 +0000
Michael <confa...@kintzios.com> wrote:
> On Tuesday, 7 March 2023 07:52:01 GMT Mickaël Bucas wrote:
> > Le mar. 7 mars 2023 à 05:36, Bryan Gardiner <b...@khumba.net> a
> > écrit :
> > > Hi folks,
> > >
> > > How can I follow Portage's compressed build logs in real time as
> > > they are generated?
> > >
> > >
> > > tail -c +1 -f build.log.gz | od -t x1
> > >
> > > but the following hangs with no output:
> > > tail -c +1 -f build.log.gz | gunzip
> >
> > > tail -c +1 -f build.log.gz | od -t x1
> > >
> > > but the following hangs with no output:
> > > tail -c +1 -f build.log.gz | gunzip
> >
> > Reading the man page, "zless" is just a wrapper around "less".
Neat, zless is a lot simpler than I thought.
> > So it should support the same options including typing "F" at the
> > end of a file to keep trying to read when the end of file is
> > reached.
> >
> > I made a small test, but it didn't work:
> > # Create a growing file
> > $ yes | nl | gzip > zless-test.gz &
> > # Try to follow at the end
> > $ zless zless-test.gz
> >
> > With ">" to go to the end and "F" to continue, I didn't get the
> > expected behavior, it stood still at the point I was viewing.
> > I don't know if it's really a bug or if I made a mistake...
> > (Don't forget to stop the growing file :) )
doesn't work, assuming zless is equivalent to that. gunzip sees the
end of the file and exits rather than waiting for additional content.
Then less sees its pipe close.
So I think the "tail -c +1 -f build.log.gz" bit is needed, and appears
to work since od shows data appended every few seconds.
> You could try:
>
> tail -c +1 -f build.log.gz | gunzip | less
>
> I think it should work, but I haven't tried it.
and gunzip isn't producing anything.
Maybe gunzip just has a massive output buffer. If I do something a bit
more complicated:
(N=0; while true; do
(( N++ ))
echo "$N" | gzip
if (( N % 4000 == 0 )); then sleep 1; fi
done) | gunzip | od -A d -t x1
then nothing is shown for 15 seconds or so, then od quickly dumps
51952 bytes of hex output, then another long pause before dumping
another 60480 bytes of hex. If I try this with "pigz -cd" instead of
"gunzip" then output starts flowing after just ~400 bytes, but sadly
that still doesn't work for build.log.gz.
Anyway, I've had another thought, that since portage is flushing lines
to the gzipped log fairly frequently, it's probably not getting the
best compression ratio. So I went and tested recompressing some logs,
and got a decent improvement:
- nodejs-18.9.1: build.log.gz is 112KB, recompressed it's 68.9KB
(38.5% savings).
- qtwebengine-5.15.8_p20230112: build.log.gz is 1.44MB, recompressed
it's 1.01MB (29.9% savings).
- gtk+-3.24.37: build.log.gz is 157KB, recompressed it's 120KB
(23.6% savings).
So maybe I'm better off abandoning this feature, and setting something
up to compress build logs after the fact. Then good ol' tail -f will
work just fine.
Thanks, cheers,
Bryan
Andreas Stiasny
Mar 9, 2023, 9:40:04 AM3/9/23
to
On 09.03.23 08:46, Bryan Gardiner wrote:
> Hi, thanks! --force and --stdout don't seem to help in this case.
Sorry, I was wrong. The problem is not the actual use of gunzip but the
> Hi, thanks! --force and --stdout don't seem to help in this case.
fact that you are trying to decompress a file without reading it from
the beginning. Usually this is not possible.
zcat build.log.gz | tail -n 30
would show you the last 30 lines but no new content because zcat or
gunzip is not waiting for new input. You could try gztool
https://github.com/circulosmeos/gztool/
"gztool -T" should work like "tail -f".
Andreas
Peter Humphrey
Mar 9, 2023, 10:50:05 AM3/9/23
to
On Thursday, 9 March 2023 14:31:32 GMT Andreas Stiasny wrote:
>You could try gztool
>
> https://github.com/circulosmeos/gztool/
>
> "gztool -T" should work like "tail -f".
That looks very handy. But:
>You could try gztool
>
> https://github.com/circulosmeos/gztool/
>
> "gztool -T" should work like "tail -f".
Compiling it from the github source fails because I haven't found all the
prerequisites.
Fetching a Debian compiled version and running deb2targz on it ends up with a
.so file, which where to put?
I haven't found it in any Gentoo overlays.
How can I get hold of a version for Gentoo?
--
Regards,
Peter.
Matt Connell
Mar 9, 2023, 11:00:05 AM3/9/23
to
On Thu, 2023-03-09 at 15:49 +0000, Peter Humphrey wrote:
> Fetching a Debian compiled version and running deb2targz on it ends up with a
> .so file, which where to put?
app-arch/deb2targz exists. Would probably satisfy the need.
> Fetching a Debian compiled version and running deb2targz on it ends up with a
> .so file, which where to put?
Dale
Mar 9, 2023, 11:10:04 AM3/9/23
to
there.
https://github.com/gentoo-mirror/slonko
It appears to be called gztool.
Hope that helps.
Dale
:-) :-)
Peter Humphrey
Mar 9, 2023, 12:10:04 PM3/9/23
to
--
Regards,
Peter.
Bryan Gardiner
Mar 9, 2023, 11:20:04 PM3/9/23
to
find, thanks a bunch!
- Bryan
0 new messages