apt-get -qq install

[Victor Sudakov]

Dear Colleagues,

I'm trying to quiet apt's output by using `apt-get -qqy` in a CI/CD
pipeline, however I still see ugly stuff like this in my CI/CD log:

Selecting previously unselected package php-common.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%

and this:

Scanning processes... [ ]
Scanning processes... [ ]
Scanning processes... [ ]
Scanning processes... [= ]
Scanning processes... [= ]
Scanning processes... [== ]
Scanning processes... [== ]
Scanning processes... [=== ]
Scanning processes... [=== ]
Scanning processes... [==== ]
Scanning processes... [==== ]
Scanning processes... [===== ]
Scanning processes... [===== ]
Scanning processes... [====== ]
Scanning processes... [====== ]
Scanning processes... [======= ]
Scanning processes... [======= ]
Scanning processes... [======== ]
Scanning processes... [======== ]
Scanning processes... [========= ]
Scanning processes... [========= ]

Don't you think the `-qq` modifier should spare me these things?

I don't actually like the idea of redirecting apt-get's output to
/dev/null because I want to see the list of packages installed, but without
these pseudographics. Do you think it's possible? Any ideas?

--
Victor Sudakov VAS4-RIPE
http://vas.tomsk.ru/
2:5005/49@fidonet

[David Wright]

On Tue 02 Aug 2022 at 18:27:22 (+0000), Victor Sudakov wrote:
> I'm trying to quiet apt's output by using `apt-get -qqy` in a CI/CD
> pipeline, however I still see ugly stuff like this in my CI/CD log:
>
> Selecting previously unselected package php-common.
> (Reading database ...
> (Reading database ... 5%

[ … ]
>
> and this:
>
> Scanning processes... [ ]
> Scanning processes... [= ]
[ … ]

>
> Don't you think the `-qq` modifier should spare me these things?
>
> I don't actually like the idea of redirecting apt-get's output to
> /dev/null because I want to see the list of packages installed, but without
> these pseudographics. Do you think it's possible? Any ideas?

The packages installed and the terminal output are logged in
/var/log/apt/{history,term}.log respectively (which I never rotate).

Cheers,
David.

[Victor Sudakov]

Which will be lost in a CI/CD environment because the VM image is ephemeral.

BTW I've tried setting `sudo DEBIAN_FRONTEND=noninteractive apt-get -qqy --no-install-recommends install ...` and
`sudo apt-get -qqy -o Dpkg::Use-Pty=0 ...` to no avail, the rubbish is still there.

[davidson]

On Tue, 2 Aug 2022 Victor Sudakov wrote:
> Dear Colleagues,
>
> I'm trying to quiet apt's output by using `apt-get -qqy` in a CI/CD
> pipeline, however I still see ugly stuff like this in my CI/CD log:
>
> Selecting previously unselected package php-common.
> (Reading database ...
> (Reading database ... 5%
> (Reading database ... 10%

[snip]

>
> and this:
>
> Scanning processes... [ ]
> Scanning processes... [ ]
> Scanning processes... [ ]
> Scanning processes... [= ]
> Scanning processes... [= ]
> Scanning processes... [== ]
> Scanning processes... [== ]

[snip]

> Don't you think the `-qq` modifier should spare me these things?

I would expect so too.

> I don't actually like the idea of redirecting apt-get's output to
> /dev/null because I want to see the list of packages installed, but without
> these pseudographics. Do you think it's possible? Any ideas?

In the file

/usr/share/doc/apt/examples/configure-index.gz

I notice there is a stanza that begins like so:

quiet "<INT>" {
NoUpdate "<BOOL>"; // never update progress information - included in -q=1
NoProgress "<BOOL>"; // disables the 0% -> 100% progress on cache generation and stuff

I don't know whether this will help you, but it does look suggestive.

--
Ce qui est important est rarement urgent
et ce qui est urgent est rarement important
-- Dwight David Eisenhower

[Victor Sudakov]

Thank you, how do you activate this option? I've just tried
`apt-get -o quiet::NoProgress=true -qqy ...` but the "Reading database ... 5%"
stuff is still there.

[David Christensen]

I prefer the idea of collecting all of the data and then writing queries
and reports to print the interesting bits. If you keep all of the data,
you can apply future queries and reports to past data. But if you
filter the data up front and save one report format, your future options
are limited.


David

[Victor Sudakov]

David Christensen wrote:
> >
> > I don't actually like the idea of redirecting apt-get's output to
> > /dev/null because I want to see the list of packages installed, but without
> > these pseudographics. Do you think it's possible? Any ideas?
>
>
> I prefer the idea of collecting all of the data and then writing queries
> and reports to print the interesting bits. If you keep all of the data,
> you can apply future queries and reports to past data. But if you
> filter the data up front and save one report format, your future options
> are limited.

(Un)fortunately this is a CI/CD pipeline, the VM and its data will be
gone forever after the build. Unless I care to keep apt output as an
artifact somewhere which is IMHO an overkill. I just want an concise
CI/CD log without interactive bells and whistles like progress
indicators.

[David Christensen]

Without seeing your CI/CD pipeine -- create a patterns file containing
regular expressions that match the lines you want removed, then add a
grep(1) filter into the pipeline:

2022-08-02 16:58:46 dpchrist@laalaa ~
$ cat remove-patterns.txt
bar
quux

2022-08-02 16:58:50 dpchrist@laalaa ~
$ echo -e "foo\nbar\nbaz\nquux" | egrep -v -f remove-patterns.txt
foo
baz


David

[Victor Sudakov]

David Christensen wrote:
> On 8/2/22 15:53, Victor Sudakov wrote:
> >
> > (Un)fortunately this is a CI/CD pipeline, the VM and its data will be
> > gone forever after the build. Unless I care to keep apt output as an
> > artifact somewhere which is IMHO an overkill. I just want an concise
> > CI/CD log without interactive bells and whistles like progress
> > indicators.
>
>
> Without seeing your CI/CD pipeine -- create a patterns file containing
> regular expressions that match the lines you want removed, then add a
> grep(1) filter into the pipeline:

Of course I can think of many workarounds. The original question was
however why "-qq" was not working and what I was missing in "apt"
usage.

[davidson]

On Tue, 2 Aug 2022 Victor Sudakov wrote:

I would do it that way too.

> but the "Reading database ... 5%" stuff is still there.

Yeah, after some unsatisfying experimentation I've had no luck either.

Maybe this bug report will interest you:

#539617 - dpkg: Add option to suppress progress display while reading database
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=539617

[Tim Woodall]

On Tue, 2 Aug 2022, Victor Sudakov wrote:

> David Wright wrote:
>> On Tue 02 Aug 2022 at 18:27:22 (+0000), Victor Sudakov wrote:
>>> I'm trying to quiet apt's output by using `apt-get -qqy` in a CI/CD
>>> pipeline, however I still see ugly stuff like this in my CI/CD log:
>>>
>>> Selecting previously unselected package php-common.
>>> (Reading database ...
>>> (Reading database ... 5%

>> [ ? ]

>>>
>>> and this:
>>>
>>> Scanning processes... [ ]
>>> Scanning processes... [= ]

>> [ ? ]

>>>
>>> Don't you think the `-qq` modifier should spare me these things?
>>>
>>> I don't actually like the idea of redirecting apt-get's output to
>>> /dev/null because I want to see the list of packages installed, but without
>>> these pseudographics. Do you think it's possible? Any ideas?
>>
>> The packages installed and the terminal output are logged in
>> /var/log/apt/{history,term}.log respectively (which I never rotate).
>
> Which will be lost in a CI/CD environment because the VM image is ephemeral.
>
> BTW I've tried setting `sudo DEBIAN_FRONTEND=noninteractive apt-get -qqy --no-install-recommends install ...` and
> `sudo apt-get -qqy -o Dpkg::Use-Pty=0 ...` to no avail, the rubbish is still there.
>
>

something strange - I see this in my console log:

$ less essential-phase1-sid-i386.log
/bin/sh: 0: can't access tty; job control turned off
# # # Get:1 http://ftp.uk.debian.org/debian sid InRelease [192 kB]
Get:2 http://ftp.uk.debian.org/debian sid/main Sources [9857 kB]
Get:3 http://ftp.uk.debian.org/debian sid/main i386 Packages [9185 kB]
Get:4 http://ftp.uk.debian.org/debian sid/main Translation-en [6861 kB]
Fetched 26.1 MB in 20s (1311 kB/s)
Reading package lists...
# # Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following packages will be upgraded:
apt base-files libc-bin util-linux
4 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 3690 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://ftp.uk.debian.org/debian sid/main i386 base-files i386 12.2 [70.1 kB]
Get:2 http://ftp.uk.debian.org/debian sid/main i386 util-linux i386 2.38-6 [1258 kB]
Get:3 http://ftp.uk.debian.org/debian sid/main i386 apt i386 2.5.2 [1522 kB]
Get:4 http://ftp.uk.debian.org/debian sid/main i386 libc-bin i386 2.33-8 [840 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 3690 kB in 2s (1981 kB/s)
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory)
(Reading database ... 6645 files and directories currently installed.)
Preparing to unpack .../base-files_12.2_i386.deb ...
Unpacking base-files (12.2) over (12.2~tjw) ...
Setting up base-files (12.2) ...
(Reading database ... 6645 files and directories currently installed.)
...


Which is coming from:
chroot ${workdir} <<CHEOF
set -eu
export DEBIAN_FRONTEND=noninteractive
apt-get update
echo "man-db man-db/auto-update boolean false" | debconf-set-selections
apt-get -o APT::Install-Recommends=false -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" -y upgrade </dev/null

[Curt]

On 2022-08-02, Victor Sudakov <v...@sibptus.ru> wrote:
>
> Dear Colleagues,
>
> I'm trying to quiet apt's output by using `apt-get -qqy` in a CI/CD
> pipeline, however I still see ugly stuff like this in my CI/CD log:

Quiet level 2 implies -y.

> Selecting previously unselected package php-common.

That's dpkg ouput, isn't it? So maybe add '-o=Dpkg::Use-Pty=0' to your command.




--

[David Christensen]

You're right -- I made no attempt to figure out why apt(8) apparently
does not implement proper "quiet", "real-quiet", etc., options.


Call it pragmatism. I've been down plenty of rabbit holes trying to
trouble-shoot software; and I expect apt(8) is non-trivial. How many
hours have you and others spent on this issue? Have you found the
"correct" solution? Using grep(1) in a command pipeline to filter out
unwanted lines is a known technique. It took me a few minutes to write
the example, and it works.


David

[Victor Sudakov]

davidson wrote:
> > Thank you, how do you activate this option? I've just tried
> > `apt-get -o quiet::NoProgress=true -qqy ...`
>
> I would do it that way too.
>
> > but the "Reading database ... 5%" stuff is still there.
>
> Yeah, after some unsatisfying experimentation I've had no luck either.
>
> Maybe this bug report will interest you:
>
> #539617 - dpkg: Add option to suppress progress display while reading database
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=539617

So, it's an old problem since 2009, and still unresolved? Thank you for the link.

On the one hand, it's sad. On the other hand, I'm glad I'm not the only one who is annoyed.

[Victor Sudakov]

David Christensen wrote:
> >>>
> >>> (Un)fortunately this is a CI/CD pipeline, the VM and its data will be
> >>> gone forever after the build. Unless I care to keep apt output as an
> >>> artifact somewhere which is IMHO an overkill. I just want an concise
> >>> CI/CD log without interactive bells and whistles like progress
> >>> indicators.
> >>
> >>
> >> Without seeing your CI/CD pipeine -- create a patterns file containing
> >> regular expressions that match the lines you want removed, then add a
> >> grep(1) filter into the pipeline:
> >
> > Of course I can think of many workarounds. The original question was
> > however why "-qq" was not working and what I was missing in "apt"
> > usage.
>
>
> You're right -- I made no attempt to figure out why apt(8) apparently
> does not implement proper "quiet", "real-quiet", etc., options.
>
>
> Call it pragmatism. I've been down plenty of rabbit holes trying to
> trouble-shoot software; and I expect apt(8) is non-trivial. How many
> hours have you and others spent on this issue?

The link Davidson has provided shows that many people have spent some
time on it since 2009.

> Have you found the
> "correct" solution? Using grep(1) in a command pipeline to filter out
> unwanted lines is a known technique. It took me a few minutes to write
> the example, and it works.

Yes, I think I'll have to resort to a workaround once it's now clear
that this is a bug. I just don't like the idea of creating a
workaround when there is an official way to do something, but this is
clearly not the case.

[Victor Sudakov]

Curt wrote:
> >
> >
> > I'm trying to quiet apt's output by using `apt-get -qqy` in a CI/CD
> > pipeline, however I still see ugly stuff like this in my CI/CD log:
>
> Quiet level 2 implies -y.

An extra -y won't do any harm, especially if some day someone decides
to remove one -q to debug the pipeline for example.

>
> > Selecting previously unselected package php-common.
>
> That's dpkg ouput, isn't it? So maybe add '-o=Dpkg::Use-Pty=0' to your command.

I have already written in <YumiAG+O...@admin.sibptus.ru> that I've tried

setting `sudo DEBIAN_FRONTEND=noninteractive apt-get -qqy --no-install-recommends install ...`

and `sudo apt-get -qqy -o Dpkg::Use-Pty=0 ...` but the rubbish is still there.

[Curt]

On 2022-08-03, Victor Sudakov <v...@sibptus.ru> wrote:
>
>
> Curt wrote:
>> >
>> >
>> > I'm trying to quiet apt's output by using `apt-get -qqy` in a CI/CD
>> > pipeline, however I still see ugly stuff like this in my CI/CD log:

>> Quiet level 2 implies -y.
>
> An extra -y won't do any harm, especially if some day someone decides
> to remove one -q to debug the pipeline for example.

So what's acceptable in the input is intolerable in the output.

[Victor Sudakov]

That's correct. In fact, the apt command with its arguments is not
even visible in the CI/CD log, only its output is.