Bug#1010806: apt: Avoid color output on monochrome terminals

[Axel]

Package: apt
Version: 2.2.4
Severity: normal

Dear Maintainer,

The apt command shows colors on monochrome terminals (even on a dumb terminal),
please avoid this. Included is a patch which i believe will solve the problem
and only allows output on terminals which actually have 'color' in them; the
same logic as to when to color the default bash prompt.
I have not tested this patch.

Kind regards,
Axel


--- private-output.cc.orig 2022-05-10 14:18:27.611767218 +0200
+++ private-output.cc 2022-05-10 14:20:17.511703336 +0200
@@ -86,7 +86,8 @@
SigWinch(0);
}

- if(!isatty(1))
+ char *term = getenv("TERM");
+ if(!isatty(1) || strstr(term, "color") == NULL)
{
_config->Set("APT::Color", "false");
_config->Set("APT::Color::Highlight", "");

[Daniel Abrecht]

Hi

I'm not really involved with apt development, but I've strong opinions
when it comes to terminal applications.

The value of $TERM refers to a terminfo / termcap definition / file. If
a terminal supports color, and which escape sequences they need for
them, is specified inside those files.
The name of the files (and thus the value of the $TERM variable), does
not necessarily have to contain "color" if the terminfo / termcap
definition specify the capabilities for color, and vice versa.
Additionally, not all terminals may use the same escape sequence for
colors either.

I think the root cause of the problem is escape sequences being
hardcoded in apt, that simply should never be done. An application
should always make use of the terminfo database when outputting any
escape sequence.
This will also make it trivial not to output escape sequences on dumb
terminals, since there'll simply be no cap / escape sequence in the
terminfo definition to output in that case.
For colors, this may be helpful:
https://stackoverflow.com/questions/42734416/how-to-set-colors-in-terminal-using-terminfo

It seams there are a few places with hard coded escape sequences, but
it's not too bad yet:
https://salsa.debian.org/search?search=033&group_id=2051&project_id=228&scope=&search_code=true&snippets=false&repository_ref=main&nav_source=navbar

Regards,
Daniel Abrecht

[Adam Borowski]

On Tue, May 10, 2022 at 02:38:40PM +0200, Axel wrote:
> The apt command shows colors on monochrome terminals (even on a dumb terminal),
> please avoid this. Included is a patch which i believe will solve the problem
> and only allows output on terminals which actually have 'color' in them; the
> same logic as to when to color the default bash prompt.
> I have not tested this patch.

> --- private-output.cc.orig 2022-05-10 14:18:27.611767218 +0200
> +++ private-output.cc 2022-05-10 14:20:17.511703336 +0200
> @@ -86,7 +86,8 @@
> SigWinch(0);
> }
>
> - if(!isatty(1))
> + char *term = getenv("TERM");
> + if(!isatty(1) || strstr(term, "color") == NULL)
> {
> _config->Set("APT::Color", "false");
> _config->Set("APT::Color::Highlight", "");

You mean monochrome terminals such as "xterm", "rxvt", "linux", ...?

I haven't seen an actual monochrome terminal since the first half of the
90's. But even those accepted color codes and gracefully ignored them.
To find a terminal that speaks a different language you'd need to go
another decade into the past.

Thus, I guess that what you want is some global switch to disable color
in programs. But there's no common characteristic of a terminal that'd
allow autodetecting your wishes.


Meow!
--
⢀⣴⠾⠻⢶⣦⠀ Eight legs good, four legs bad! -- when your drider pwns a
⣾⠁⢠⠒⠀⣿⡁ smelly goodie centaur.
⢿⡄⠘⠷⠚⠋⠀ Rearkick OP -- my grandpa's brother-in-law got one-shotted
⠈⠳⣄⠀⠀⠀⠀ from full hp in RL, please nerf!

[Julian Andres Klode]

On Tue, May 10, 2022 at 04:54:21PM +0200, Adam Borowski wrote:
> On Tue, May 10, 2022 at 02:38:40PM +0200, Axel wrote:
> > The apt command shows colors on monochrome terminals (even on a dumb terminal),
> > please avoid this. Included is a patch which i believe will solve the problem
> > and only allows output on terminals which actually have 'color' in them; the
> > same logic as to when to color the default bash prompt.
> > I have not tested this patch.
>
> > --- private-output.cc.orig 2022-05-10 14:18:27.611767218 +0200
> > +++ private-output.cc 2022-05-10 14:20:17.511703336 +0200
> > @@ -86,7 +86,8 @@
> > SigWinch(0);
> > }
> >
> > - if(!isatty(1))
> > + char *term = getenv("TERM");
> > + if(!isatty(1) || strstr(term, "color") == NULL)
> > {
> > _config->Set("APT::Color", "false");
> > _config->Set("APT::Color::Highlight", "");
>
> You mean monochrome terminals such as "xterm", "rxvt", "linux", ...?
>
> I haven't seen an actual monochrome terminal since the first half of the
> 90's. But even those accepted color codes and gracefully ignored them.
> To find a terminal that speaks a different language you'd need to go
> another decade into the past.
>
> Thus, I guess that what you want is some global switch to disable color
> in programs. But there's no common characteristic of a terminal that'd
> allow autodetecting your wishes.

apt respects https://no-color.org/ NO_COLOR environment variable,
and I believe this is the way to go for this.

For completeness, my bashrc does the following, so it looks at
tput setaf 1 (change color to red to see if there's color available).

case "$TERM" in
xterm-color|*-256color) color_prompt=yes;;
esac

# uncomment for a colored prompt, if the terminal has the capability; turned
# off by default to not distract the user: the focus in a terminal window
# should be on the output of commands, not on the prompt
force_color_prompt=yes

if [ -n "$force_color_prompt" ]; then
if [ -x /usr/bin/tput ] && tput setaf 1 >&/dev/null; then
# We have color support; assume it's compliant with Ecma-48
# (ISO/IEC-6429). (Lack of such support is extremely rare, and such
# a case would tend to support setf rather than setaf.)
color_prompt=yes
else
color_prompt=
fi
fi

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en

[Axel Scheepers]

Hi,

Regarding terminal capabilities,

On Tue, May 10, 2022 at 6:04 PM Julian Andres Klode <j...@debian.org> wrote:
> > in programs. But there's no common characteristic of a terminal that'd
> > allow autodetecting your wishes.

The 'proper' way would be to query the terminfo entry Co I think. The best way
to make a terminal program color capable would be to use a terminal library
like ncurses which handles this for you and has a function `has_colors(void);`
which does the right thing(tm) :)

Kind regards,
Axel

[Axel Scheepers]

Hi,

Thanks for your quick response and thoughtful comments,

On Tue, May 10, 2022 at 6:04 PM Julian Andres Klode <j...@debian.org> wrote:
> On Tue, May 10, 2022 at 04:54:21PM +0200, Adam Borowski wrote:
> > You mean monochrome terminals such as "xterm", "rxvt", "linux", ...?

Yes, and screenreaders or other accessibility tools. A linux/unix terminal can
be a great interface for people with visibility problems.

> > I haven't seen an actual monochrome terminal since the first half of the
> > 90's. But even those accepted color codes and gracefully ignored them.
> > To find a terminal that speaks a different language you'd need to go
> > another decade into the past.

My first crt one was a vt-52 compatible :)
There are other uses however like the mentioned accessibility tools. It's the
same reason some people like/have to use the ed editor or the edbrowse(r).

> > Thus, I guess that what you want is some global switch to disable color
> > in programs. But there's no common characteristic of a terminal that'd
> > allow autodetecting your wishes.
>
> apt respects https://no-color.org/ NO_COLOR environment variable,
> and I believe this is the way to go for this.

Oh that's really nice. I've requested this via other channels so I'm glad
it is implemented now. Thanks!

> For completeness, my bashrc does the following, so it looks at
> tput setaf 1 (change color to red to see if there's color available).
>
> case "$TERM" in
> xterm-color|*-256color) color_prompt=yes;;
> esac

I'm referring to:

# set a fancy prompt (non-color, unless we know we "want" color)

case "$TERM" in
xterm-color|*-256color) color_prompt=yes;;
esac

from /etc/skel/.bashrc (which imho also should avoid using color at all :).
The part you refer to is for forcing color output.

Thanks again for checking for NO_COLOR, I hope this can be used for
more things in the future.

Kind regards,
Axel

[Adam Borowski]

The terminfo entries stopped being maintained by late 80's. And even if
they were, every new terminal would need to wait several years before it can
have its definition known by operating systems (today, distributions).
The effect? Most terminals identify as "xterm", "xterm-256color", or
"rxvt". For example both libvt (Gnome-Terminal, etc) and Konsole claim to
be an xterm...

And even if $TERM->terminfo were usable, a serial console has no way to pass
env vars. As an install/rescue tool, apt gets run over a serial console
pretty often.

Thus, using terminfo is definitely not a "Right Thing" this millenium.
Most new programs just hardcode the codes, assuming a vt100-like terminal
with a common set of capabilities. This includes color, as the last
terminal without color that I remember was Windows 3.X/95's telnet.exe
(which, per the vt100 language, ignored unknown SGR codes gracefully).

Ie, this patch doesn't work, and I see no way to make it work.


Meow!
--
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Imagine there are bandits in your house, your kid is bleeding out,
⢿⡄⠘⠷⠚⠋⠀ the house is on fire, and seven giant trumpets are playing in the
⠈⠳⣄⠀⠀⠀⠀ sky. Your cat demands food. The priority should be obvious...

[Axel Scheepers]

Hi,

On Mon, May 16, 2022 at 6:30 PM Adam Borowski <kilo...@angband.pl> wrote:
> The terminfo entries stopped being maintained by late 80's. And even if
> they were, every new terminal would need to wait several years before it can
> have its definition known by operating systems (today, distributions).
> The effect? Most terminals identify as "xterm", "xterm-256color", or
> "rxvt". For example both libvt (Gnome-Terminal, etc) and Konsole claim to
> be an xterm...
>
> And even if $TERM->terminfo were usable, a serial console has no way to pass
> env vars. As an install/rescue tool, apt gets run over a serial console
> pretty often.

I think you are mistaken here. There's no need to pass environment variables,
if you run a serial terminal you are supposed to set the proper type, by using
systemd's Environment setting in serial-getty@service or otherwise
(i'm more used to gettytab). Not the other way around on the client side.
A common setting, certainly for a rescue terminal, is vt100. When you look at
terminfo you'll see it defines some common things like how to handle backspace
(this should sound familiar when you've been running *nix for some
time as it was
a big problem in the dialup era). When your function or arrow keys don't work
properly for instance this is the way to fix that.

> Thus, using terminfo is definitely not a "Right Thing" this millenium.
> Most new programs just hardcode the codes, assuming a vt100-like terminal
> with a common set of capabilities. This includes color, as the last
> terminal without color that I remember was Windows 3.X/95's telnet.exe
> (which, per the vt100 language, ignored unknown SGR codes gracefully).

Using curses (and therefor terminfo/termcap) has been the proper way to
handle different terminals for years. Using hardcoded ansi only has become
popular the last decade or so. A vt100 does *not* support color, a thing which
terminfo tells you when you run infocmp.

$ infocmp | grep color
# Reconstructed via infocmp from file: /lib/terminfo/s/screen-256color
screen-256color|GNU Screen with 256 colors,
colors#0x100, cols#80, it#8, lines#24, pairs#0x10000,

$ export TERM=vt100
$ infocmp | grep color
$

Even if there are terminals which are basically 'dumb' (a serial
braille terminal
comes to mind) it should be possible to disable all color output by setting
TERM=xterm-mono when you run an xterm. It's quite rude to use color on a
monochrome terminal like this.

> Ie, this patch doesn't work, and I see no way to make it work.

The patch was the smallest addition I could think about without including
a dependency on curses. Please reconsider using color only on terminals
which 'want' to use them.

Kind regards,

Axel

[Daniel Abrecht]

Am 2022-05-16 18:29, schrieb Adam Borowski:
> On Tue, May 10, 2022 at 07:34:25PM +0200, Axel Scheepers wrote:
> The terminfo entries stopped being maintained by late 80's.

This doesn't seam to be true. The terminfo files seam to mostly come
from ncurses-term,
the ncurses package seams still to be maintained and getting updates
(https://tracker.debian.org/pkg/ncurses).
Looking at the upstream release notes, which state that they are form
October 21, 2021, quiet a few new ones have been added:
https://invisible-island.net/ncurses/announce.html#h3-database

Therefore, it seams terminfo entries are still being maintained to this
day.

> And even if
> they were, every new terminal would need to wait several years before
> it can
> have its definition known by operating systems (today, distributions).

It's not like new kinds of real terminals are produced anymore, at least
I don't know of any.
In case of terminal emulators, they have to be packaged anyway, so a the
terminfo file
can be added to the appropriate package at the same time.

> The effect? Most terminals identify as "xterm", "xterm-256color", or
> "rxvt". For example both libvt (Gnome-Terminal, etc) and Konsole claim
> to
> be an xterm...

There are terminals who choose to be compatible to xterm. It probably
has more to do with
programs wrongly hardcoding escape sequences than anything else, and
there is nothing wrong
with making an xterm-compatible terminal. However, that doesn't change
the fact that other
terminals exist as well. In addition to this, if terminals implement
special new escape sequences
(think about recent things like sixel for example), there is no way
around an appropriate
terminfo to make known that it's supported.

> And even if $TERM->terminfo were usable, a serial console has no way to
> pass
> env vars. As an install/rescue tool, apt gets run over a serial
> console
> pretty often.

It works automatically for terminal emulators. Tools like getty can set
the TERM variable.
Yes, only the operator knows what is connected to a serial console. In
cases where it is not
known, personally, I think a dumb terminal should be assumed to maximize
compatibility.
But I guess in most cases, something like vt100 or xterm will often
work, so I don't mind it
too much. People who really need to can still override it. It's not
automagic, but it works.

> Thus, using terminfo is definitely not a "Right Thing" this millenium.
> Most new programs just hardcode the codes, assuming a vt100-like
> terminal
> with a common set of capabilities.

I'm happy to report, that aside from apt, I'm not using any of those
programs.
Yes, there are lazy modern programmers insisting on doing it wrong.
Those programs
will simply fail on various terminals. Let's not promote that, it won't
right it.

Setting TERM works, it works well, and it solves the problem in the best
way possible.
Just because it can't always do it automatically, doesn't mean we should
give up and leave things unfixably broken.
terminfo is the right thing. It is the only way to deal with these
terminals. This will never change.

Regards,
Daniel Abrecht

[Adam Borowski]

On Mon, May 16, 2022 at 08:15:16PM +0200, Axel Scheepers wrote:
> On Mon, May 16, 2022 at 6:30 PM Adam Borowski <kilo...@angband.pl> wrote:
> > The terminfo entries stopped being maintained by late 80's. And even if
> > they were, every new terminal would need to wait several years before it can
> > have its definition known by operating systems (today, distributions).
> > The effect? Most terminals identify as "xterm", "xterm-256color", or
> > "rxvt". For example both libvt (Gnome-Terminal, etc) and Konsole claim to
> > be an xterm...
> >
> > And even if $TERM->terminfo were usable, a serial console has no way to pass
> > env vars. As an install/rescue tool, apt gets run over a serial console
> > pretty often.
>
> I think you are mistaken here. There's no need to pass environment variables,
> if you run a serial terminal you are supposed to set the proper type, by using
> systemd's Environment setting in serial-getty@service or otherwise
> (i'm more used to gettytab). Not the other way around on the client side.

And how, pray tell, may I know in advance what terminal will be used on the
other end of the serial link? Especially if it's a box people ssh into from
random places, using a plethora of various terminals on their side?
That's the most typical setup these days.

> A common setting, certainly for a rescue terminal, is vt100. When you look at
> terminfo you'll see it defines some common things like how to handle backspace
> (this should sound familiar when you've been running *nix for some
> time as it was
> a big problem in the dialup era).

"vt100" is not a common setting for some decades now, and it differs quite a
bit from what modern terminals offer. Both by lacking capabilities people
take for granted, and by having some no sane terminal bothers to implement
today. Thus it's a pretty bad choice.

> When your function or arrow keys don't work
> properly for instance this is the way to fix that.

Most function keys are not representable via termcap nor terminfo, and TERM
strings are overloaded anyway between diverse terminals (like xterm vs libvt
vs konsole vs ...).

> > Thus, using terminfo is definitely not a "Right Thing" this millenium.
> > Most new programs just hardcode the codes, assuming a vt100-like terminal
> > with a common set of capabilities. This includes color, as the last
> > terminal without color that I remember was Windows 3.X/95's telnet.exe
> > (which, per the vt100 language, ignored unknown SGR codes gracefully).
>
> Using curses (and therefor terminfo/termcap) has been the proper way to
> handle different terminals for years.

Until late 80s maybe. By the 90s using curses in a diverse setting meant
lackingdetailslikespaces (this example is IRIX curses vs Linux console), as
vendors stopped maintaining termcap databases.

>A vt100 does *not* support color, a thing which terminfo tells you when you
> run infocmp.

vt100 also stopped being manufactured before the average user has been even
born.

> > Ie, this patch doesn't work, and I see no way to make it work.
>
> The patch was the smallest addition I could think about without including
> a dependency on curses. Please reconsider using color only on terminals
> which 'want' to use them.

The only way I can see your idea to work, would be a list of ancient
terminals that lack color (either handcrafted or queried via termcap), and
relying on the user setting one of those. But that way would be way way too
niche to bother implementing in apt, I'd say.

Thus, as ways to disable color go, I don't see your patch as viable.


Meow!
--
⢀⣴⠾⠻⢶⣦⠀ Aryans: split from other Indo-Europeans ~2900-2000BC → Ural →
⣾⠁⢠⠒⠀⣿⡁ Bactria → settled 2000-1000BC in northwest India.
⢿⡄⠘⠷⠚⠋⠀ Gypsies: came ~1000AD from northern India; aryan.
⠈⠳⣄⠀⠀⠀⠀ Germans: IE people who came ~2800BC to Scandinavia; not aryan.

[Axel Scheepers]

Op wo 25 mei 2022 21:36 schreef Adam Borowski <kilo...@angband.pl>:

And how, pray tell, may I know in advance what terminal will be used on the
other end of the serial link?  Especially if it's a box people ssh into from
random places, using a plethora of various terminals on their side?
That's the most typical setup these days.

You can't really in that case; the terminal emulator will set the proper value. Maybe it's wise to run tset in that case so if it's not what is expected one can set it themselves. Or maybe they have set them themselves as I do;

I agree that this is the most common setup but it doesn't change a thing really. I set my x terminals to xterm-mono. When I use colors many things become unreadable. Switching between black on white and white on black is also troublesome without creating specialised palettes.

The only way I can see your idea to work, would be a list of ancient
terminals that lack color (either handcrafted or queried via termcap), and
relying on the user setting one of those.  But that way would be way way too
niche to bother implementing in apt, I'd say.

Or the other way around as I proposed. The entire problem of knowing what and how to use color goes away when curses is used. Would you accept a patch for using that instead?

Kind regards,

Axel

[Axel Scheepers]

101...@bugs.debian.org

On Thu, 26 May 2022 11:03:25 +0200 Axel Scheepers
<axel.sch...@gmail.com> wrote:
> Or the other way around as I proposed. The entire problem of knowing what
> and how to use color goes away when curses is used. Would you accept a
> patch for using that instead?

I've forgot about this but how about the following?

#include <curses.h>
#include <term.h>

int
ncolors(void)
{
int ncolors = -1;

if (setupterm(NULL, 1, NULL) == OK) {
ncolors = tigetnum("colors");
del_curterm(cur_term);
}

return ncolors;
}

This doesn't initialise the terminal so you can still do the
progressbar without clearing the screen and/or use hardcoded ansi
codes for colors if you want. It returns the number of colors for the
used terminal or -1 if color is not supported. Maybe the progressbar
should be optional in that case too. I think it would be a nice
addition and one doesn't have to use apt-get and/or apt-cache
instead.
Also, maybe using italics, bold and reverse video instead of colors
can be considered? They usually do the right thing with either
background. Anyway, I hope you can implement the above function to
make at least monochrome and/or dumb terminals behave.

Kind regards,
Axel

[Axel Scheepers]

Using 'ncolors' for both function and variable isn't really nice but
as is the function will exit when you use it with an unknown terminal
which might not be ok. Maybe using the following would be better;

#include <curses.h>
#include <term.h>

int
ncolors(void)
{

int err, n = -1;

if (setupterm(NULL, 1, &err) == OK) {
n = tigetnum("colors");
del_curterm(cur_term);
}

return n;
}

You should also link ncurses (which brings in a bit of cruft).
Kind regards,
Axel