Patch 8.1.1146

[Bram Moolenaar]

Patch 8.1.1146
Problem: In MS-Windows console colors in a terminal window are wrong.
Solution: Use the ansi index also for 16 colors. (Ken Takata)
Files: src/terminal.c


*** ../vim-8.1.1145/src/terminal.c 2019-04-06 22:01:20.756989404 +0200
--- src/terminal.c 2019-04-11 11:23:46.712315123 +0200
***************
*** 2433,2440 ****
if (color->ansi_index != VTERM_ANSI_INDEX_NONE)
{
/* First 16 colors and default: use the ANSI index, because these
! * colors can be redefined. */
! if (t_colors >= 16)
return color->ansi_index;
switch (color->ansi_index)
{
--- 2433,2440 ----
if (color->ansi_index != VTERM_ANSI_INDEX_NONE)
{
/* First 16 colors and default: use the ANSI index, because these
! * colors can be redefined, we use the RGB values. */
! if (t_colors > 256)
return color->ansi_index;
switch (color->ansi_index)
{
***************
*** 3604,3609 ****
--- 3604,3610 ----
for (; index < 16; index++)
{
VTermColor color;
+
color.red = (unsigned)(rgb[index] >> 16);
color.green = (unsigned)(rgb[index] >> 8) & 255;
color.blue = (unsigned)rgb[index] & 255;
*** ../vim-8.1.1145/src/version.c 2019-04-11 11:19:21.553778627 +0200
--- src/version.c 2019-04-11 11:39:27.027580424 +0200
***************
*** 773,774 ****
--- 773,776 ----
{ /* Add new patch number below this line */
+ /**/
+ 1146,
/**/

--
"Women marry men hoping they will change. Men marry women hoping
they will not. So each is inevitably disappointed."
- Einstein

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Jason Franklin]

Environment: GNOME Terminal on Ubuntu 18.04

git bisect shows this commit as changing the colors of my terminal window.

See before and after images attached.

I can fix the problem with:

diff --git a/src/terminal.c b/src/terminal.c
index c7dc23f90..8bea9481a 100644
--- a/src/terminal.c
+++ b/src/terminal.c
@@ -2434,7 +2434,7 @@ color2index(VTermColor *color, int fg, int *boldp)

{
/* First 16 colors and default: use the ANSI index, because these

* colors can be redefined, we use the RGB values. */

- if (t_colors > 256)
+ if (t_colors >= 256)

return color->ansi_index;
switch (color->ansi_index)
{

As usual, feedback/explanation is appreciated. :)

[Bram Moolenaar]

That code has already changed, the condition was removed. And it works
fine for most users. Perhaps you need to adjust your setup?

--
A parent can be arrested if his child cannot hold back a burp during a church
service.
[real standing law in Nebraska, United States of America]

[Jason Franklin]

> Perhaps you need to adjust your setup?

I'm perfectly fine with that, but I'm at a loss for what should be
adjusted.

I'll poke around. If you have any suggestions, let me know!

Thanks,
Jason

[Jason Franklin]

Bram,

I did a good bit of work to script up a demonstration of this problem.

I believe something is now broken with how Vim interprets custom color
palettes in 256-color terminal windows. I can reproduce the problem without
using my own .vim directory.

I wrote a short script (attached) to serve as a sanity check for how colors
are displayed in Vim terminal windows.

Again, my environment is GNOME Terminal on Ubuntu 18.04. In GNOME Terminal,
I have selected the Solarized font color palette for my profile. To make the
text more clear, you may want to use the Solaried dark background theme.
Other color palettes may be broken as well, but some appear to work
properly.

To reproduce:

0. Use solarized font palette in all terminal windows in GNOME Terminal
1. Compile latest vim (way after this patch, code here already changed)
2. ./vim --clean
3. :terminal, then run "colors" script to see the color table
4. open a second terminal window
5. run "colors" script in the new terminal window outside of vim
6. compare the tables
7. you'll see the tables are not the same, as they should be

In case your environment does not match mine, I have included screenshots of
the color table in both a Vim (./vim --clean) terminal window and in
a regular GNOME Terminal window.

I believe this issue, though it seems small, is important because it breaks
a really popular color scheme configuration.

Hopefully, you'll find the attached script useful, and it may give me some
direction. It could be that I need to go to the GNOME project to get
someone to work on this, but I believe the problem is here.

Thanks,
Jason Franklin

[Bram Moolenaar]

Hmm, this relies on "tput" which I think is not a great tool. Not only
does it not show the escape sequences used, it also has some obvious
problems. For exampl, in my 256 color xterm it reports only 8 colors.
In your screenshots the bold colors are clearly wrong.

Please use raw escape sequences, so that we can see what actually
happens.

--
CART DRIVER: Bring out your dead!
We follow the cart through a wretched, impoverished plague-ridden village.
A few starved mongrels run about in the mud scavenging. In the open
doorway of one house perhaps we jug glimpse a pair of legs dangling from
the ceiling. In another doorway an OLD WOMAN is beating a cat against a
wall rather like one does with a mat. The cart passes round a dead donkey
or cow in the mud. And a MAN tied to a cart is being hammered to death by
four NUNS with huge mallets.
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

[Jason Franklin]

I rewrote my script to be much simpler.  It shows both the escape codes being printed and

the word "example" in each color.

> In your screenshots the bold colors are clearly wrong.

I discovered this was caused by a GNOME Terminal setting called "Show bold text in bright colors".

This was enabled, and it caused the colors to appear off.

The colors are still different in Vim terminal and a plain GNOME Terminal.

I have attached:

1. A screenshot of my GNOME Terminal color settings, so you can see exactly what I'm using.

2. The new colors script (much simpler, no tput).

3. Image of colors running in plain GNOME Terminal.

4. Image of colors running in vim terminal window (./vim --clean, latest vim).

I hope this can give some indication of what the heck is going on. :/

Best,

Jason Franklin

[Bram Moolenaar]

It does. I can now see that in a terminal the bold attribute is used
both to make the color bright (add 8 to the color index) and to make the
text bold.

Must be caused by enabling "use bright color for bold" in the vterm
settings. It was using "t_colors >= 16", but that should probably be
"t_colors < 16", since the bold-bright mechanism makes most sense for a
terminal with 8 colors. Fortunately, the change doesn't break a test,
thus there is good hope this just works. And perhaps I can add a test
for the fix... yes.

--
ARTHUR: Old woman!
DENNIS: Man!
ARTHUR: Man. I'm sorry. Old man, What knight live in that castle over there?
DENNIS: I'm thirty-seven.

[Jason Franklin]

Bram,

Thank you so much for taking the time to work with me on this.

Unfortunately, I still can't get the colors to match on the latest build (patch 8.1.1330).  I should note that the color scheme is much closer than it was before, but it looks like some of the colors are being modified when displayed in the Vim terminal window.  The latest colors script still does not match in both cases.

I have attached an updated version of the output of "colors" in both GNOME Terminal and in the Vim terminal window (vim at latest build).

The point of interest is this:  The color for yellow and for white is darker in the Vim terminal window.  You can see this when positioning the screenshots side by side.  I have noticed that other color schemes also show this same result (i.e., darker colors in Vim terminal).  I will provide screenshots upon request.  I don't know what might cause this... I did a git blame on the latest terminal.c file and I can't see anything with colors that seems odd.

It may be that all of the colors here are slightly darker in the vim terminal window.  It's so slight that it may be hard to tell.  Again, a side-by-side comparison is the only way to really see this.

I also want to note that showing bold text in bright colors has been disabled in GNOME Terminal.

Thanks,

Jason

[Ken Takata]

Hi,

2019/5/14 Tue 5:58:30 UTC+9 Jason Franklin wrote:
> Bram,
>
> Thank you so much for taking the time to work with me on this.
>
> Unfortunately, I still can't get the colors to match on the latest build (patch 8.1.1330). I should note that the color scheme is much closer than it was before, but it looks like some of the colors are being modified when displayed in the Vim terminal window. The latest colors script still does not match in both cases.
>
> I have attached an updated version of the output of "colors" in both GNOME Terminal and in the Vim terminal window (vim at latest build).
>
> The point of interest is this: The color for yellow and for white is darker in the Vim terminal window. You can see this when positioning the screenshots side by side. I have noticed that other color schemes also show this same result (i.e., darker colors in Vim terminal). I will provide screenshots upon request. I don't know what might cause this... I did a git blame on the latest terminal.c file and I can't see anything with colors that seems odd.
>
> It may be that all of the colors here are slightly darker in the vim terminal window. It's so slight that it may be hard to tell. Again, a side-by-side comparison is the only way to really see this.
>
> I also want to note that showing bold text in bright colors has been disabled in GNOME Terminal.
>
> Thanks,
> Jason

I also noticed that the colors on Cygwin + mintty with 256-color mode are
wrong since 8.1.1146. Especially, yellow becomes brown.
The following patch seems to fix the problem:

--- a/src/terminal.c
+++ b/src/terminal.c
@@ -2432,6 +2432,8 @@ color2index(VTermColor *color, int fg, i

if (color->ansi_index != VTERM_ANSI_INDEX_NONE)
{

+ if (t_colors == 256)
+ return color->ansi_index;
// The first 16 colors and default: use the ANSI index.
switch (color->ansi_index)
{

Regards,
Ken Takata

[Jason Franklin]

Thanks, Ken.

> The following patch seems to fix the problem:

It does fix the problem for me.  With the patch applied, I can't tell the difference in the output of "colors" in a Vim terminal window and in GNOME Terminal directly.

However, it seems like this is a band-aid and not a real fix.  Why exactly do we need to avoid the switch on a 256 color terminal?  I would like to understand why it works instead of just accepting that it works for me...  Can anyone offer an explanation?  It will make me feel better. :)

Thanks again to both of you for following up on this!

Best,

Jason Franklin

[Bram Moolenaar]

Jason Franklin wrote:

> Thank you so much for taking the time to work with me on this.
>
> Unfortunately, I still can't get the colors to match on the latest build
> (patch 8.1.1330). I should note that the color scheme is much closer than
> it was before, but it looks like some of the colors are being modified when
> displayed in the Vim terminal window. The latest colors script still does
> not match in both cases.
>
> I have attached an updated version of the output of "colors" in both GNOME
> Terminal and in the Vim terminal window (vim at latest build).
>
> The point of interest is this: The color for yellow and for white is
> darker in the Vim terminal window. You can see this when positioning the
> screenshots side by side. I have noticed that other color schemes also
> show this same result (i.e., darker colors in Vim terminal). I will
> provide screenshots upon request. I don't know what might cause this... I
> did a git blame on the latest terminal.c file and I can't see anything with
> colors that seems odd.
>
> It may be that all of the colors here are slightly darker in the vim
> terminal window. It's so slight that it may be hard to tell. Again, a
> side-by-side comparison is the only way to really see this.
>
> I also want to note that showing bold text in bright colors has been
> disabled in GNOME Terminal.

I cannot reproduce this in xterm. I see no reason why you see a
difference.

--
DENNIS: Oh, very nice. King, eh! I expect you've got a palace and fine
clothes and courtiers and plenty of food. And how d'you get that? By
exploiting the workers! By hanging on to outdated imperialist dogma
which perpetuates the social and economic differences in our society!

[Bram Moolenaar]

Ken Takata wrote:

> I also noticed that the colors on Cygwin + mintty with 256-color mode are
> wrong since 8.1.1146. Especially, yellow becomes brown.
> The following patch seems to fix the problem:
>
> --- a/src/terminal.c
> +++ b/src/terminal.c
> @@ -2432,6 +2432,8 @@ color2index(VTermColor *color, int fg, i
>
> if (color->ansi_index != VTERM_ANSI_INDEX_NONE)
> {
> + if (t_colors == 256)
> + return color->ansi_index;
> // The first 16 colors and default: use the ANSI index.
> switch (color->ansi_index)
> {

That is not correct, because this lookup is for the first 16 colors,
which are always translated this way, no matter if using 16, 88 or 256
colors.

There might be another condition when the lookup should not happen.
Perhaps it should only be done for xterm?

--
OLD WOMAN: King of the WHO?
ARTHUR: The Britons.
OLD WOMAN: Who are the Britons?

[Jason Franklin]

> I cannot reproduce this in xterm.  I see no reason why you see a

> difference.

When I run xterm on Ubuntu 18.04, I get an 8 color terminal.  So, I'm unable to reproduce the problem with xterm as well.

It's only in GNOME Terminal that I see the difference in color.