[vim/vim] Problem with UTF-8 encoding (#8363)

[Filip Demski (Glamhoth)]

Instructions: Replace the template text and remove irrelevant text (including this line)

Describe the bug
Somewhere between versions 8.2.0814 and 8.2.1704 an encoding bug was introduced. In version 0814 every UTF-8 character was displaying properly like that:

but after upgrading to 1704 (and still present in 2891)

I also noticed that reloading file to a different encoding, for example by running
:e ++enc=utf16
and then back to UTF-8
:e ++enc=utf8
fixes that opened file.

To Reproduce

1. Start Vim in version >=8.2.1704
2. Try to type a UTF-8 character (e.g. « by pressing alt-gr and 9) or open a file that contains one

Expected behavior
Characters should be displayed properly.

Environment (please complete the following information):

• Vim version: 8.2.2891
• OS: Arch Linux, kernel 5.12.8
• Terminal: st

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

[Filip Demski (Glamhoth)]

#8075 seems to tackle the same issue

[Christian Brabandt]

could you bisect the vim patch?

[Filip Demski (Glamhoth)]

could you bisect the vim patch?

Yes, it seems that 8.2.0936 is the one
a45551a

[Bram Moolenaar]

What is your 'encoding' set to? Does it change between the Vim versions you tried?
When editing a file , what is 'fileencoding' set to? Does it change when you use different ++enc arguments?
Does typing CTRL-L help at all?
Did you start Vim with "vim --clean" ?

[Filip Demski (Glamhoth)]

What is your 'encoding' set to? Does it change between the Vim versions you tried?

It's (correctly) set to UTF-8, it doesn't change between versions.

When editing a file , what is 'fileencoding' set to?

It's also (correctly) set to UTF-8.

Does it change when you use different ++enc arguments?
Does typing CTRL-L help at all?

Actually, yes! It seems that any buffer/screen refresh helps (so there is actually no need to change encodings like i mentioned in the issue, just 'rereading' is enough).

Did you start Vim with "vim --clean" ?

Yes, I even deleted .vim directory and .vimrc altogether to be sure.

[Jürgen Krämer]

What does :verbose set fencs? say?

[Filip Demski (Glamhoth)]

Okay, I just tried a different terminal (Alacritty) and everything works as it should, so it has to be a problem with st.

What does :verbose set fencs? say?

fileencodings=ucs-bom,utf-8,default,latin1

[Christian Brabandt]

so does an :au VimEnter * refresh! help? What about :set t_u7= in your .vimrc?

[Filip Demski (Glamhoth)]

so does an :au VimEnter * refresh! help

au BufEnter * redraw! helps

What about :set t_u7= in your .vimrc?

This does work too

[Christian Brabandt]

then perhaps report to the st maintainers as well

[Bram Moolenaar]

This very much sounds like the terminal can't handle some more complicated escape codes and gets in a bad state until the screen is cleared. Don't see a reason to look into the Vim side of this.

[Bram Moolenaar]

Closed #8363.

[Shane-XB-Qian]

how about following case:
│SELECT was copied by mouse - its 'right key' (chose 'copy' of its popup menu)
if pasted into vim by "+p (or "*p), it would show as \u2502SELECT
or insert mode - pasted by 'shift+insert' then it would be │SELECT
// tried on xterm.
looks such case existed long time, any idea if it was expected or a bug?

[Shane-XB-Qian]

not sure which patch but looks version v8.2.3095 i can not reproduce above my case now... good! :-)