Encoding to utf-8 changes umlaut in status line

[J S]

Hi vim users,

I can't figure out a problem with utf-8 in (g)vim. I would like to use utf-8 as encoding but the status line changes when I do that (set encoding=utf-8) and german Umlaute are not displayed correctly any more. There is some odd character representation shown, e.g. <e4> instead of ä. Strangely though this seems to be a problem only for the lower case letters. If I hit i for insert mode the status line changes to EINFÜGEN (german for INSERT), which is correct.
When I reencode to latin1 everything is back with Umlaut.

Is there a way to get correctly displayed Umlaute with utf-8 in the status line?
Thanks,
John
--
http://dynamic.xkcd.com/random/comic/

[Ben Fritz]

On Monday, November 12, 2012 11:00:46 AM UTC-6, J S wrote:
> Hi vim users,
>
> I can't figure out a problem with utf-8 in (g)vim. I would like to use utf-8
> as encoding but the status line changes when I do that (set encoding=utf-8)
> and german Umlaute are not displayed correctly any more. There is some odd
> character representation shown, e.g. <e4> instead of ä. Strangely though this
> seems to be a problem only for the lower case letters. If I hit i for insert
> mode the status line changes to EINFÜGEN (german for INSERT), which is
> correct.
>
>
> When I reencode to latin1 everything is back with Umlaut.
>

Try setting 'encoding' in your .vimrc, at the very beginning, instead of after
Vim starts up. :help 'encoding' says:

NOTE: Changing this option will not change the encoding of the
existing text in Vim. It may cause non-ASCII text to become invalid.
It should normally be kept at its default value, or set when Vim
starts up.

In other words, DON'T change 'encoding' after Vim starts up. You should only
ever change it once at the beginning of your .vimrc.

My guess is that you are setting your statusline to text containing non-ASCII
characters, then setting 'encoding', which causes Vim to reinterpret the bytes
which are valid for Latin1 as UTF-8 without conversion.

See http://vim.wikia.com/wiki/Working_with_Unicode

[Benjamin Fritz]

CCed back to vim_use mailing list, and rearranged for proper bottom-posting
format. Please include the list on your response and bottom-post as I do below.

On Mon, Nov 12, 2012 at 11:13 AM, J S <drst...@googlemail.com> wrote:
> 2012/11/12 Ben Fritz <fritzo...@gmail.com>

>>
>> On Monday, November 12, 2012 11:00:46 AM UTC-6, J S wrote:
>> > Hi vim users,
>> >
>> > I can't figure out a problem with utf-8 in (g)vim. I would like to use
>> > utf-8
>> > as encoding but the status line changes when I do that (set
>> > encoding=utf-8)
>> > and german Umlaute are not displayed correctly any more. There is some
>> > odd
>> > character representation shown, e.g. <e4> instead of ä.
>> >

>> > When I reencode to latin1 everything is back with Umlaut.
>> >
>>
>> Try setting 'encoding' in your .vimrc, at the very beginning, instead of
>> after
>> Vim starts up.
>>

>> [SNIP]

>>
>> My guess is that you are setting your statusline to text containing
>> non-ASCII
>> characters, then setting 'encoding', which causes Vim to reinterpret the
>> bytes
>> which are valid for Latin1 as UTF-8 without conversion.
>>
>

> I actually have that setting in my vimrc and only figured out that the
> encoding is the source of the problem after I commented out that statement.
> But I think you are right concerning the wrong reinterpretation of latin1
> bytes when converting to utf-8. I don't know where the status line text
> comes from otherwise I could change it to something not so sensitive to
> encoding.
>

WHERE in your .vimrc is it? It should be pretty much the very first line in the
file. You can put "set nocompatible" above it, but not much else. I suspect
wherever your statusline gets set happens before your set encoding=utf-8 line.

To find where your statusline is being set, try the following command:

:verbose set statusline?

Also see the output of the :scriptnames command to see what might be getting
sourced beforehand.

What part of your statusline contains the ä character? Is it a filename or
something else? Another possible source of problems is if you specify the ä
character literally, like "set statusline=än\ example\ statusline". If you do
this, you'll need to tell Vim what the encoding of the file setting the
statusline is in, via the :scriptencoding command, prior to setting the
statusline.

[J S]

Thanks Ben,
the encoding setting is the first thing in my vimrc. For testing purposes I only had that line in a testrc file. The character representation is still wrong.

Besides, I think I am not talking about the 'real' statusline since playing with the "set statusline" command changes the line above the one I was talking about. I guess the line I am talking about isn't even called statusline. What I mean is the line giving the status on commands (e.g. "wote xyz lines to file" on a :w) so I thought it is eponymous. Maybe it's called command line instead?
J.

2012/11/12 Benjamin Fritz <fritzo...@gmail.com>

--
http://dynamic.xkcd.com/random/comic/

[Tony Mechelynck]

On 12/11/12 22:47, J S wrote:
> Thanks Ben,
> the encoding setting is the first thing in my vimrc. For testing
> purposes I only had that line in a testrc file. The character
> representation is still wrong.
>
> Besides, I think I am not talking about the 'real' statusline since
> playing with the "set statusline" command changes the line above the one
> I was talking about. I guess the line I am talking about isn't even
> called statusline. What I mean is the line giving the status on commands
> (e.g. "wote xyz lines to file" on a :w) so I thought it is eponymous.
> Maybe it's called command line instead?
> J.

Maybe Vim doesn't know in which language you want your messages
displayed. In Console mode, there is a 'termencoding' option which tells
Vim in which encoding it should communicate with the terminal. The
default for that option is empty, which means "use 'encoding'", but if
you change 'encoding' the OS won't know it and from then on Vim and the
display (and keyboard) will be talking at cross-purposes. Another
possibility is the locale (see :help :lang).

So:

if has('multi_lang') " if it doesn't, there is no :lang command
if has('unix') || has('mac')
" Vim musst wei�en, dass wir Berichten
" auf Deutsch, und in UTF-8, w�nschen zu sehen.
lang de_DE.UTF-8
else
" I hope the following is correct for Windows
" please check and, if necessary, use trial-and-error
lang German_Germany.10646
endif
endif
if has('multi_byte') " if it doesn't, no 'encoding' etc.
if &enc !~? '^u' " if it is not already Unicode
if &tenc == ""
" don't clobber keyboard/display settings
" keep the old 'encoding' value
" for console I/O
let &tenc = &enc
endif
set enc=utf-8 " now we may change it
endif
set fencs=ucs-bom,utf-8,latin1
endif



Best regards,
Tony.
--
Misery loves company, but company does not reciprocate.

[J S]

Tony,
thanks for your tips. Your suggestion led me to this answer: http://superuser.com/questions/207264/gvim-utf-8-in-windows which resolved my issues.
J.

2012/11/13 Tony Mechelynck <antoine.m...@gmail.com>

On 12/11/12 22:47, J S wrote:

Thanks Ben,
the encoding setting is the first thing in my vimrc. For testing
purposes I only had that line in a testrc file. The character
representation is still wrong.

Besides, I think I am not talking about the 'real' statusline since
playing with the "set statusline" command changes the line above the one
I was talking about. I guess the line I am talking about isn't even
called statusline. What I mean is the line giving the status on commands
(e.g. "wote xyz lines to file" on a :w) so I thought it is eponymous.
Maybe it's called command line instead?
J.

Maybe Vim doesn't know in which language you want your messages displayed. In Console mode, there is a 'termencoding' option which tells Vim in which encoding it should communicate with the terminal. The default for that option is empty, which means "use 'encoding'", but if you change 'encoding' the OS won't know it and from then on Vim and the display (and keyboard) will be talking at cross-purposes. Another possibility is the locale (see :help :lang).

So:

if has('multi_lang')    " if it doesn't, there is no :lang command
        if has('unix') || has('mac')

                " Vim musst weißen, dass wir Berichten
                " auf Deutsch, und in UTF-8, wünschen zu sehen.

                lang de_DE.UTF-8
        else
                " I hope the following is correct for Windows
                " please check and, if necessary, use trial-and-error
                lang German_Germany.10646
        endif
endif
if has('multi_byte')    " if it doesn't, no 'encoding' etc.
        if &enc !~? '^u' " if it is not already Unicode
                if &tenc == ""
                        " don't clobber keyboard/display settings
                        " keep the old 'encoding' value
                        " for console I/O
                        let &tenc = &enc
                endif
                set enc=utf-8   " now we may change it
        endif
        set fencs=ucs-bom,utf-8,latin1
endif



Best regards,
Tony.
--
Misery loves company, but company does not reciprocate.

--
http://dynamic.xkcd.com/random/comic/