alpine 2.0 not displaying well special characters in external tools

[Sabih Gerez]

I am using PC-Alpine 2.0. In two situations, I see that Alpine wrongly
converts special characters:

1. when displaying HTML in an external browser (Firefox).

2. when including received text in a reply as submitted to an external
editor (gvim).

Here are two examples.

The received message is in character set iso-8859-1 and contains the
(Dutch) word: "financiële". When reading the message in Alpine, the
word displays well (an "e" with diaresis). If asked to view the HTML
attachment in Firefox the word displays as "financiële". When asked
to reply to the message, the original message as quoted in the
external editor again displays as: "financiële".

Another message is in character set Windows-1252. It contains the
Dutch word "foto s" which is displayed as "fotoâ " both in Firefox as
in gvim.

In both cases, Thunderbird has no problem dealing with the special
characters. The character also displays well in Alpine's internal
editor when the external-editor feature is disabled for composing
replies.

It seems to me that Alpine applies some kind of conversion when
forwarding an HTML attachment to Firefox as well as when quoting an
original message in a reply for an external editor. I have, however,
not been able to find anything on this in the documentation.

I have also tried Alpine in Linux. There the same problem occurs when
viewing HTML in external tool (Konqueror), but the characters are
transferred well to an external editor in a reply (vi).

I appreciate your help. Thanks in advance.

--
Sabih Gerez
WWW: http://www.ewi.utwente.nl/~gerezsh/

[Andreas Prilop]

On Fri, 14 Dec 2012, Sabih Gerez wrote:

> Content-Type: MULTIPART/MIXED;

[X] downgrade-multipart-to-text

> Content-Type: TEXT/PLAIN; charset=ISO-8859-15
>
> foto=92s

Byte x92 is not an apostrophe (’) in ISO-8859-15.
Get rid of ISO-8859-15 and set your charset to Windows-1252
or to UTF-8.

After you have made these corrections, post again.

[Sabih Gerez]

On Sat, 15 Dec 2012, Andreas Prilop wrote:

> On Fri, 14 Dec 2012, Sabih Gerez wrote:
>
> > Content-Type: MULTIPART/MIXED;
>
> [X] downgrade-multipart-to-text
>
> > Content-Type: TEXT/PLAIN; charset=ISO-8859-15
> >
> > foto=92s
>

> Byte x92 is not an apostrophe (’) in ISO-8859-15.

> Get rid of ISO-8859-15 and set your charset to Windows-1252
> or to UTF-8.
>
> After you have made these corrections, post again.

Hello Andreas,

Thanks for your reply. Setting "downgrade-multipart-to-text"
unfortunately does not help for the problems that I mentioned in my
previous post (viz. displaying HTML in an external browser and getting
the correct rendering of quoted text in gvim).

My character set was already set to UTF-8.

I do not have any clue where the following content type comes from:

> Content-Type: TEXT/PLAIN; charset=ISO-8859-15

ISO-8859-15 is nowhere mentioned in my pinerc. Is it possible that the
newsgroup software of my provider (xs4all.nl) adds this?

I would appreciate any help to solve my original problem.

Thanks, Sabih

[Andreas Prilop]

On Sat, 15 Dec 2012, Sabih Gerez wrote:

> My character set was already set to UTF-8.

What are your settings for
display-character-set
keyboard-character-set
posting-character-set
?
See
http://www.washington.edu/alpine/tech-notes/low-level.html#char-set

> ISO-8859-15 is nowhere mentioned in my pinerc. Is it possible
> that the newsgroup software of my provider (xs4all.nl) adds this?

I don't know.

[Sabih Gerez]

On Mon, 17 Dec 2012, Andreas Prilop wrote:

> On Sat, 15 Dec 2012, Sabih Gerez wrote:
>
> > My character set was already set to UTF-8.
>
> What are your settings for
> display-character-set
> keyboard-character-set
> posting-character-set
> ?
> See
> http://www.washington.edu/alpine/tech-notes/low-level.html#char-set

Hello Andreas,

Thanks for your reaction. I had already read the techical note that
you mention, but it does not help me to get further.

So, I have a message consisting of:

1.1 ~233 lines Text/PLAIN (charset: ISO-8859-1 "Latin 1 (Western Europe)")
1.2 ~855 lines Text/HTML (charset: ISO-8859-1 "Latin 1 (Western Europe)")

display-character-set refers to the terminal in which Alpine is
running according to the technical note. Whether it is unset, set to
UTF-8 or ISO-8859-1, inside Alpine I have the correct rendering:
financiële (so the first e displays with diaeresis/umlaut).

Irrespective of the setting for display-charater-set, asking to
display Part 1.2 of the message in an external tool (Firefox in PC
Alpine, Konqueror in Linux) displays: financiële.

posting-character-set is used for messages composed by Alpine. I have
set it to UTF-8.

keyboard-character-set refers to the keyboard, so to input in
composing a message, not to displaying received messages.

The technical note does not mention character-set translations for
external tools such as a web browser.

I am afraid that I have just to accept the situation. Many mailings in
HTML start with a link to "view this message in your web browser".
Opening that link in the web browser solves my first problem. And if I
really want the correct rendering when no such link is present, the
workaround is to launch Thunderbird as my second-choice mail client
after Alpine.

My second problem, viz. the garbled quoted text when replying, is
solved by using Alpine in Linux or by copy-pasting the quoted text in
Windows from Alpine into gvim (quite an ugly workaround).

[Andreas Prilop]

On Tue, 18 Dec 2012, Sabih Gerez wrote:

> The technical note does not mention character-set translations for
> external tools such as a web browser.

Alpine uses UTF-8 internally. It seems that Alpine exports
the document always in this encoding UTF-8.
But an HTML document may still contain the encoding information
"charset=ISO-8859-1"; the web browser will then apply ISO-8859-1.

The only solution seems to select manually the correct encoding
in your web browser, for example
View > Character Encoding > More Encodings > Unicode > UTF-8
in Firefox.

[Sabih Gerez]

I already had this setting in Firefox; it does not help. I am afraid
that it does not override the settings given in HTML. If I ask Firefox
to display the HTML source, I see:

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">

So, it seems that alpine modifies the content of the HTML attachment
and translates the content to UTF-8, but "forgets" to modify the
"meta" header.

If I save the HTML source from Firefox, modify the mentioned line to:

<meta http-equiv="Content-Type" content="text/html; charset=utf-8">

and then load this file in Firefox, I get the correct rendering.

[Sabih Gerez]

Now that I understand that alpine always uses UTF-8 internally, I
fixed the redering problem in gvim by instructing gvim to use UTF-8 as
well:

Editor = C:\Program Files (x86)\Vim\vim73\gvim.exe -c "set encoding=utf-8"

This helps me, Andreas thanks!