MIME Encoded From Field In Header
Ian Hilt
pine. My question is why doesn't it parse MIME encoded From
fields correctly. I exported the relevant messages to Kmail, and
it parses these fields with no problems. I have built the system
with UTF-8 support. In fact, alpine displays the MIME encoded
body of the email message correctly, just not MIME encoded From
fields.
Any help is appreciated.
--
Ian Hilt
ian.hilt (at) gmail.com
GnuPG key: 0x4AFC1EE3
tim.the...@gmail.com
> I'm using alpine 1.10 which I understand is based heavily on
> pine. My question is why doesn't it parse MIME encoded From
> fields correctly. I exported the relevant messages to Kmail, and
> it parses these fields with no problems. I have built the system
> with UTF-8 support. In fact, alpine displays the MIME encoded
> body of the email message correctly, just not MIME encoded From
> fields.
I was just going to ask what I think is the same question, though I
didn't
realize that the characters in the From that are causing me problems
were
MIME encoded. I assumed the problem was that they weren't encoded at
all,
and it was actually curses that was not displaying them correctly,
because they're
beyond 7-bit ASCII.
When I get a message from someone that has characters beyond 7-bit
ASCII
in the From, it completely messes up the message index screen, such
that sometimes
the subject of a message is moved up or down so it appears to be
associated with
a completely different message.
I've built alpine 1.10 on x86_64-sun-solaris2.10 using
CC=cc
CFLAGS="-Xa -xc99=all -xO2 -xstrconst -KPIC -mt -xtarget=native -m64
-xarch=native"
and executing configure with
./configure --prefix=/local --exec-prefix=/local --build "x86_64-sun-
solaris2.10" --disable-nls
with additional configure flags so it can find the locally installed
OpenLDAP, krb5, and OpenSSL.
The resulting alpine binary is linked to the following libraries:
$ldd /local/bin/alpine
libgssapi_krb5.so.2 => /local/krb5/lib/64/libgssapi_krb5.so.
2
libkrb5.so.3 => /local/krb5/lib/64/libkrb5.so.3
libcom_err.so.3 => /local/krb5/lib/64/libcom_err.so.3
librt.so.1 => /lib/64/librt.so.1
libpthread.so.1 => /lib/64/libpthread.so.1
libtcl8.4.so => /local/lib/64/libtcl8.4.so
libgss.so.1 => /usr/lib/64/libgss.so.1
libldap-2.3.so.0 => /local/lib/64/libldap-2.3.so.0
liblber-2.3.so.0 => /local/lib/64/liblber-2.3.so.0
libresolv.so.2 => /lib/64/libresolv.so.2
libgen.so.1 => /lib/64/libgen.so.1
libnsl.so.1 => /lib/64/libnsl.so.1
libsocket.so.1 => /lib/64/libsocket.so.1
libdb-4.6.so => /local/BerkeleyDB/lib/64/libdb-4.6.so
libsasl.so.1 => /usr/lib/64/libsasl.so.1
libcrypto.so.0.9.8 => /local/openssl/lib/64/libcrypto.so.
0.9.8
libcurses.so.1 => /lib/64/libcurses.so.1
libssl.so.0.9.8 => /local/openssl/lib/64/libssl.so.0.9.8
libthread.so.1 => /lib/64/libthread.so.1
libc.so.1 => /lib/64/libc.so.1
libk5crypto.so.3 => /local/krb5/lib/64/libk5crypto.so.3
libkrb5support.so.0 => /local/krb5/lib/64/libkrb5support.so.
0
libaio.so.1 => /lib/64/libaio.so.1
libmd.so.1 => /lib/64/libmd.so.1
libdl.so.1 => /lib/64/libdl.so.1
libm.so.2 => /lib/64/libm.so.2
libcmd.so.1 => /lib/64/libcmd.so.1
libdb-4.5.so => /local/BerkeleyDB/lib/64/libdb-4.5.so
libmp.so.2 => /lib/64/libmp.so.2
libscf.so.1 => /lib/64/libscf.so.1
libdoor.so.1 => /lib/64/libdoor.so.1
libuutil.so.1 => /lib/64/libuutil.so.1
I note that when configure is running, configure only checks for
"ncurses", not the system curses library:
checking for setupterm in -ltinfo... no
checking for setupterm in -lncurses... no
checking for tgetent in -ltermlib... yes
(On Solaris, libtermlib is just a link to the default system
libcurses, which is why libcurses.so.1 shows up in the ldd output even
though alpine thinks its linking with libtermlib).
Any suggestions for things to try?
Thanks
Tim
Ian Hilt
wrote:
> On May 10, 1:53 pm, Ian Hilt <ian.h...@gmail.com> wrote:
>> I'm using alpine 1.10 which I understand is based heavily on
>> pine. My question is why doesn't it parse MIME encoded From
>> fields correctly. I exported the relevant messages to Kmail, and
>> it parses these fields with no problems. I have built the system
>> with UTF-8 support. In fact, alpine displays the MIME encoded
>> body of the email message correctly, just not MIME encoded From
>> fields.
>
> I was just going to ask what I think is the same question, though I
> didn't
> realize that the characters in the From that are causing me problems
> were
> MIME encoded. I assumed the problem was that they weren't encoded at
> all,
> and it was actually curses that was not displaying them correctly,
> because they're
> beyond 7-bit ASCII.
What I'm referring to as MIME encoded starts with =?ISO-8859-1?Q? and
ends with ?=. The interesting thing is that I've looked at
emails in my gmail account by "Show Original" and it displays
the From field as =?UTF-8?Q? instead of =?ISO-8859-1?Q?.
I'm using fetchmail to download these emails from gmail. AFAICT,
the field content is not longer than 75 characters which is the
limit for MIME decoding according to RFC 1342 and 2047.
Also, I know that my system can read these fields, as I've said
that Kmail decodes the exact same email with no problems.
I was using IMAP to download these emails. I switched to POP3
and I haven't seen any of this behavior since.
I'll post updates as I find out more information.
Mark Crispin
> I'm using fetchmail to download these emails from gmail.
> I was using IMAP to download these emails. I switched to POP3 and I haven't
> seen any of this behavior since.
I'm not sure that I understand you. Did you switch your fetchmail from
gmail to use POP3 instead of IMAP?
Or did you use fetchmail from gmail to load your own IMAP/POP3 server
system, and then using IMAP/POP3 in Pine/Alpine?
If the latter, what IMAP server implementation are you running? This
sounds like an IMAP server bug.
-- Mark --
http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
Ian Hilt
> On Sun, 11 May 2008, Ian Hilt posted:
>> I'm using fetchmail to download these emails from gmail.
>> [snip]
>> I was using IMAP to download these emails. I switched to POP3 and I
>> haven't seen any of this behavior since.
>
> I'm not sure that I understand you. Did you switch your fetchmail from gmail
> to use POP3 instead of IMAP?
This is correct. Sorry for the confusion.
The imap config was:
poll imap.gmail.com protocol imap service 993 user ****
password **** ssl
pop3:
poll pop.gmail.com protocol pop3 service 995 uidl user ****
password **** ssl
Ian Hilt
If it wasn't clear, this is my fetchmail run control file with
the user name and password replaced with ****.
I just confirmed that POP3 preserves the MIME-encoding character
set. So, I'm guessing the problem is either in gmail's imap
servers or in fetchmail. However, I wonder why alpine didn't
parse the iso-8859-1 encoding correctly.
Mark Crispin
> set. So, I'm guessing the problem is either in gmail's imap
> servers or in fetchmail. However, I wonder why alpine didn't
> parse the iso-8859-1 encoding correctly.
So, am I correct that you are saying that Alpine is not doing any POP3 or
IMAP access, that the only access done by Alpine is to a local disk file?
If so, the fact that you used IMAP vs. POP3 with fetchmail is irrelevant
in discussing an Alpine issue (it may be relevant with Gmail, but you need
to discuss Gmail issues with the Gmail people). The only thing that is
relevant here is what appears in the disk file after fetchmail.
If Alpine displayed the content correctly with one disk file and not with
another disk file, there is a 95% certainty that either something is wrong
with the disk file or with your configuration.
In order to narrow the problem down, please make copies of both disk files
(the "working" one and the "not working" one) and submit both of them to
the alpine-...@u.washington.edu bug reporting address. We will look
at those files.
If the "working" one is correctly MIME formatted and the "not working" one
is incorrectly MIME formatted, then Alpine is off the hook; the problem is
in whatever procedure that you used to create the files (you reported
fetchmail) and you need to persue the problem further with the developers
of that procedure.
If the "working" one is incorrectly MIME formatted and the "not working"
one is correctly MIME formatted, then the problem is probably in your
configuration and we will work with you to get that set up correctly.
If both are correctly MIME formatted, then it's an Alpine bug and we will
look into fixing it.
If both are incorrectly MIME formatted, then the problem is once again in
whatever created the files, and it was just luck that one of them happened
to work (remember the rule of Garbage In, Garbage Out).
So, we'll wait to hear from you and proceed based upon the information you
supply us. Thanks!
Ａｎｄｒｅａｓ Ｐｒｉｌｏｐ
> My question is why doesn't it parse MIME encoded From fields correctly.
Which From? Give an example!
Ian Hilt
Have you read the thread yet? It should be obvious what I'm
talking about. I'm referring to the From field in the header of
an email message. The reason I didn't give an example is that I
didn't want to divulge the email address in question to a public
newsgroup. Anyway, I've already taken Mark Crispin's advice and
submitted the "working" and "not working" emails to
alpine-...@u.washington.edu for review.
However, if you have any suggestions, I'm all ears!
Mark Crispin
On Tue, 13 May 2008, Ian Hilt posted:
-- Mark --
http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.
Ian Hilt
> When did you send your report? I don't recall seeing it.
Yesterday. The address is alpine-...@u.washington.edu
correct? Is there any particular way I should have composed the
message?
Ａｎｄｒｅａｓ Ｐｒｉｌｏｐ
> Ａｎｄｒｅａｓ Ｐｒｉｌｏｐ wrote:
See?! So it does decode!
My name in the From address is
=?UTF-8?B?77yh772O772E772S772F772B772T44CA77yw772S772J772M772P772Q?=
That's what I call an example!
Please select
[X] downgrade-multipart-to-text
in your preferences (at least for news).
This multipart is a nuisance for news.
Ian Hilt
I fail to see how this will solve my problem.
Mark Crispin
>> Please select
>> [X] downgrade-multipart-to-text
>> in your preferences (at least for news).
>> This multipart is a nuisance for news.
> I fail to see how this will solve my problem.
It won't solve Ian's problem, and it will cause other problems.
Ａｎｄｒｅａｓ Ｐｒｉｌｏｐ
> On Wed, 14 May 2008, Ian Hilt posted:
>
> It won't solve Ian's problem, and it will cause other problems.
Do *you* have any problems replying to me directly?
--
Top-posting.
What's the most irritating thing on Usenet?
Ａｎｄｒｅａｓ Ｐｒｉｌｏｐ
> Ａｎｄｒｅａｓ Ｐｒｉｌｏｐ wrote:
>
> I fail to see how this will solve my problem.
Problem? What problem?
You have again shown that Alpine does in fact decode
MIME-encoded From addresses.
You are either a liar or clueless.
Mark Crispin
> On Wed, 14 May 2008, Mark Crispin wrote:
>> It won't solve Ian's problem, and it will cause other problems.
> Do *you* have any problems replying to me directly?
That is not relevant.
I believe that Ian has a real problem. What you told him to do does
nothing to resolve his issue, whatever the cause may be.
I do not think that Ian appreciates you telling him to do something that
does not address his problem and will cause other problems. Nor do I
think that Ian appreciates you calling him "liar or clueless."
I don't appreciate it either. I also do not appreciate the distraction
when I am trying to do my job by assisting a novice user.
If you are unable and/or to assist Ian in a constructive way, then please
shut up.
Mark Crispin
Thank you very much for supplying us with your test data. I now
understand what is going on.
Unfortunately, the problem is that the message that displays incorrectly
is non-compliant with the email standard. Whatever generated that message
is broken. Specific details follow.
In the message that displays correctly, the From: header line is:
From: flam...@gmail.com (Diego 'Flameeyes' =?utf-8?Q?Petten=C3=B2?=)
which translates to a local-part of:
flameeyes
at the domain name:
gmail.com
with a comment of:
(Diego 'Flameeyes' =?utf-8?Q?Petten=C3=B2?=)
This is an old (pre-1977) syntax. The modern syntax for incorporating
personal names is:
From: Diego 'Flameeyes' =?utf-8?Q?Petten=C3=B2?= <flam...@gmail.com>
which translates to a phrase of:
Diego 'Flameeyes' =?utf-8?Q?Petten=C3=B2?=
with a local-part of:
flameeyes
at the domain name:
gmail.com
The problem with that old syntax is that technically text within
parentheses are comments and should be ignored. The post-1977 syntax
which has a specific "phrase" that is associated as a personal name for
the email address followed by a "route address" (the thing inside the
angle brackets).
In the comment (old syntax) or phrase (modern syntax), there are three
words:
Diego
'Flameeyes'
=?utf-8?Q?Petten=C3=B2?=
The third word is a what is called an "encoded word" in MIME, and
translates to
Pettenò
the last character being Unicode U+00F2 LATIN SMALL LETTER O WITH GRAVE.
This all assembles to become:
Diego 'Flameeyes' Pettenò
In the message that displays incorrectly, the From: header line is:
From: =?ISO-8859-1?Q?flam...@gmail.com_(Diego_'Flameeyes'_Petten=F2)?=
which translates to a local-part of:
=?ISO-8859-1?Q?flameeyes
at the domain name:
gmail.com_
with a comment of
(Diego_'Flameeyes'_Petten=F2)
and unknown extra crud at the end of
?=
It is important to understand that syntactically,
=?ISO-8859-1?Q?flameeyes
is a perfectly valid local-part and
gmail.com_
is a perfectly valid domain. In both cases, the characters all comply
with the RFC 2822 dot-atom syntax. The only inkling to any parser that
anything is wrong is the unknown extra crud of "?=".
Now, suppose we interpret this word as a MIME encoded word instead of a
local-part@domain. We then get
From: flam...@gmail.com (Diego 'Flameeyes' Pettenò)
which looks reasonable to a human.
The problem is the in the From: line we can only interpret MIME encoded
words in a phrase (because the MIME specification says so, that's why!);
and if there is a phrase it must be followed by a route-addr (which is a
"<" local-part "@" domain ">" sequence). There is no route-addr,
therefore there is no phrase.
What's more, the interpretation of MIME encoded words happens AFTER the
components of the From: components are determined, not before.
The bottom line is that whatever generated
From: =?ISO-8859-1?Q?flam...@gmail.com_(Diego_'Flameeyes'_Petten=F2)?=
is broken. The GIGO (Garbage In, Garbage Out) principle applies.
I have no idea if the broken program was the sender's MUA (which
identifies itself as "Gnus v5.13"), Gmail, or fetchmail-6.3.8.
If, as you claim, Kmail displays this broken thing "correctly", then it
must be that Kmail (incorrectly) applied MIME encoded word decoding to a
From: that consists of a single word. This is a Kmail bug because it can
only do this as part of a phrase, and a phrase requires a following
route-addr.
If Kmail successfully replies to this From:, that is an even worse bug in
Kmail; that would mean that it applies MIME encoded word decoding without
regard to syntax.
Ian Hilt
On Fri, 16 May 2008 at 4:54pm -0700, Mark Crispin wrote:
> Ian -
>
> Thank you very much for supplying us with your test data. I now understand
> what is going on.
Thank you for your prompt and detailed response.
> Unfortunately, the problem is that the message that displays incorrectly is
> non-compliant with the email standard. Whatever generated that message is
> broken. Specific details follow.
[...]
> The bottom line is that whatever generated
> From:
> =?ISO-8859-1?Q?flam...@gmail.com_(Diego_'Flameeyes'_Petten=F2)?=
> is broken. The GIGO (Garbage In, Garbage Out) principle applies.
>
> I have no idea if the broken program was the sender's MUA (which identifies
> itself as "Gnus v5.13"), Gmail, or fetchmail-6.3.8.
I have since configured alpine to connect directly to
{imap.gmail.com/ssl}INBOX. I no longer observe this behavior.
After I read your message initially, I re-downloaded my emails via
fetchmail using IMAP just to once again confirm that the problem
was still there. It in fact was. That, seemingly, leaves
fetchmail-6.3.8. However, I really like alpine's IMAP support,
so I might ditch fetchmail completely.
> If, as you claim, Kmail displays this broken thing "correctly", then it must
> be that Kmail (incorrectly) applied MIME encoded word decoding to a From:
> that consists of a single word. This is a Kmail bug because it can only do
> this as part of a phrase, and a phrase requires a following route-addr.
>
> If Kmail successfully replies to this From:, that is an even worse bug in
> Kmail; that would mean that it applies MIME encoded word decoding without
> regard to syntax.
It does in fact do both things.
> -- Mark --
Again, thank you very much for your support in this matter.
Erik Quaeghebeur
>
>> If, as you claim, Kmail displays this broken thing "correctly", then it
>> must be that Kmail (incorrectly) applied MIME encoded word decoding to a
>> From: that consists of a single word. This is a Kmail bug because it can
>> only do this as part of a phrase, and a phrase requires a following
>> route-addr.
>>
>> If Kmail successfully replies to this From:, that is an even worse bug in
>> Kmail; that would mean that it applies MIME encoded word decoding without
>> regard to syntax.
On Sat, 17 May 2008, Ian Hilt wrote:
>
> It does in fact do both things.
I will file a bug report for kmail if neither of you has done so already
(or wishes to do it). May I assume you do not object to me blatantly
copying large parts of this post's parent & grandparent?
Cheers,
Erik
Ian Hilt
I have not done it and your assumption is correct.
> Cheers,
>
> Erik
Mark Crispin
> I will file a bug report for kmail if neither of you has done so already (or
> wishes to do it). May I assume you do not object to me blatantly copying
> large parts of this post's parent & grandparent?
Sounds like a good idea.
Erik Quaeghebeur
>>
>>> If, as you claim, Kmail displays this broken thing "correctly", then it
>>> must be that Kmail (incorrectly) applied MIME encoded word decoding to a
>>> From: that consists of a single word. This is a Kmail bug because it can
>>> only do this as part of a phrase, and a phrase requires a following
>>> route-addr.
>>>
>>> If Kmail successfully replies to this From:, that is an even worse bug in
>>> Kmail; that would mean that it applies MIME encoded word decoding without
>>> regard to syntax.
>
> On Sat, 17 May 2008, Ian Hilt wrote:
>>
>> It does in fact do both things.
>
> I will file a bug report for kmail
FYI: <http://bugs.kde.org/show_bug.cgi?id=69007>, already reported at the
end of 2003.
Erik
Ian Hilt
> I have since configured alpine to connect directly to
> {imap.gmail.com/ssl}INBOX. I no longer observe this behavior.
> After I read your message initially, I re-downloaded my emails via
> fetchmail using IMAP just to once again confirm that the problem
> was still there. It in fact was. That, seemingly, leaves
> fetchmail-6.3.8. However, I really like alpine's IMAP support,
> so I might ditch fetchmail completely.
Subsequent testing revealed that when copying from my Gmail
account to another account -- gmx.com -- the From line was
incorrectly MIME encoded. I also checked fetchmail's FAQ which
indicates that Gmail does not use a standards compliant imap
implementation. I suppose that lets fetchmail off the hook.
--
Ian Hilt
ian.hilt (at) gmx.com
GnuPG key: 0x4AFC1EE3