Exchange Server 2003 & iPhone character set issues
Gerald Vogt
I am trying to fix an issue with iPhones accessing mailboxes on a
Japanese Exchange Server 2003 (part of an SBS 2003) with ActiveSync.
Generally it works including Push Mail and calender sync etc.
Only for some messages the iPhone cannot show the contents of the e-mail
but shows an error "This message has not been downloaded from the server."
So far I have found out that it is due to a character encoding issues. I
guess it is because it is a Japanese Windows and Exchange server and the
issues are with other non-Japanese 8 bit character sets.
For tests I sent four e-mail messages from Thunderbird (messages from
Outlook seem to work fine) to a mail box on that server with different
encodings (if it matters: it is a single server with backend and
frontend on the same machine).
1. body contains only 7bit characters.
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
2. body contains 8bit characters (e.g. äöü)
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
3. body is utf-8 encoded and contains umlauts (äöü)
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
4. body contains 8 bit characters (äöü) with quoted-printable
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable
All four messages show the correct header lines and each message is
correctly encoded. All messages look correctly when accessing the
mailbox through Outlook, OWA, or even when accessing the OMA URL through
a normal web browser.
However, on the iPhone only message 1 & 4 are O.K. For message 2 the
iPhone does not show the body but then mentioned error message instead.
For message 3 the iphone shows some random Japanese characters instead
of the umlauts (äöü).
To me it seems as if it is some character encoding issue, either on the
Exchange server or on the iPhone. In particular, as the utf-8 encoded
message no. 3 shows some random Japanese characters I think it is
because the Exchange server is Japanese.
Unfortunately I don't know Exchange Server 2003 very well. Is there an
easy way to debug what the server sends to the iPhone?
Or are there any settings which influence what character encodings the
server uses for ActiveSync access? There are only iPhones for mobile
access as this time. But of course, changing character sets should not
break compatibility with other character sets: the server must still
correctly handle message in Japanese using japanese character sets,
German using iso-8859-1, and sometimes even mixed (utf-8 encoded, I think).
Thanks a lot!
Gerald
John Fullbright
quoted-printable encoding at the discretion of the sending application.
It's the task of the receiving application (the iPhone) to interpret it
correctly. In that it does not, the issue is with the iPhone. Have you
tried posting on an iPhone discussion list?
"Gerald Vogt" <vo...@spamcop.net> wrote in message
news:uhCC3y3f...@TK2MSFTNGP02.phx.gbl...
Gerald Vogt
> The body is text and therefore can be sent in either 7bit, 8bit, or
> quoted-printable encoding at the discretion of the sending application.
Correct. But the Exchange Server does conversions at various places,
e.g. for OWA. Are you sure the Server does not make any conversions for
OMA or ActiveSync?
> It's the task of the receiving application (the iPhone) to interpret it
> correctly. In that it does not, the issue is with the iPhone. Have you
> tried posting on an iPhone discussion list?
The iPhone does interpret any of those encodings correctly, if I access
non-exchange servers through IMAP. It does interpret the encodings and
the contents correctly. It also shows utf-8 contents correctly and does
not convert some utf-8 characters to japanese characters.
To me, these japanese characters strongly suggest that it is related
with the Exchange Server, because that's the end that is Japanese. How
should the iPhone know that the Exchange Server is Japanese??
Gerald
Milind Naphade
domain controller servers? If not any language other than English wont be
rendered to its respective characters by Exchange Servers.
M
"Gerald Vogt" <vo...@spamcop.net> wrote in message
news:#XQhmLAg...@TK2MSFTNGP05.phx.gbl...
Gerald
--
Milind Naphade
Gerald Vogt
> Do you have the language pack installed on all your backend, front end and
> domain controller servers? If not any language other than English wont be
> rendered to its respective characters by Exchange Servers.
It's a Japanese Windows 2003 SBS with a Japanese Exchange Server 2003. I
don't think there are any language packs installed (how would I check
that?).
Gerald
John Fullbright
Control panel - regional settings.
I don't believe OWA or OMA is going to decode the message and then encode it
again. Even if it did, Exchange would default to UTF8 and quoted printable
which was your scenario 4. The application that sent the message does the
encoding. In your case, it seems that every time it's using an 8bit
encoding scheme, the iPhone application tanks.
If you used an Outlook client that didn't have the language support files
for <language of your choice>, and sent it an 8bit (non utf) message part,
then you's see question marks. You would see them precisely because
Exchange didn't decode/encode or otherwise mangle the data. That's why I
think it's an iPhone problem. The language support aspect is a good point
though. Does the iPhone have the necessary language support files installed
for the language you are trying to render? I think I'd try that line of
thought first.
John
"Gerald Vogt" <vo...@spamcop.net> wrote in message
news:emlLfCHg...@TK2MSFTNGP03.phx.gbl...
John Fullbright
encoding, I'd hazard a guess that in scenario two and three wher it's 8bit
encoded and non-UTF, it was using a DBCS character set that the iPhone does
not have the support files installed for.
?????????????????????????.
John
"Gerald Vogt" <vo...@spamcop.net> wrote in message
news:emlLfCHg...@TK2MSFTNGP03.phx.gbl...
Gerald Vogt
> Language support on an Exchange 2003 server is a Windows 2003 thing.
> Control panel - regional settings.
Thanks. I'll check that.
> I don't believe OWA or OMA is going to decode the message and then encode it
> again. Even if it did, Exchange would default to UTF8 and quoted printable
This seems not correct:
1. OWA must encode the contents differently because it delivers the
contents in HTML. At least characters like ampersand & must be encoded
to display properly.
2. OWA uses iso-8859-1 in HTML for all my tests messages here, even
those sent with UTF-8 encoding. Japanese characters are sent using HTML
entities.
> which was your scenario 4. The application that sent the message does the
> encoding. In your case, it seems that every time it's using an 8bit
> encoding scheme, the iPhone application tanks.
Yes. I agree with you. Each time 8bit characters are sent through
ActiveSync there is a problem. But the question is: does the Exchange
Server delivers something incorrectly through ActiveSync or does the
iPhone interpret it incorrectly when received through ActiveSync. The
iPhone handles 8bit characters correctly when received through IMAP.
> If you used an Outlook client that didn't have the language support files
> for <language of your choice>, and sent it an 8bit (non utf) message part,
> then you's see question marks. You would see them precisely because
> Exchange didn't decode/encode or otherwise mangle the data. That's why I
> think it's an iPhone problem. The language support aspect is a good point
> though. Does the iPhone have the necessary language support files installed
> for the language you are trying to render? I think I'd try that line of
> thought first.
The iPhone renders any e-mail in any encoding of my examples correctly
if the e-mail is delivered through IMAP. There is definitively nothing
missing on the iPhones do show each of those messages correctly.
The problem only appears if the messages are delivered from an Exchange
Server through ActiveSync. Thus either the iPhone handles the messages
through ActiveSync differently or the Exchange Server delivers something
differently (or both). I don't know how to find out what the actual
problem is. I don't know whether it is possible to tell the Exchange
Server to make a debugging dump of the data delivered to the iPhone. The
iPhone does not allow this.
Gerald
Gerald Vogt
> Having more that a bit of experience supporting DCBS characters an 8bit
> encoding, I'd hazard a guess that in scenario two and three wher it's 8bit
> encoded and non-UTF, it was using a DBCS character set that the iPhone does
> not have the support files installed for.
No. 2 is ISO-8859-1 8bit and No. 3 is UTF-8 8bit. No. 2 is not double
byte. No. 3 is standard UTF-8 which is displayed correctly with other
mail servers. E-mails with umlauts and japanese characters mixed sent as
utf-8 are received and rendered absolutely correctly on the iPhone when
delivered through IMAP.
Gerald
Zenon Mousmoulas
Apologies for digging up an old thread.
On Wednesday, January 28, 2009 5:20:45 AM UTC+2, Gerald Vogt wrote:
> [...]
> if the e-mail is delivered through IMAP. There is definitively nothing
> missing on the iPhones do show each of those messages correctly.
>
> The problem only appears if the messages are delivered from an Exchange
> Server through ActiveSync. Thus either the iPhone handles the messages
> through ActiveSync differently or the Exchange Server delivers something
> differently (or both). I don't know how to find out what the actual
> problem is. I don't know whether it is possible to tell the Exchange
> Server to make a debugging dump of the data delivered to the iPhone. The
> iPhone does not allow this.
I send an e-mail which is delivered to an Exchange 2003 SP2 server. This is a single FE/BE server also running ActiveSync.
The mail looks like this:
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
The body is correctly UTF-8 encoded, I have confirmed this by snooping the SMTP traffic when it is delivered to Exchange SMTP. It contains this line:
ουαν μορ δοκιμή utf-8
It corresponds to the following hex byte stream (greek characters are 2 bytes each):
CEBF CF85 CEB1 CEBD 20 CEBC CEBF CF81 20 CEB4 CEBF CEBA CEB9 CEBC CEAE 20 75 74 66 2D 38 0D 0A
Encoding this byte stream according to ISO-8859-7, where each character is one byte, I come up with a totally different set of characters.
The mapping is provided here:
http://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-7.TXT
But this shows conveniently in the same table what each byte maps to in each encoding:
http://michelf.com/docs/utf8-confusion.php?from=896&to=959
http://michelf.com/docs/utf8-confusion.php?from=960&to=1023
This is the result:
ΞΏΟ<0x85 control>Ξ±Ξ½ ΞΌΞΏΟ<0x81 control> δοκιμΞ<0xAE not-in-table> utf-8
Some characters (such as 0xAE) don't look good, so I also tried encoding according to similar-related Windows-1253:
http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1253.TXT
It gives this result:
ΞΏΟ<0x85 horizontal ellipsis>Ξ±Ξ½ ΞΌΞΏΟ<0x81 undefined> δοκιμΞ<0xAE registered sign> utf-8
Then I convert to UTF-8 and arrive to the following byte stream. You can observe there is a mix of ISO-8859-7 and Windows-1253 for some "special" characters; it is not a mistake.
CE9E CE8F CE9F E280A6 (=windows-1253 0x85) CE9E C2B1 CE9E C2BD 20 CE9E CE8C CE9E CE8F CE9F C281 (=iso-8859-7 0x81) 20 CE9E CE84 CE9E CE8F CE9E CE8A CE9E CE89 CE9E CE8C CE9E C2AE (=windows-1253 0xAE) 20 75 74 66 2D 38 0D 0A
Encoded it looks like this:
ΞΏΟ…Ξ±Ξ½ ΞΌΞΏΟ Ξ΄ΞΏΞΊΞΉΞΌΞ® utf-8
This is exactly what Exchange serves to iOS devices via ActiveSync. Following the protocol specification, the MIME message (complete with headers) comes in a MIMEData element, inside WBXML (WAP Binary XML), within the HTTP response (Content-Type: application/vnd.ms-sync.wbxml). WBXML is UTF-8 encoded, the third byte (0x6A) in the WBXML byte stream is the tag signaling this, but I think it's the only supported encoding anyway. This response comes after iOS issues Fetch for the particular item, containing a MIMESupport tag with value 2.
Before that, iOS sends a Sync request for the same message with MIMESupport=2 and Truncation/MIMETruncation tags with value 1. The response includes the truncated message body (no headers) in a Body tag. This is the text iOS shows as a preview in the message list. It should be noted this does not include the garbled line shown above, but rather the original, correctly-encoded UTF-8 byte stream.
It is also noteworthy that in both cases there is an InternetCPID tag with a value of 65001, which is the code page ID for UTF-8.
I have confirmed the above by snooping on the HTTP traffic.
As noted in previous posts, this happens only for ActiveSync clients. I have only tested this with iOS; I tried to do the same with some Android phone, but it would only Fetch the Body (MIMESupport=0). Other clients (OWA/OMA, MAPI, IMAP) get the original message intact.
Obviously Exchange ignores the content-type header in the message and/or mis-interprets the body as something other than UTF-8, actually a (weird) mix of ISO-8859-7 and Windows-1253. But then it thinks it should convert to UTF-8 in order to deliver the message to the EAS client. When the client sees the message, it can no longer recover from this double conversion, even if it would override the content-type header. Therefore I think it is safe to say that iOS can do nothing but present the garbled message, as shown above.
I have no idea what Exchange component may be responsible for this mis-conversion, but I think it happens on the fly, since the original message seems to be preserved and delivered to every other client.
I have also not found any knobs to control this behavior. I find it unlikely there would be any, as this almost certainly feels like a bug.
I have only found one workaround, or actually less than half of that:
I can turn off 8bitmime ESMTP verb advertisement by Exchange SMTP, then put an MX in front of it (such as Postfix) that will conform to this and do MIME output conversion to quoted-printable when relaying to Exchange. This takes care of inbound mail, at the expense of breaking things such as DKIM signatures on 8-bit mail.
However Exchange SMTP does not enforce this and will happily accept 8-bit CTE-mail, despite the absence of the verb. Therefore, a Mozilla Thunderbird user, for example, can still send 8-bit mail that will be delivered garbled to another user's iOS device. One could move e-mail submission by such clients to a separate MTA, so as to have mail converted again by the time it reaches Exchange; but the workaround gets somewhat too complicated.
I can only hope someone has a better idea for a workaround and/or can provide some insight about what goes wrong with Exchange ActiveSync in this case.
Thanks,
Z.
alie...@gmail.com
My workaround for this:
* Disable LookOut extension for Thunderbird
* In Thunderbird Edit->Preferences->Display->Formatting->Advanced:
Set Fonts For: Turkish
Font Control: Both choices checkboxes selected
Character Encodings: Outgoing->ISO-8859-9, Incoming->UTF-8
"When possible, use the default character encoding in replies" <-UNCHECKED
--
AliGeyik.com
vbal...@gmail.com
"I can turn off 8bitmime ESMTP verb advertisement by Exchange SMTP, then put an MX in front of it (such as Postfix) that will conform to this and do MIME output conversion to quoted-printable when relaying to Exchange. This takes care of inbound mail, at the expense of breaking things such as DKIM signatures on 8-bit mail."
Zenon Mousmoulas
> Where on the exchange server do you do this?
>
>
>
> "I can turn off 8bitmime ESMTP verb advertisement by Exchange SMTP, then put an MX in front of it (such as Postfix) that will conform to this and do MIME output conversion to quoted-printable when relaying to Exchange. This takes care of inbound mail, at the expense of breaking things such as DKIM signatures on 8-bit mail."
>
Subtract 4194304 to turn off 8bitmime. The smtp server should pick up the new value shortly after that.