OEAPI vs Unicode

[alexander goldberg]

Hi,

Whether OEAPI is totally incompatible with Unicode?

I played around international messages and found OEAPI cannot process
Unicode data correctly. All functions I tried (GetBodyProp,
GetSubject, GetFilename) damaged Unicode data which were not
compatible with current user Windows locale.

For testing I generated 3 files with Russian, Hebrew and Arabic names.
I created new message, attached these files, placed same 3 Russian/
Hebrew/Arabic words into "Subject" and into "Body" and sent message.

Then I receive this message from the mail server and made sure Outlook
Express shows Subject, Body and Attachments correctly.

Then I started my Delphi plugin and OEAPI VB Demo and I could see ALL
non-english and non-european symbols were converted into question
signs ("?")

I tried to create other messages which contains only Russian, only
Arabic or only Hebrew texts with the same result. I repeated these
tests in Windows Vista and same problems occured.

Its catastrophe for me. I must not create Unicode-incompatible plugin.
(cry)

Dear Nektra support, is there any way to modify OEAPI for Unicode-
compatibility during next 1-2 monthes or I must forget about this
beautiful library, do not purchase it but spend many-many days in OE/
WM investigation by myself ?

Thank you,
Alex Goldberg

Required details:
OEAPI 3.2.1 evaluation
Windows XP SP2
Outlook Express 6.00.2900.2180
Borland Delphi 2007 / OEAPI VB demo

[Nektra OEAPI Support]

Hi again Alex,

OEAPI is enconding agnostic, in order to use non-english characters you
must encode them as specified with RFC 2047.
Outlook Express is an ANSI application and requires the operating
system's support for other languages to be installed to properly display
non-english characters, if the required fonts are not installed it will
display question marks (?) instead.

Other limitation of OE is that it will not display the subject correctly
if the correct charset is not specified in a MIME header. It requires
this in the header of the message:

MIME-Version: 1.0
Content-Type: text/plain;
format=flowed;
charset="utf-8";
reply-type=original
Content-Transfer-Encoding: 8bit

You may need to change this if you are using other encoding or html format.

Thank you for contacting Nektra's OEAPI Support.

Regards,

Federico

--
Nektra Advanced Computing
http://www.nektra.com

[alexander goldberg]

Hi Federico,

> OEAPI is enconding agnostic,

I suspected that COM server cannot be "encoding agnostic" but it must
work in Unicode. All OEAPI interfaces provide string data as BSTR.
Microsoft claims here http://msdn2.microsoft.com/en-us/library/ms221069.aspx
that BSTR's datastring is a "string of unicode characters (wide
characters, also known as double byte characters)."

Since OEAPI reads/writes parts of e-mail messages into BSTR it must
encode different MIME single-byte sequences into double-byte BSTR.
Therefore your library should contain parts of code which decode/
encode MIME. Am I right?

Now I'm talking about error in the MIME encoding in OEAPI. Outlook
Express encoded into MIME all non-english characters in the name of
attachments when the message was created.

Now I call OEMessage.GetFilename function and it should return me
attachment's filename as BSTR. OEAPI returns me filename in BSTR
neither as valid Unicode data nor MIME single-byte string but as
string of the "question" marks.

>in order to use non-english characters you
> must encode them as specified with RFC 2047.

Please take into account - I've spoken about incoming message which
had been created by remote mail client, and this message was CORRECTLY
mime-encoded. Now OE/WM display this incoming message correctly but
OEAPI does not.

> Outlook Express is an ANSI application and requires the operating
> system's support for other languages to be installed to properly display
> non-english characters, if the required fonts are not installed it will
> display question marks (?) instead

You are not right. OE6 is partially Unicode-compatible. As I wrote
already I tested message with Russian/Arabic/Hebrew characters and OE
displayed Attachments, Subject and Body text properly (except
message's subject in the message list box).

Moreover I can't but mention that Windows Mail is fully Unicode-
compatible and it displays multi-language messages absolutely
correctly. But OEAPI under WM does not work properly with characters
which are not present in the current User Locale (not in the installed
fonts).

> Other limitation of OE is that it will not display the subject correctly
> if the correct charset is not specified in a MIME header. It requires
> this in the header of the message:
>
> MIME-Version: 1.0
> Content-Type: text/plain;
> format=flowed;
> charset="utf-8";
> reply-type=original
> Content-Transfer-Encoding: 8bit

As I have already stated OE does not display mime-encoded Subject
correctly in the messages list box but it displays Unicode in the
message view box and in the Message View/Edit windows too.

> You may need to change this if you are using other encoding or html format.

I'm sorrry for possible misunderstanding between us. Now I try to read
just incoming messages, and I cannot influence these messages creation
process but I'm sure that they were created correctly (OE and WM show
these messages without problem).

I'm ready to send you the above mentioned message with multi-languages
data strings as .eml file and I'm ready to upload somewhere
screenshots with correct OE and incorrrect OEAPI demo screens. Fill
free to contact me for any additional information you need, to help
you reproduce the problem I've come across.

Hope hearing from you soon, since your reply is crucial for my further
plans and actions.

Alex Goldberg

[alexander goldberg]

Hi Federico,

> I'm ready to send you the above mentioned message with multi-languages
> data strings as .eml file and I'm ready to upload somewhere
> screenshots with correct OE and incorrrect OEAPI demo screens. Fill
> free to contact me for any additional information you need, to help
> you reproduce the problem I've come across.

I created several screen shots and uploaded them with .eml file for
previewing. Please look at these pictures for the problem
understanding.

Unicode multi-language (Russian, Hebrew, Arabic) message .eml file in
ZIP archive is here:
http://widenet.ks.ua/sok_shots/unicode.zip

Next screen shots have been created in Windows XP. Outlook Express
6.00.2900.2180 and OEAPI VB demo are running.
Please take into account that current user's locale is "Russian".
Since OEAPI reads and displays "Russian" texts correctly but other non-
english characters were converted into 'question marks'.

http://widenet.ks.ua/sok_shots/oeapi_xp_subjects.png
http://widenet.ks.ua/sok_shots/oeapi_xp_file_attachments.png
http://widenet.ks.ua/sok_shots/oeapi_xp_plain_bodies.png

Next screen shots have been created in Windows Vista. Windows Mail
6.00.6000.16386 and OEAPI VB demo are running.
Please take into account that current user's locale is "Spanish".
Since OEAPI reads and displays any non-english characters as 'question
marks'. WM shows all text data correctly.

http://widenet.ks.ua/sok_shots/oeapi_vista_subjects.png
http://widenet.ks.ua/sok_shots/oeapi_vista_file_attachments.png
http://widenet.ks.ua/sok_shots/oeapi_vista_plain_bodies.png

Fill free to contact me for any additional information you need, to
help you reproduce the problem I've come across.

Alex Goldberg

[Nektra OEAPI Support]

Hi Alex:

alexander goldberg wrote:
> Since OEAPI reads/writes parts of e-mail messages into BSTR it must
> encode different MIME single-byte sequences into double-byte BSTR.
> Therefore your library should contain parts of code which decode/
> encode MIME. Am I right?
>

The enconding and decoding operations are done by OE/WinMailf, OEAPI
only requests that data to them.

> Now I'm talking about error in the MIME encoding in OEAPI. Outlook
> Express encoded into MIME all non-english characters in the name of
> attachments when the message was created.
>
> Now I call OEMessage.GetFilename function and it should return me
> attachment's filename as BSTR. OEAPI returns me filename in BSTR
> neither as valid Unicode data nor MIME single-byte string but as
> string of the "question" marks.
>

This seems to be a bug in the GetFilename() and GetBodyProp() methods of
OEAPI.

> Please take into account - I've spoken about incoming message which
> had been created by remote mail client, and this message was CORRECTLY
> mime-encoded. Now OE/WM display this incoming message correctly but
> OEAPI does not.
>

Sorry, I've erroneously interpreted that you wanted to write new
messages, not to read existing ones.

> You are not right. OE6 is partially Unicode-compatible. As I wrote
> already I tested message with Russian/Arabic/Hebrew characters and OE
> displayed Attachments, Subject and Body text properly (except
> message's subject in the message list box).
>

We know that OE is not a Unicode application and some properties like
To, From, Subject are stored as ANSI strings, regardless of the encoding
specified in the messages, that's the information OE displays in the
message list, and that's the information OEAPI gets, so this fields
won't appear correctly. Nevertheless this properties will still be
readable directly from the messages using the GetBodyProp() method, once
it's fixed.

> Moreover I can't but mention that Windows Mail is fully Unicode-
> compatible and it displays multi-language messages absolutely
> correctly. But OEAPI under WM does not work properly with characters
> which are not present in the current User Locale (not in the installed
> fonts).

We are using the same OE interfaces that that work with ANSI on Windows
Mail, we will make research about WM's Unicode interfaces, so we'll need
time to check this out.

> I'm ready to send you the above mentioned message with multi-languages
> data strings as .eml file and I'm ready to upload somewhere
> screenshots with correct OE and incorrrect OEAPI demo screens. Fill
> free to contact me for any additional information you need, to help
> you reproduce the problem I've come across.
>
> Hope hearing from you soon, since your reply is crucial for my further
> plans and actions.
>
> Alex Goldberg
>

Thanks for providing those messages, we have checked this out and it
seems that this is a bug on GetBodyProp() and GetFilename() methods.

Thank you for your reports.

[alexander goldberg]

Hi Federico,

> > Now I call OEMessage.GetFilename function and it should return me
> > attachment's filename as BSTR. OEAPI returns me filename in BSTR
> > neither as valid Unicode data nor MIME single-byte string but as
> > string of the "question" marks.
>
> This seems to be a bug in the GetFilename() and GetBodyProp() methods of
> OEAPI.

Please take you account on other methods which returns BSTR data too.
For example GetSubject() and GetNormalSubject() methods have same
bug.

> We know that OE is not a Unicode application and some properties like
> To, From, Subject are stored as ANSI strings, regardless of the encoding
> specified in the messages, that's the information OE displays in the
> message list, and that's the information OEAPI gets, so this fields

Hmm... I just checked non-european characters in "To" and "From"
header properties (OE6). Really some problem exists when OE inserts
data from the "Address Book" (it seems Address Book is not Unicode-
compatible by itself).

But in case you enter address manually it will be stored as MIME
correctly. Moreover OE6 create/read/display 'Subject' property
correctly (as I mentioned earlier). Look at the my new message's
header please:

Return-Path: <a...@senderok.han.ks.ua>
Delivered-To: a...@senderok.han.ks.ua
Message-ID: <003b01c84145$80f68e90$8201...@office.han.ks.ua>
From: <a...@senderok.han.ks.ua>
To: =?utf-8?B?2KfZhNio2K3YqyDRgNGD0YHRgdC60LjQuSDXkNeZ16DXmNeo16DXmA==?
= <a...@senderok.han.ks.ua>
Subject: =?utf-8?B?
2KfZhNio2K3YqyDRgNGD0YHRgdC60LjQuSDXkNeZ16DXmNeo16DXmA==?=
Date: Tue, 18 Dec 2007 09:13:34 +0200
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_0038_01C84156.445BA9F0"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.3138
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198

> won't appear correctly. Nevertheless this properties will still be
> readable directly from the messages using the GetBodyProp() method, once
> it's fixed.

Thank yoy very much. Can you clarify yet another question please:

If I'll create message by myself and use SetBodyProp(),
SetBodyFromFile(), AttachFile() whether I must specify non-english
text/filenames as Unicode data string (BSTR) or I have to encoded them
into MIME by myself beforehand?

> Thanks for providing those messages, we have checked this out and it
> seems that this is a bug on GetBodyProp() and GetFilename() methods.

Thank you very much for your excellent support.

Alex Goldberg

[Nektra OEAPI Support]

Hi  again Alex:

alexander goldberg wrote:

Please take you account on other methods which returns BSTR data too.
For example GetSubject() and GetNormalSubject() methods have same
bug.
  

I repeat, this is a limitation of OE because it stores From, To and Subject as LPSTR strings, so this will not change, but using the fixed GetBodyProp() method and specifying OE_PID_HDR_FROM/TO/SUBJECT as propId you will correctly get what's stored in the message.

Hmm... I just checked non-european characters in "To" and "From"
header properties (OE6). Really some problem exists when OE inserts
data from the "Address Book" (it seems Address Book is not Unicode-
compatible by itself).

But in case you enter address manually it will be stored as MIME
correctly. Moreover OE6 create/read/display 'Subject' property
correctly (as I mentioned earlier). 

OE is not a Unicode application, it calls functions like MessageBoxA instead of MessageBoxW. That's why some properties, like the Subject, are stored as ANSI strings for fast access, regardless of the original encoding.
OEAPI makes queries to OE to quickly access this properties, and it returns incorrect data if it's not encoded for the default locale, still GetBodyProp() accesses the content of the message without quering OE.

If I'll create message by myself and use SetBodyProp(),
SetBodyFromFile(), AttachFile()  whether I must specify non-english
text/filenames as Unicode data string (BSTR) or I have to encoded them
into MIME by myself beforehand?
  

We will need to make some tests about this, GetBodyProp() has a bug with the Unicode contained in a BSTR and it is possible that this also affects SetBodyProp.

Thank you very much for your excellent support.

You are welcome. Thank you for your reports.


Regards,

Federico.

[alexander goldberg]

Hi again Federico,

> I repeat, this is a limitation of OE because it stores From, To and Subject as LPSTR strings, so this will not change,

>OEisnota Unicode application, it calls functions like MessageBoxA instead of MessageBoxW. That's why some properties, like the Subject,

>are stored as ANSI strings for fast access, regardless of the original encoding.
> OEAPI makes queries to OE to quickly access this properties, and it returns incorrect data if it's not encoded for the default locale, still
>GetBodyProp() accesses the content of the message without quering OE.

Ok, I think I understand the problem. I suspect you mean OE has
special interface methods which returns "Subject" etc incorrectly
decoded from the UTF-8 in the single-byte string?

It's really a hard problem. I hope you could solve it. By the way OE
(same as WM) has header's data encoded by UTF8-MIME. Since it could
read and display Unicode symbols correctly:
.................................
Subject: =?utf-8?B?
2KfZhNio2K3YqyDRgNGD0YHRgdC60LjQuSDXkNeZ16DXmNeo16DXmA==?=
.................................

>>If I'll create message by myself and use SetBodyProp(), SetBodyFromFile(), AttachFile() whether I must specify non-english text/filenames as Unicode
>>data string (BSTR) or I have to encoded them into MIME by myself beforehand?
>We will need to make some tests about this, GetBodyProp() has a bug with the Unicode contained in a BSTR and it is possible that this also
>affects SetBodyProp

Thank you again. Inform me about results of your tests please.

Alex Goldberg

[alexander goldberg]

Hi Federico,

Have you any good news about this problem with incorrect decoding
Unicode data in OE/WM I reported last year?

We have to release first public version of our own software soon but
incorrect decoding of international messages really hamper the
progress.

Is there any chance we'll receive bugfix or new OEAPI version in the
near future?

Alex Goldberg


On Dec 19 2007, 9:58 am, alexander goldberg

<alexander.goldb...@webceo.ks.ua> wrote:
> Hi again Federico,
>
> > I repeat, this is a limitation of OE because it stores From, To and Subject as LPSTR strings, so this will not change,

> >OEisnotaUnicodeapplication, it calls functions like MessageBoxA instead of MessageBoxW. That's why some properties, like the Subject,

> >are stored as ANSI strings for fast access, regardless of the original encoding.
> > OEAPI makes queries to OE to quickly access this properties, and it returns incorrect data if it's not encoded for the default locale, still
> >GetBodyProp() accesses the content of the message without quering OE.
>
> Ok, I think I understand the problem. I suspect you mean OE has
> special interface methods which returns "Subject" etc incorrectly
> decoded from the UTF-8 in the single-byte string?
>
> It's really a hard problem. I hope you could solve it. By the way OE
> (same as WM) has header's data encoded by UTF8-MIME. Since it could

> read and displayUnicodesymbols correctly:

> .................................
> Subject: =?utf-8?B?
> 2KfZhNio2K3YqyDRgNGD0YHRgdC60LjQuSDXkNeZ16DXmNeo16DXmA==?=
> .................................
>
> >>If I'll create message by myself and use SetBodyProp(), SetBodyFromFile(), AttachFile() whether I must specify non-english text/filenames asUnicode
> >>data string (BSTR) or I have to encoded them into MIME by myself beforehand?

> >We will need to make some tests about this, GetBodyProp() has a bug with theUnicodecontained in a BSTR and it is possible that this also

[Nektra OEAPI Support]

Hi Alex:

As we informed, we have been working on fixing the GetBodyProp() method, so specifying OE_PID_HDR_FROM/TO/SUBJECT as propId on the next release would correctly get the message's properties.

We'll let you know when the release is ready.

Thank you for contacting Nektra's OEAPI Support.

Regards,

Federico