Thunderbird is still using two MIME decoders, jsmime and Core::Network - Why?

[Jörg Knobloch]

Hi all,

If you're interested, please take a look at
https://bugzilla.mozilla.org/show_bug.cgi?id=1146099#c22.

Thunderbird uses two MIME decoders, the jsmime one and the M-C::Network one.

The jsmime one is used here:
https://dxr.mozilla.org/comm-central/search?q=mimeConverter-%3EDecodeMimeHeader&redirect=false
(five times).

The M-C:Network one is used here:
https://dxr.mozilla.org/comm-central/search?q=DecodeRFC2047Header&redirect=false
(once).

That means that the entire system has been changed over to jsmime, minus
the one call site in
mailnews/mime/src/comi18n.cpp

The two decoders give different results in some cases and therefore
cause inconsistencies in the UI. Specifically, the jsmime decoder works
better, that means, it is more error tolerant to malformed headers.
That's what this bug 1146099 is about.

I don't know why the last call site wasn't changed over, maybe is was
left out deliberately. The only reason I could think of would be a
threading issue. Apparently XPCOM written in JavaScript must run on
Gecko's main thread; that was the issue why Outlook import doesn't work
any more since jsmime was introduced.

Simply switching decoders in comi18n.cpp gives the desired result in
manual testing and Mozmill tests but causes massive Xpcshell test failures.

Any insight, especially from Joshua, would be most welcome.

Jörg.

[Joshua Cranmer 🐧]

On 7/12/2016 2:43 AM, Jörg Knobloch wrote:
> I don't know why the last call site wasn't changed over, maybe is was
> left out deliberately. The only reason I could think of would be a
> threading issue. Apparently XPCOM written in JavaScript must run on
> Gecko's main thread; that was the issue why Outlook import doesn't
> work any more since jsmime was introduced.

The main reason was because the that last usage was "internal" to
libmime. I remembered there being something in relation to RFC 2231
decoding, but a closer look suggests that my memory is fallible. It was
deliberate but also justified in part that libmime was going to die
before too long, which has obviously not turned out to be the case. :-(

There's no terribly good reason to not switch that over (except for
perhaps perf).

--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

[Jörg Knobloch]

On 14/07/2016 17:53, Joshua Cranmer 🐧 wrote:
> There's no terribly good reason to not switch that over (except for
> perhaps perf).

It's been switched over ;-)

https://hg.mozilla.org/comm-central/rev/32170e6fb2983623cb5351f742139ff841476b9b

Jörg.

[Jörg Knobloch]

On 14/07/2016 17:53, Joshua Cranmer 🐧 wrote:

There's no terribly good reason to not switch that over (except for perhaps perf).

To continue a discussion from July 2016:

It turns out that for an attachment with 300.000 lines of base64, the header(!!) decoder is called 600.000 on the filename. There we notice that XPCOM + JS is somewhat slower than C++.

Anyway, I reduced the calls to just two ;-)

Jörg.

[ISHIKAWA,chiaki]

On 2017/08/07 7:35, Jörg Knobloch wrote:
> On 14/07/2016 17:53, Joshua Cranmer 🐧 wrote:

>> There's no terribly good reason to not switch that over (*except for
>> perhaps perf*).

>
> To continue a discussion from July 2016:
>
> It turns out that for an attachment with 300.000 lines of base64, the
> header(!!) decoder is called 600.000 on the filename. There we notice
> that XPCOM + JS is somewhat slower than C++.
>
> Anyway, I reduced the calls to just two ;-)
>
> Jörg.
>
>

Great!

It seems that there are other performance issues in TB, which I noticed
during debugging such as scanning message file and rebuilding database
THREE times after a compaction (I can live with two, but three is a
little excessive.)