IDN and EAI Support in Thunderbird

[Jothan Frakes]

Hi-

I am a fellow volunteer on a separate project (the Public Suffix List) within the Mozilla community for quite a while, and I have been a member of the ICANN community since ICANN formed (and before).

Anyway, pardon this ask, as I am unfamiliar with the Thunderbird development realm.

I am writing about IDN and EAI support being added to Thunderbird, across all supported platforms.

Some context and background on IDN (Pardon this if you're familiar):

IDN = Internationalized Domain Names  (the right of the @)

EAI = Email Address Internationalizion  (Including the left of the @)

ICANN has added a significant number of Internationalized Domain Names (IDN) at the top level.

For quite a long time (I'll use Chinese as an example but this could be any allowed language - I've no bias) a person could obtain an address as IDN at the second level (ex: 中国.cc). Now countries and  companies have IDN at the top level as well. (ex: 中国.中国).  The expected behavior within software is that these domains get input or presented and appear in native form from my examples.  When sent to the resolver to perform lookups, these domains are converted to 'Punycode' (xn--fiqs8s.cc or xn--fiqs8s.xn-fiqs8s, respectively) and sent to DNS.

There are new registries that offer <SLD>.<TLD>.  There are libraries and well documented RFC for the normalization and conversion process on the to and from DNS aspect, called IDNA.

EAI is a little more complicated.  

While the domain name aspect of the process has standards in place, there is not a standard yet set for how the comprehensive email address is handled, most notably the left of the @ sign, but the objective for a user would be to have them be able to have any of the following email addresses work in the client, intermediate transports, and then reach and render correctly on the recipient side:

1] 中国@中国.中国

2] china@中国.中国

3] 中国@china.中国

4] china@china.中国

5] 中国@中国.cc

6] china@中国.cc

There are some really interesting circumstances that can occur with EIA where a right to left language like Hebrew or Arabic might be on the left hand of the @ with a left to right language TLD (Or Vice-Versa) which can reverse the display of the address and be potentially confusing.

CUTTING TO THE CHASE...
Not knowing decorum here, I'll just throw this out there....  Could I get an idea around a] IF these enhancements are possible / probable, and b] if yes, what perhaps costs are on this? I may be able to locate a patron for this effort / find / fund sourcing for accomplishing them. 

Also, I notice there is a day left in before Summer of Code deadline, but I suspect this request might be larger than a single line item (or two).  Is that an appropriate place or is this too late to reasonably accomplish this?

Again, pardon the question, and I have mad respect for the volunteers working on keeping Thunderbird going.  As a long-time volunteer on the PSL, I know it can be a lot of work and you're not expecting kudos - but I thank you for your time and service.

-Jothan

Jothan Frakes
Tel: +1.206-355-0230

[Joshua Cranmer 🐧]

On 2/19/2015 1:46 PM, Jothan Frakes wrote:

Hi-

I am a fellow volunteer on a separate project (the Public Suffix List) within the Mozilla community for quite a while, and I have been a member of the ICANN community since ICANN formed (and before).

Anyway, pardon this ask, as I am unfamiliar with the Thunderbird development realm.

I am writing about IDN and EAI support being added to Thunderbird, across all supported platforms.

Some context and background on IDN (Pardon this if you're familiar):

I am personally familiar enough to have produced a rant on thus subject, although the same is not true of everybody on the list :-)

There are some really interesting circumstances that can occur with EIA where a right to left language like Hebrew or Arabic might be on the left hand of the @ with a left to right language TLD (Or Vice-Versa) which can reverse the display of the address and be potentially confusing.

And case insensitivity, Unicode normalization, and things like C1 controls. Oh, and the IDNA2003/IDNA2008 charlie foxtrot.

CUTTING TO THE CHASE...
Not knowing decorum here, I'll just throw this out there....  Could I get an idea around a] IF these enhancements are possible / probable, and b] if yes, what perhaps costs are on this? I may be able to locate a patron for this effort / find / fund sourcing for accomplishing them.

I've thought about EAI and IDN in the past (I can't find the post right now). At present, we can send to an IDN email address, but that's about the limit of our support--the core code chokes on IDN (even if you use A-labels--some core Gecko code "helpfully" normalized that back into the U-label and screwed up the match).

The rough work list of what needs to be done:
1. Add support to use EAI/IDN addresses when composing messages, and handle appropriate downgrading when necessary [I'm already rewriting the MIME message building portion of the backend, and I'm definitely planning on making sure this gets addressed].
2. Add proper validation support for EAI/IDN addresses, and permit their use in appropriate places in the UI.
3. Ensure that we normalize addresses to the proper format for, e.g., database comparisons or searching.
4. Handle the IMAP and POP requests to enable EAI properly.
5. Support using an EAI/IDN email address as an account.
6. Miscellaneous other places where we need to support email addresses (e.g., S/MIME certificate matching).

The main problem of doing this work is first, one needs time to get all of this done, and, second, the major open-source implementations (e.g., postfix, cyrus) haven't implemented necessary protocol support yet, so I can't test real-world usage scenarios for EAI. (IDN is fairly simple test via use of VMs and DNS).

Also, I notice there is a day left in before Summer of Code deadline, but I suspect this request might be larger than a single line item (or two).  Is that an appropriate place or is this too late to reasonably accomplish this?

I'm not sure this is a good match for a GSoC project. The parts that don't rely on protocol support boil down to a few weeks or a month of hunting down issues in the UI and fixing them--not enough to fill out the summer. Protocol support requires having access to test servers which are not easy to find or set up locally. Nevertheless, I would be happy to help someone else work on this topic outside of GSoC if they were available, or maybe even through GSoC if EAI-compliant IMAP and SMTP servers were available to a student.

-- 
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

[Jothan Frakes]

Thanks Joshua...  

I agree, GSoC sounded ambitious due to deadline and possibly larger than scope and time available in GSoC.  At least for this year.

I think your list was helpful, and i authentically suspect that within that list there are a few what I call "Russian Doll" (open, new doll inside.  Open, discover new doll inside that one, repeat.) issues, dependencies, etc. that will likely need attention.

As far as abstracts that help describe the topic... I have some URLs that describe the IDN & EAI a bit better that are written up by John Levine.

http://jl.ly/Email/i18n.html
http://jl.ly/Email/i18n2.html
http://jl.ly/Email/i18n3.html

http://jl.ly/Email/i18neai.html
http://jl.ly/Email/i18neai2.html
http://jl.ly/Email/i18neai3.html

With respect to growing my understanding to where I could have conversations with potential patrons about supporting the effort to bring Thunderbird these functions, I would need to find out timing, costs, and scope.  

Also, I am not clear on how such "sponsors" could lend their support to TB for this (ie Would they have to hire some of the volunteers on contract to donate to them directly, fund through Mozilla, etc.) 

We have an opportunity to potentially have the effort get a patron or two - but the conversation to have with folks who are outside the "Mozillasphere" and are commercial players will require real pragmatic project details (how much, what does it get, how long before it starts, how long will it take them, etc).

 I need to understand these things, and/or if this is too ambitious or can't be reasonably accomplished.

-jothan

--

Jothan Frakes
Tel: +1.206-355-0230