ICU proposing to drop support for WinXP (and OS X 10.6)

[Jonathan Kew]

We make considerable (and growing) use of ICU for various aspects of
i18n support in Gecko.†

The ICU project is proposing to drop support for Windows XP and OS X
10.6 in version 58; I guess this will be released sometime shortly after
Unicode 9.0, which is due to appear in June.

Markus (in the message forwarded below) mentions October 2016; I assume
that's when they expect to end support for ICU 57.

So we need to decide how we're going to respond to this. Some options
for consideration:

(a) Adopt ICU 58 when released, and drop Gecko support for WinXP and OSX
10.6.

(b) Keep Gecko on ICU 57 and Unicode 8.0 until .... when? AFAIK, we have
not made any firm decisions regarding EOL for Firefox on these platforms.

(c) Keep Gecko on ICU 57 code, but update its data files to support
Unicode 9.0. This would take some effort on our side, though _probably_
not very much.

(d) Push back against the ICU proposal to drop these platforms, and see
if we can convince them to delay it. (No guarantees, though at least
they're asking. If we had a specific end date to propose, I'd guess that
might help our case.)

In the case of either (b) or (c), we'd also need to take responsibility
for handling any critical security issues that are discovered that
affect the no-longer-maintained version we'd be shipping (e.g. by
backporting fixes from the latest upstream version).


Thoughts?

JK


† Except on Android, where we maintain separate code to support some
features; others are simply missing.


-------- Forwarded Message --------
Subject: Re: [icu-design] [icu-support] Drop Windows XP and OSX 10.6
support
Date: Thu, 28 Apr 2016 08:55:55 -0700
From: Markus Scherer <marku...@gmail.com>
To: icu-design <icu-d...@lists.sourceforge.net>
CC: ICU support mailing list <icu-s...@lists.sourceforge.net>,
Jonathan Kew <jona...@jfkew.plus.com>



On Wed, Apr 27, 2016 at 4:30 PM, Steven Loomis <s...@icu-project.org
<mailto:s...@icu-project.org>> wrote:

Jonathan and other users,
Please comment on whether dropping Windows XP for ICU 58 will
cause significant problems.
We discussed this for 57 (as per below) but no code changes were
made.


For ICU 57, we were just thinking of removing some Windows XP-specific
synchronization code. We decided to just keep that for 57.

For ICU 58, we are looking at switching more code over to Windows
Vista/7/8 APIs because Windows XP and Windows Server 2003 only support
i18n APIs with LCID parameters and cannot support some languages at all.
Newer Windows versions added APIs that take language tag strings. This
is important for an i18n library on a major platform...

For how long do you plan to support Windows XP past October 2016? Could
you stay on ICU 57/CLDR 29/Unicode 8 until you stop supporting Windows XP?

Also, Windows Vista seems to have very low market share and seems to be
getting dropped by vendors around the same time they drop XP.
Is it ok to skip Vista and set Windows 7 as the new base for ICU 58?

Best regards,
markus

[Jeff Muizelaar]

Do we use any of the OS specific parts of ICU?

-Jeff

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Jonathan Kew]

On 28/4/16 18:11, Jeff Muizelaar wrote:
> Do we use any of the OS specific parts of ICU?

I don't know.

But even if we don't, I suspect that once they drop support for XP /
10.6, it won't be long before the project as a whole becomes
increasingly difficult to build for those targets, as it'll start
assuming support for compiler and/or runtime library features that
aren't readily available there.

JK

> <mailto:icu-s...@lists.sourceforge.net>>,
> Jonathan Kew <jona...@jfkew.plus.com <mailto:jona...@jfkew.plus.com>>

>
>
>
> On Wed, Apr 27, 2016 at 4:30 PM, Steven Loomis <s...@icu-project.org
> <mailto:s...@icu-project.org>

> <mailto:s...@icu-project.org <mailto:s...@icu-project.org>>> wrote:
>
> Jonathan and other users,
> Please comment on whether dropping Windows XP for ICU 58 will
> cause significant problems.
> We discussed this for 57 (as per below) but no code changes
> were made.
>
>
> For ICU 57, we were just thinking of removing some Windows XP-specific
> synchronization code. We decided to just keep that for 57.
>
> For ICU 58, we are looking at switching more code over to Windows
> Vista/7/8 APIs because Windows XP and Windows Server 2003 only support
> i18n APIs with LCID parameters and cannot support some languages at all.
> Newer Windows versions added APIs that take language tag strings. This
> is important for an i18n library on a major platform...
>
> For how long do you plan to support Windows XP past October 2016? Could
> you stay on ICU 57/CLDR 29/Unicode 8 until you stop supporting
> Windows XP?
>
> Also, Windows Vista seems to have very low market share and seems to be
> getting dropped by vendors around the same time they drop XP.
> Is it ok to skip Vista and set Windows 7 as the new base for ICU 58?
>
> Best regards,
> markus
>
>
> _______________________________________________
> dev-platform mailing list

> dev-pl...@lists.mozilla.org <mailto:dev-pl...@lists.mozilla.org>
> https://lists.mozilla.org/listinfo/dev-platform
>
>

[Jeff Muizelaar]

On Thu, Apr 28, 2016 at 1:39 PM, Jonathan Kew <jfkt...@gmail.com> wrote:

> On 28/4/16 18:11, Jeff Muizelaar wrote:
>
>> Do we use any of the OS specific parts of ICU?
>>
>
> I don't know.
>
> But even if we don't, I suspect that once they drop support for XP / 10.6,
> it won't be long before the project as a whole becomes increasingly
> difficult to build for those targets, as it'll start assuming support for
> compiler and/or runtime library features that aren't readily available
> there.
>

True, but ICU project but more willing to compromise on things like that
compared to the OS functionality mentioned below.

[Jeff Walden]

On 04/28/2016 10:00 AM, Jonathan Kew wrote:
> Thoughts?

Another option is to ship a WinXP-specific Firefox build that doesn't provide ICU and ECMAScript's Intl functionality.

This would break anyone's expectation that any version of Firefox past the mid-30s somewhere has Intl available in it. I don't know if anyone makes such expectations rather than feature-detecting.

Also this would make the story http://caniuse.com/ and similar sites try to tell have an asterisk by it, or something rather ugly. Given it's WinXP only (and Firefox for Android's recalcitrance ;-) may potentially be broken out into its own column by such sites), it seems to me that particular tradeoff is a bad one versus claiming full support everywhere.

But it's another option, and arguably the least work that lets us improve our internationalization support at the same rate ICU improves.

Jeff

[Ehsan Akhgari]

On 2016-04-29 1:39 AM, Jonathan Kew wrote:
> On 28/4/16 18:11, Jeff Muizelaar wrote:
>> Do we use any of the OS specific parts of ICU?
>
> I don't know.

What are those OS specific parts?

[Henri Sivonen]

On Thu, Apr 28, 2016 at 8:51 PM, Jeff Walden <jwald...@mit.edu> wrote:
> On 04/28/2016 10:00 AM, Jonathan Kew wrote:
>> Thoughts?
>
> Another option is to ship a WinXP-specific Firefox build that doesn't provide ICU and ECMAScript's Intl functionality.

I'm very opposed to this (unless the XP-specific thing isn't actually
XP-specific but 45 ESR delivered to XP over the normal "release" AUS
channel).

With ESR, we practically already committed to supporting XP about
three years past Microsoft stopping security patches for IE and XP
itself and the year past Google stopping security patches for Chrome
on XP. At a time when we should be focusing our resources on being
competitive on the platforms that will be relevant going forward, it
makes no sense for us to put even more resources into supporting XP.

Also, I think we should stop talking about ICU as if it was only the
back end for the ECMAScript Intl API. Given that we have ICU for that,
we should be actively getting rid of Netscape-era code for the C++
consumers of the same functionality that is exposed to JS via the
ECMAScript Intl API. That is, we should be get rid of non-ICU
collation and normalization, for example, by first making nsICollation
ICU-backed only and then flattening nsICollation callers to call ICU
collation directly. (Until such time that the Rust ecosystem grows a
pure-Rust library that can replace ICU as the back end of the
ECMAScript Intl API.)

> Given it's WinXP only (and Firefox for Android's recalcitrance ;-)

I think the situation where Firefox for Android is holding back
Gecko's ability to improve the codebase by getting rid of Netscape-era
code makes no sense, either. I think a Platform should put its foot
down and refuse letting Gecko development be handcuffed like this.
Taking the Android situation is a permit to introduce more build
configurations that omit ICU is entirely backwards.

So I think we should take option a': Drop XP and Snow Leopard support
on trunk and push ESR builds to the non-ESR update channel on XP and
Snow Leopard through the life of 45 ESR.

--
Henri Sivonen
hsiv...@hsivonen.fi
https://hsivonen.fi/

[sn...@snorp.net]

On Friday, April 29, 2016 at 2:44:25 AM UTC-5, Henri Sivonen wrote:
>
> > Given it's WinXP only (and Firefox for Android's recalcitrance ;-)
>
> I think the situation where Firefox for Android is holding back
> Gecko's ability to improve the codebase by getting rid of Netscape-era
> code makes no sense, either. I think a Platform should put its foot
> down and refuse letting Gecko development be handcuffed like this.
> Taking the Android situation is a permit to introduce more build
> configurations that omit ICU is entirely backwards.

The Fennec team has been very clear about why they oppose inclusion of ICU in bug 1215247. This type of attitude is why we have people in the Firefox org wanting to axe Gecko. The engineers in Platform consistently want to dismiss mobile-specific issues, and this is a great example. If you really want to get ICU into Fennec, find a way to do it without bloating the APK size instead of bullying the Fennec folks.

James

[Nicholas Nethercote]

It's natural that engineers in different parts of the organisation and
community will have different priorities.

Let's all take a deep breath and keep the discussion calm, please.

Nick

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org

> https://lists.mozilla.org/listinfo/dev-platform

[L. David Baron]

On Friday 2016-04-29 10:43 +0300, Henri Sivonen wrote:
> On Thu, Apr 28, 2016 at 8:51 PM, Jeff Walden <jwald...@mit.edu> wrote:

> > On 04/28/2016 10:00 AM, Jonathan Kew wrote:
> >> Thoughts?
> >
> > Another option is to ship a WinXP-specific Firefox build that doesn't provide ICU and ECMAScript's Intl functionality.
>

> I'm very opposed to this (unless the XP-specific thing isn't actually
> XP-specific but 45 ESR delivered to XP over the normal "release" AUS
> channel).
>
> With ESR, we practically already committed to supporting XP about
> three years past Microsoft stopping security patches for IE and XP
> itself and the year past Google stopping security patches for Chrome
> on XP. At a time when we should be focusing our resources on being
> competitive on the platforms that will be relevant going forward, it
> makes no sense for us to put even more resources into supporting XP.
>
> Also, I think we should stop talking about ICU as if it was only the
> back end for the ECMAScript Intl API. Given that we have ICU for that,
> we should be actively getting rid of Netscape-era code for the C++
> consumers of the same functionality that is exposed to JS via the
> ECMAScript Intl API. That is, we should be get rid of non-ICU
> collation and normalization, for example, by first making nsICollation
> ICU-backed only and then flattening nsICollation callers to call ICU
> collation directly. (Until such time that the Rust ecosystem grows a
> pure-Rust library that can replace ICU as the back end of the
> ECMAScript Intl API.)

I still find it sad that ECMAScript Intl came (as I understand it)
very close to just standardizing on a piece of software (ICU), and
also find it disturbing that we're going to extend that
standardization on a particular piece of software (possibly even
more rigidly) into other areas. I think many of the arguments we
made against standardizing on SQLite seem to apply to ICU as well,
such as security risk and needing to reverse-engineer when writing
future implementations of parts of the Web platform.

While I expect that some of the features that Intl provides (from
ICU data) are worthwhile in terms of codesize, I'm certainly not
confident that they all are. I have similar worries about other
large chunks of code that land in our tree...

And when I say worthwhile, I'm talking not just about whether the
feature is intrinsically valuable, but whether it's actually going
to be used by Web developers to get that value to users.

> > Given it's WinXP only (and Firefox for Android's recalcitrance ;-)
>

> I think the situation where Firefox for Android is holding back
> Gecko's ability to improve the codebase by getting rid of Netscape-era
> code makes no sense, either. I think a Platform should put its foot
> down and refuse letting Gecko development be handcuffed like this.
> Taking the Android situation is a permit to introduce more build
> configurations that omit ICU is entirely backwards.
>

> So I think we should take option a': Drop XP and Snow Leopard support
> on trunk and push ESR builds to the non-ESR update channel on XP and
> Snow Leopard through the life of 45 ESR.

I think enough of our users are on Windows XP that decisions about
dropping Windows XP are not a purely engineering decision. (Do we
still have more Windows XP users than we have on all non-Windows
platforms combined?) Pushing those users to ESR without buy-in from
all parts of the organization will likely lead to worse engineering
problems than having to support XP (e.g., having to support 45ESR
substantially longer).


How much value does ICU get from dropping Windows XP support? Can
we push back on their plans to do so, at least for the parts that we
use? (It also seems to be that we need to answer the question,
already raised in this thread, about whether the parts that are
expensive for them to support intersect at all with the parts that
we use.)

-David

--
𝄞 L. David Baron http://dbaron.org/ 𝄂
𝄢 Mozilla https://www.mozilla.org/ 𝄂
Before I built a wall I'd ask to know
What I was walling in or walling out,
And to whom I was like to give offense.
- Robert Frost, Mending Wall (1914)

[Henri Sivonen]

On Sat, Apr 30, 2016 at 11:26 PM, L. David Baron <dba...@dbaron.org> wrote:
> On Friday 2016-04-29 10:43 +0300, Henri Sivonen wrote:
> I still find it sad that ECMAScript Intl came (as I understand it)
> very close to just standardizing on a piece of software (ICU),

Looking at the standard, it seems intentionally vague about what data
sources are supported, and that's not good for a Web standard.
However, it seems to me that in practice there is no standardized
dependency on ICU but on the CLDR database maintained by Unicode.org.
In a C or C++ program, the easiest and least-NIH way to expose CLDR is
to use ICU like Google and Apple do and like we do on desktop. I'm not
sure what Microsoft does, considering that these days they are no
longer opposed to using open source software, but I believe that Edge
exposes CLDR via some non-ICU Microsoft-developed mechanism. So it
seems like there are two independent interoperable implementations as
far as code goes.

> and
> also find it disturbing that we're going to extend that
> standardization on a particular piece of software (possibly even
> more rigidly) into other areas.

As noted, these other areas are why I care about having ICU
unconditionally available on all platforms that Gecko runs on and why
I think it's harmful when ICU is blocked on one platform. Also, as
noted, I don't care about ICU per se, but I care about being able to
treat operations like Unicode normalization and locale-aware collation
as foundational operations whose availability or correctness is not
optional. I think it would be ideal if we had a library or set of
libraries written in Rust that provided this functionality, but until
such a library written in Rust shows up, ICU is the only option on the
table today that is (if bundled in Gecko in its latest version)
correct and cross-platform consistent.

I think it is harmful that we have to maintain abstractions for
foundational operations to support a configuration where the back end
isn't correct (to latest Unicode data) and cross-platform consistent.
Until Rust-based replacements show up, the most reasonable way to
perform operations that depend on Unicode.org data is to bundle ICU
and to call its APIs directly without abstraction layers in between.

Again, talking about ICU as just an enabler of the ECMAScript
Internationalization API is a bad framing for the issue, because it
makes it seem like blocking ICU "just" turns off one fairly recent Web
API. Yet, Gecko has needs for functionality exposed by ICU in various
places. For example:

* Hyphenation, spellchecking, layout, gfx and JavaScript parsing need
access to the character properties of the Unicode database. Currently,
we duplicate ICU functionality (in out-of-date manner I believe) to
implement these in libxul.

* Internationalized domain names and text shaping need Unicode
Normalization. Currently, we duplicate ICU functionality (in
out-of-date manner I believe) to implement these in libxul.

* IndexedDB, XPath, XUL, SQL storage and history search UI use
locale-sensitive sorting. Currently, we duplicate ICU functionality on
Android for these by calling into the thread-unsafe C standard library
API for this stuff. This is fundamentally broken, because the design
of the C standard library is fundamentally broken: In the C standard
library, there's no to ask for comparison according to a given locale.
Instead, we set the locale process-wide (all threads!), then ask the C
standard library to do a comparison, and then unset the locale
process-wide.

* Parts of the Firefox UI do locale-sensitive datetime formatting in
a way that calls to legacy platform APIs duplicating ICU function in a
manner that imports system-specific bugs.

* Based on open bugs, it seems we duplicate ICU functionality for
bidi, but it's not clear to me if we're already building that part of
ICU anyway and the relative correctness is unclear to me.

I think it's neither good use of developer time nor holistic
management of product size in bytes to have this duplication sprinkled
around. (Though I don't believe that getting rid of the above
duplication of ICU functionality would add up to the size of ICU
itself: We should expect ICU to be a net addition to APK size in any
case.)

It's worth noting that the above items split into on one hand the
Unicode character property database and associated algorithms
(normalization, bidi, line breaking, script run identification) and on
the other hand the CLDR database and associated algorithms
(locale-sensitive sorting, date formatting, number formatting, etc.).
We have more foundational dependency needs on the former than the
latter, but the discussion about ICU size as well as the ECMAScript
Internationalization API exposure is mainly about the latter.

Again, ideally, we'd have a an actively-maintained Rust library for
the Unicode character property database and associated algorithms and
another actively-maintained Rust library the CLDR database and
associated algorithms. But absent such libraries showing up, ICU is
what we have available and should use until then. That is, I think we
should systematically use a single source of Gecko-bundled
foundational functionality for the Unicode character property database
and associated algorithms and I think we should systematically use a
single source of Gecko-bundled foundational functionality for the CLDR
database and associated algorithms. (And we've already turned off a
bunch of stuff in ICU that doesn't appear on the above list.)

> I think many of the arguments we
> made against standardizing on SQLite seem to apply to ICU as well,
> such as security risk and needing to reverse-engineer when writing
> future implementations of parts of the Web platform.

Is there any actual evidence that non-ICU exposure of the CLDR
data-based operations wasn't interoperable with ICU exposure of the
CLDR data-based operations?

> While I expect that some of the features that Intl provides (from
> ICU data) are worthwhile in terms of codesize, I'm certainly not
> confident that they all are. I have similar worries about other
> large chunks of code that land in our tree...

What other chunks?

What bothers me the most regarding size of what we ship is

* Failure to make the most out of compression (i.e. Zopfli) before
objecting to the addition of new things stuff. I've brought this up
before, but just now, I downloaded the (en-US API level 15) APK for
Fennec 46 and ran ImageOptim (https://imageoptim.com/mac) on the PNG
files included directly in the APK (i.e. not the one hidden inside
omni.ja). ImageOptim says: "Saved 311KB out of 1.7MB. 28.6% per file
on average (up to 94.3%)." (There wasn't a single already-optimal PNG
there!) Additionally, the same exercise could be repeated for images
in omni.ja. Then all the XML and JS could be Zopflified. The bundled
.ttf files could be turned into Brotli-compressed WOFF2 files. All as
a matter of being smarter about how we use lossless compression on
stuff that we already losslessly compress without adding any novel
decompression capability to the product and without revisiting the
decision not to compress .so files. (Though maybe we should look into
building ICU as a separate .so and seeing how well Zopfli compresses
the CLDR data.)

* Failure to remove old mostly useless cruft like nsIEntityConverter
before blocking new useful stuff like ICU. (Not the same size, sure.)

* Landing JS polyfills for platform features that we implement
natively when checking in front end code that imports cross-browser JS
libraries. (I was particularly saddened to see Hello land a JS
polyfill for TextDecoder at the time when I was working on removing
dead weight from our Gecko-level encoding converters.)

> And when I say worthwhile, I'm talking not just about whether the
> feature is intrinsically valuable, but whether it's actually going
> to be used by Web developers to get that value to users.

Sadly, exposure of CLDR data-based operations to the Web (whether via
the ECMAScript Internationalization API or IndexedDB) is the sort of
this that is unlikely to deliver value until Web app developers feel
that it's OK to unconditionally rely on the browser providing that
functionality. In that sense, refusal to ship the feature on some
platform citing lack of use on the Web is exactly the sort of thing
that will significantly lower the probability that use on the Web will
take off thereby making the feature dead weight even on platforms
where it is shipped.

> I think enough of our users are on Windows XP that decisions about
> dropping Windows XP are not a purely engineering decision.

I don't recall it ever being explicitly articulated why having x
number or percentage of users on XP leads to the conclusion that XP
must remain supported. Having x users on XP is quite different from
having x users on a more current platform, because XP itself is no
longer getting security patches and because our competitors are no
longer delivering security patches for their browsers and XP.

If the concern is that it's wrong to stop delivering security patches
for x users, surely it can't be Mozilla's responsibility to
indefinitely continue to deliver browser security patches to users who
already tolerate running a system that doesn't get security patches
and for which other browsers have already earlier stop delivering
security patches. One might argue that it's wrong to give these users
the impression that it's OK to continue to stay on a system that
doesn't get security patches.

If the concern is that the users would switch to another browser,
making a dent in in Firefox's overall observed market share, as long
as the users stick to XP, what would they switch to? Security patches
for IE and Chrome have already stopped earlier and considering that XP
support was actually ripped out of Chromium, Opera and other Chromium
re-packagers probably won't be able to continue to support XP,
either).

If the concern is that users would switch to another browser when they
switch off XP having felt abandoned on XP, why should we expect that
to be the case considering that IE and Chrome abandoned them earlier?

As far as I can tell, the main effect that we should expect from
dropping XP or putting XP on ESR is the market share of the *latest*
Firefox dropping (as opposed to the market share of Firefox in general
dropping). Maybe that's the concern and maybe everyone is supposed to
know that that's the concern, but I think it would still help to
articulate what the concern is instead of just pointing to having lots
of users on XP.

> (Do we
> still have more Windows XP users than we have on all non-Windows
> platforms combined?) Pushing those users to ESR without buy-in from
> all parts of the organization will likely lead to worse engineering
> problems than having to support XP (e.g., having to support 45ESR
> substantially longer).

That's certainly a risk of pushing XP to ESR.

> (It also seems to be that we need to answer the question,
> already raised in this thread, about whether the parts that are
> expensive for them to support intersect at all with the parts that
> we use.)

Indeed, but on our side, we should remember tally every thread like
this as part of the cost of supporting XP on our side.

[Boris Zbarsky]

On 4/29/16 11:30 AM, sn...@snorp.net wrote:
> The Fennec team has been very clear about why they oppose inclusion of ICU in bug 1215247.

Sort of. There's been a fair amount of moving of goalposts to get from
https://bugzilla.mozilla.org/show_bug.cgi?id=1215247#c14 to
https://bugzilla.mozilla.org/show_bug.cgi?id=1215247#c43 as far as I can
tell.

I sympathize with the Fennec team's position here: The amount of code in
libxul keeps growing (not always by as little as possible, I agree!) as
we add support for more stuff the web is coming to depend on, but some
of the features being added are perhaps not a big deal in the markets
that want a small APK download. It's not clear to me who (if anyone)
knows what features these are; clearly the JS Intl API (yes, not the
only reason to include ICU) is one of them, but are there others we've
identified?

Of course https://bugzilla.mozilla.org/show_bug.cgi?id=1215247#c43 more
or less flat-out disagrees with the suggestion that we should have fewer
Gecko configurations, on a much broader front than ICU support...

I know we have places where we use more space than we should in Gecko,
and in particular some places where we have traded off space for speed
by having largish static data tables instead of more dynamic checks...
not to mention having static bindings code instead much smaller dynamic
XPConnect code. This tradeoff was very conscious, akin to Fennec's
decision to not compress .so, but may have been the wrong one for Fennec
in practice.

If we, as an organization, really want to try to reduce the size of the
Fennec APK, and are actually willing to put platform resources into it
(which requires either hiring accordingly or starving other goals, in
the usual way), then we should do that. So far I've unfortunately seen
precious little willingness to staff such an effort appropriately. :(

> This type of attitude is why we have people in the Firefox org wanting to axe Gecko.

For the Android case, I expect the only viable replacement that hits the
desired size limits would be an iOS-like solution, right? That is, a UI
using whatever browser engine is already installed on the device?

Just to be clear as to what our real alternatives are here.

> The engineers in Platform consistently want to dismiss mobile-specific issues

I think you're painting with a _very_ broad brush here.

-Boris

[Jim Blandy]

What are the distributions of memory and flash sizes for the devices people
currently run Fennec on? It'll be almost impossible to have a good
discussion about Fennec size without those numbers. I seem to remember that
is data we felt was okay to collect.

[Henri Sivonen]

On Mon, May 2, 2016 at 2:37 AM, Jim Blandy <jbl...@mozilla.com> wrote:
> What are the distributions of memory and flash sizes for the devices people
> currently run Fennec on? It'll be almost impossible to have a good
> discussion about Fennec size without those numbers. I seem to remember that
> is data we felt was okay to collect.

We should also be data-driven about understanding where the bytes go.
In particular, I think functionality-neutral size reductions should be
done before blocking new functionality from landing. In addition to
unoptimally compressed PNGs, there's unminified JS in Fennec (i.e. the
JS comments are shipped).

[Milan Sreckovic]

This. Dropping the XP support is *completely* not an engineering decision. It isn’t even a community decision. It is completely, 100% MoCo driven Firefox product management decision, as long as the numbers of users are where they are.

It is good to have these conversations, about potential pros of removing the support for XP, as they will inform us in the future. In the immediate term, it will be very frustrating for people to think that any technical information they provide would be enough to push us in the direction of dropping XP support.

—
- Milan

> On Apr 30, 2016, at 16:26 , L. David Baron <dba...@dbaron.org> wrote:

> I think enough of our users are on Windows XP that decisions about

> dropping Windows XP are not a purely engineering decision. (Do we

[Justin Dolske]

On 4/30/16 1:26 PM, L. David Baron wrote:

>> So I think we should take option a': Drop XP and Snow Leopard support
>> on trunk and push ESR builds to the non-ESR update channel on XP and
>> Snow Leopard through the life of 45 ESR.
>
> I think enough of our users are on Windows XP that decisions about
> dropping Windows XP are not a purely engineering decision. (Do we
> still have more Windows XP users than we have on all non-Windows
> platforms combined?) Pushing those users to ESR without buy-in from
> all parts of the organization will likely lead to worse engineering
> problems than having to support XP (e.g., having to support 45ESR
> substantially longer).

I know you and Henri know this, but because I've seen it come up in a
number of other places...

Moving XP users to ESR is not, by itself, a solution. All moving users
to ESR does is allow them to keep running a supported Firefox for a
little bit longer than immediately dropping it. (Equivalently, it allows
the dropping XP support in the code a bit earlier than the final EOL
date in Firefox.)

Once an EOL decision on XP has been reached, we may (or may not!) opt to
coordinate that with an ESR release. I'd note that we didn't do that
with the recent OSX 10.6-10.8 desupport, and ISTR having not done that
for some other platform in the past because it means maintaining
build&test infrastructure for an extended period.

In other words, these are details of what to do with XP users after a
product decision has been made to actually end support.

Justin

[Margaret Leibovic]

On Sun, May 1, 2016 at 6:54 AM, Henri Sivonen <hsiv...@hsivonen.fi> wrote:

>
> What bothers me the most regarding size of what we ship is
>
> * Failure to make the most out of compression (i.e. Zopfli) before
> objecting to the addition of new things stuff. I've brought this up
> before, but just now, I downloaded the (en-US API level 15) APK for
> Fennec 46 and ran ImageOptim (https://imageoptim.com/mac) on the PNG
> files included directly in the APK (i.e. not the one hidden inside
> omni.ja). ImageOptim says: "Saved 311KB out of 1.7MB. 28.6% per file
> on average (up to 94.3%)." (There wasn't a single already-optimal PNG
> there!) Additionally, the same exercise could be repeated for images
> in omni.ja.

We do optimize images before they land in the tree, although we usually use
pngcrush, and there may be some older assets that landed before we made
this common practice. However, Sebastian ran an analysis recently and found
that there's actually not much left to optimize (~35kb):
https://bugzilla.mozilla.org/show_bug.cgi?id=1266156

What you may actually be seeing is the fact that AAPT's optimization tool
may actually increase the size of optimized PNGs:
https://bugzilla.mozilla.org/show_bug.cgi?id=1266156

> Then all the XML and JS could be Zopflified. The bundled
> .ttf files could be turned into Brotli-compressed WOFF2 files.

We recently landed a change to make fonts downloadable by default, so these
aren't even included in our APK anymore:
https://bugzilla.mozilla.org/show_bug.cgi?id=1194338

We also have a GSoC student this summer who's going to work on making more
parts of the APK downloadable at runtime.

We landed a change a while back to minify JS, and we verified this morning
that all JS seems to be minified in components/chrome/toolkit:
https://bugzilla.mozilla.org/show_bug.cgi?id=1039902

I think the Firefox for Android APK size issue merits its own discussion,
outside the context of this ICU thread. I'd encourage anyone who's
interested in helping out to take a look at the meta bug where we've been
tracking our effort to reduce APK size:
https://bugzilla.mozilla.org/show_bug.cgi?id=942609

There are a lot of ideas in there, but we haven't had time to explore
fixing them all.

Margaret

[Jeff Walden]

On 04/29/2016 08:30 AM, sn...@snorp.net wrote:
> The engineers in Platform consistently want to dismiss mobile-specific issues, and this is a great example. If you really want to get ICU into Fennec, find a way to do it without bloating the APK size instead of bullying the Fennec folks.

Your point is taken, modulo quibbling (that some might characterize more strongly). Not all size questions should be/are viewed alike by Platform or even by individual engineers. For example, I heard about a recent few-hundred-K hit purely for quality of service, which seemed a bit much to me and you both. (Deliberately not saying whodunit.) But okay.

DOM features rarely die, so there's a limit to how much code we can remove. And web developers (and by extension end users visiting their sites) expect browsers to do increasingly many things, so we must continue adding code. Intl isn't special; it's just one new feature of many (albeit one with fundamentally irreducible complexity, human communication being beyond the control even of web standards bodies, let alone us). This story will never change with Gecko.

But we can't fully serve developers with the stagnating system-based engine. It'll be difficult to impossible to offer new layout/rendering features, new DOM features, hypothetical browser-integration improvements into existing features, new JS features that aren't standard library additions, or new JS standard library additions *that are optimally fast in JITs*. It'll be functionally impossible to affect the development of standards having particular (or perhaps sole) importance on mobile, using an uncontrollable rendering engine. Yet another browser shell does not promote "choice and innovation on the Internet", especially so on mobile where increasingly more browsing occurs now.

Gecko, like any big codebase, has flab. I can't estimate how much. But there's practically zero possibility it has flab matching the numbers suggested in <https://bugzilla.mozilla.org/show_bug.cgi?id=1215247#c14>, implicitly desiring a 50%+ size reduction, no matter how far Platform bends over backward, or how cooperative Platform engineers are.

Jeff

[Jeff Walden]

On 04/30/2016 01:26 PM, L. David Baron wrote:
> I still find it sad that ECMAScript Intl came (as I understand it)
> very close to just standardizing on a piece of software (ICU)

I'm fuzzy on the details as well, but I don't believe it was ever going to be the case that the spec would be "do what ICU does". Language *had* to be hand-wavy, because what is the custom in one language now, will not always be the custom. So some flexibility must be granted to accommodate this, as well as to not (in effect) specify entire languages as they exist right now.

Now, in *practice* it might work out that behaviors would be what ICU does. But even there I'm not so sure about it. What ICU does, as I understand it, is approximately only what CLDR (all the language/locale-specific information about formatting, pluralization, word-breaking, etc.), plus standardized things like IETF BCP 47, tells it to. And CLDR's purview is not really areas where technical debate/disagreement/dissent makes a whole lot of sense, because it's just setting down what the rest of the world does.

In any event, the "doomsday" scenario did not occur, so we're safe.

> and also find it disturbing that we're going to extend that
> standardization on a particular piece of software (possibly even
> more rigidly) into other areas.

Using a library to do certain things we do other ways right now, in sometimes inferior fashion, doesn't seem inherently objectionable to me. So long as the library's internal decisions don't bleed too far into the visible API, which I don't believe they did here.

> I think many of the arguments we
> made against standardizing on SQLite seem to apply to ICU as well,
> such as security risk and needing to reverse-engineer when writing
> future implementations of parts of the Web platform.

I believe this should be mitigated by its being CLDR that will be the true dependency, and CLDR not containing anything but the most highly constrained of "algorithms" (for things like pluralization). But I'm happy to be corrected/enlightened by someone who understands ICU/CLDR better.

> I think enough of our users are on Windows XP that decisions about
> dropping Windows XP are not a purely engineering decision.

Unfortunately I agree.

Jeff

[Anne van Kesteren]

On Tue, May 3, 2016 at 2:17 AM, Jeff Walden <jwald...@mit.edu> wrote:
> Using a library to do certain things we do other ways right now, in sometimes inferior fashion, doesn't seem inherently objectionable to me. So long as the library's internal decisions don't bleed too far into the visible API, which I don't believe they did here.

Note that the ECMA-402 folks have hinted that if Microsoft adopted ICU
it would make it much easier to expose a bunch of new stuff and have
been pushing Microsoft in that direction. Were that to happen you have
software monoculture, bugs that cannot be fixed, etc. It's especially
strange that a standards body is encouraging this kind of thing.


--
https://annevankesteren.nl/

[Jason Orendorff]

It's not just strange. It's against Ecma's explicit organization-wide
policy.

-j

On Tue, May 3, 2016 at 1:13 AM, Anne van Kesteren <ann...@annevk.nl> wrote:

> On Tue, May 3, 2016 at 2:17 AM, Jeff Walden <jwald...@mit.edu> wrote:

> > Using a library to do certain things we do other ways right now, in
> sometimes inferior fashion, doesn't seem inherently objectionable to me. So
> long as the library's internal decisions don't bleed too far into the
> visible API, which I don't believe they did here.
>

> Note that the ECMA-402 folks have hinted that if Microsoft adopted ICU
> it would make it much easier to expose a bunch of new stuff and have
> been pushing Microsoft in that direction. Were that to happen you have
> software monoculture, bugs that cannot be fixed, etc. It's especially
> strange that a standards body is encouraging this kind of thing.
>
>
> --
> https://annevankesteren.nl/

[Henri Sivonen]

On Mon, May 2, 2016 at 10:49 PM, Margaret Leibovic
<mlei...@mozilla.com> wrote:
> On Sun, May 1, 2016 at 6:54 AM, Henri Sivonen <hsiv...@hsivonen.fi> wrote:
>>
>>
>> What bothers me the most regarding size of what we ship is
>>
>> * Failure to make the most out of compression (i.e. Zopfli) before
>> objecting to the addition of new things stuff. I've brought this up
>> before, but just now, I downloaded the (en-US API level 15) APK for
>> Fennec 46 and ran ImageOptim (https://imageoptim.com/mac) on the PNG
>> files included directly in the APK (i.e. not the one hidden inside
>> omni.ja). ImageOptim says: "Saved 311KB out of 1.7MB. 28.6% per file
>> on average (up to 94.3%)." (There wasn't a single already-optimal PNG
>> there!) Additionally, the same exercise could be repeated for images
>> in omni.ja.
>
>
> We do optimize images before they land in the tree, although we usually use
> pngcrush, and there may be some older assets that landed before we made this
> common practice. However, Sebastian ran an analysis recently and found that
> there's actually not much left to optimize (~35kb):
> https://bugzilla.mozilla.org/show_bug.cgi?id=1266156
>
> What you may actually be seeing is the fact that AAPT's optimization tool
> may actually increase the size of optimized PNGs:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1266156

I'm not familiar with how AAPT operates, but pngcrush alone is not the
state of the art.

I decompressed the APK and the decompressed omni.ja for Fennec
multilocale nightly for May 1 and ran ImageOptim
(https://imageoptim.com/mac) on the result. ImageOptim found 2 MB of
bitmaps and was able to make them 532.9 KB smaller.

Spot-checking the result indicates, that the optimization results were
obtained by combining pngcrush with Zopfli or AdvPNG (that is,
pngcrush+Zopfli vs. pngcrush+AdvPNG being better varies from image to
image).

It would be good to adopt a rule that in-product bitmaps must be
processed with ImageOptim until a better tool comes along. The problem
with ImageOptim is that it's Mac-only (unless you go through the
trouble of extracting the cross-platform tools that it puts a GUI
over), but I'd expect folks who create bitmap assets to have access to
Macs. :-)

>> Then all the XML and JS could be Zopflified. The bundled
>> .ttf files could be turned into Brotli-compressed WOFF2 files.
>
>
> We recently landed a change to make fonts downloadable by default, so these
> aren't even included in our APK anymore:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1194338
>
> We also have a GSoC student this summer who's going to work on making more
> parts of the APK downloadable at runtime.

Awesome. I should have immediately looked at the nightly builds
instead of the latest release.

(Looks like Zopfli was investigated in bug
https://bugzilla.mozilla.org/show_bug.cgi?id=1173894 .)

> On Mon, May 2, 2016 at 3:28 AM, Henri Sivonen <hsiv...@hsivonen.fi> wrote:
>>
>> On Mon, May 2, 2016 at 2:37 AM, Jim Blandy <jbl...@mozilla.com> wrote:
>> > What are the distributions of memory and flash sizes for the devices
>> > people
>> > currently run Fennec on? It'll be almost impossible to have a good
>> > discussion about Fennec size without those numbers. I seem to remember
>> > that
>> > is data we felt was okay to collect.
>>
>> We should also be data-driven about understanding where the bytes go.
>> In particular, I think functionality-neutral size reductions should be
>> done before blocking new functionality from landing. In addition to
>> unoptimally compressed PNGs, there's unminified JS in Fennec (i.e. the
>> JS comments are shipped).
>
>
> We landed a change a while back to minify JS, and we verified this morning
> that all JS seems to be minified in components/chrome/toolkit:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1039902

There's still unminified JS in the May 1 nightly. By spot-checking, JS
under modules/ in omni.ja seems unminified.

[Mike Hommey]

On Tue, May 03, 2016 at 03:14:35PM +0300, Henri Sivonen wrote:
> (Looks like Zopfli was investigated in bug
> https://bugzilla.mozilla.org/show_bug.cgi?id=1173894 .)

And bug 1234008, where you'll learn that zopflipng removes PNG chunks
that are important to us, including frames in animated PNGs.

Mike

[zbran...@mozilla.com]

Hi David,

I'm one of the editors of ECMA 402 and a champion of multiple proposals there. I'd like to respond to your comment:

On Saturday, April 30, 2016 at 1:26:53 PM UTC-7, David Baron wrote:
> I still find it sad that ECMAScript Intl came (as I understand it)
> very close to just standardizing on a piece of software (ICU), and
> also find it disturbing that we're going to extend that
> standardization on a particular piece of software (possibly even
> more rigidly) into other areas. I think many of the arguments we
> made against standardizing on SQLite seem to apply to ICU as well,
> such as security risk and needing to reverse-engineer when writing
> future implementations of parts of the Web platform.

I disagree with this statement. While we are definitely looking into ICU APIs as one of the prior knowledge cases, we don't necessarily follow ICU. We design APIs based on what we see people request the most.
In some cases, we align our API with ICU because we believe ICU got it right (DateTimeFormat), in others we go with our own API (PluralRules, UnitFormat, DurationFormat) and in yet others, we standardize something that ICU wants to pick from us (NumberFormat.formatToParts).

What's important here, is that we deliberately write the spec to not depend on ICU and our reference implementation (Intl.js) is pure JS with no dependency on ICU so we are fairly certain that you don't need ICU to implement JS Intl API.

What we do try to standardize around is CLDR as a kind of "wikipedia for i18n data". CLDR is just a database and having all major companies contribute to it makes it very powerful in giving us access to all the data we may need for internal and JS Intl API needs.

>
> While I expect that some of the features that Intl provides (from
> ICU data) are worthwhile in terms of codesize, I'm certainly not
> confident that they all are. I have similar worries about other
> large chunks of code that land in our tree...
>
> And when I say worthwhile, I'm talking not just about whether the
> feature is intrinsically valuable, but whether it's actually going
> to be used by Web developers to get that value to users.

We create APIs based on user needs. The way we determine what should stay in user land and what is worth standardizing is of course subjective, but we try to aim for lowering the barrier to write good multi-lingual applications so, obviously, we prioritize what's commonly used.

> How much value does ICU get from dropping Windows XP support? Can
> we push back on their plans to do so, at least for the parts that we
> use? (It also seems to be that we need to answer the question,
> already raised in this thread, about whether the parts that are
> expensive for them to support intersect at all with the parts that
> we use.)

Unfortunately, it seems that ICU decided to drop Win XP support in ICU 58. Maybe we can provide them strong reasons for not doing that?

We're currently starting an effort to deploy new L10n/I18n infrastructure for Firefox. While working on some of our most common needs (PluralRules, RelativeTimeFormat, UnitFormat), we reported bugs in CLDR and they are being fixed in time for CLDR 30.
So while we may not need to update ICU for a while and could potentially get stuck on ICU 57 (I don't have enough knowledge to understand what might be the cost of that), I'd like to make sure we can move forward with updating CLDR in Gecko.

Thanks,
zb.

[Philip Chee]

On 03/05/2016 08:17, Jeff Walden wrote:
> On 04/30/2016 01:26 PM, L. David Baron wrote:
>> I still find it sad that ECMAScript Intl came (as I understand it)
>> very close to just standardizing on a piece of software (ICU)
>
> I'm fuzzy on the details as well, but I don't believe it was ever
> going to be the case that the spec would be "do what ICU does".
> Language *had* to be hand-wavy, because what is the custom in one
> language now, will not always be the custom. So some flexibility must
> be granted to accommodate this, as well as to not (in effect) specify
> entire languages as they exist right now.
>
> Now, in *practice* it might work out that behaviors would be what ICU
> does. But even there I'm not so sure about it. What ICU does, as I
> understand it, is approximately only what CLDR (all the
> language/locale-specific information about formatting, pluralization,
> word-breaking, etc.), plus standardized things like IETF BCP 47,
> tells it to. And CLDR's purview is not really areas where technical
> debate/disagreement/dissent makes a whole lot of sense, because it's
> just setting down what the rest of the world does.

Given the "two different implementations rule" is there any suitable
alternative to ICU? I mean besides rolling our own.

Phil

--
Philip Chee <phi...@aleytys.pc.my>, <phili...@gmail.com>
http://flashblock.mozdev.org/ http://xsidebar.mozdev.org
Guard us from the she-wolf and the wolf, and guard us from the thief,
oh Night, and so be good for us to pass.

[Jeff Walden]

On 05/14/2016 06:58 AM, Philip Chee wrote:
> Given the "two different implementations rule" is there any suitable
> alternative to ICU? I mean besides rolling our own.

No, or at least not something cross-platform. It's probably possible to do something using Windows APIs, that would only be shippable there. (And maybe only on particular versions of Windows, with a decent chance XP's not in that set.)

Jeff