Windows XP and Vista Long Term Support Plan

[Peter Dolanjski]

Hello All,

tl;dr: Tentative plan is to move Windows XP/Vista users to ESR 52.  Feel free to comment/discuss.

In this email, I will lay out a tentative long term plan for supporting Windows XP and Vista users.  The point of this email is to solicit feedback about the plan from a broader audience.  I am purposefully keeping the email somewhat high level, but if you have specific questions about the details, I'm sure various folks on the engineering/release engineering teams can help answer them.

If you have relevant information to include in the bug you can find it here: Move Windows XP and Vista to Firefox ESR 52. 
I realize the discussion on this topic could get lively (which is completely understandable) - I just ask that you keep in mind good intent.
We hope to finalize plans over the next few weeks.

Goal
Continue to provide an excellent browsing experience and ongoing security updates to Windows XP and Vista users (inclusive of current Firefox users and users on browsers which are no longer supported who want to switch).  At the same time, reduce the opportunity cost of supporting XP and Vista (15 and 9 years since release, respectively).

Tentative Plan

• Move Windows XP and Vista users to an Extended Support Release when v52 branches
• Continue to provide security updates for the life of ESR 52 (well into 2018)
• Monitor user numbers as ESR 52 reaches end of life (2018), continue security patches for XP/Vista users on ESR 52 branch if user numbers justify it (exact threshold TBD)

Background
Firefox is the only major browser which still provides new releases to Windows XP and Vista users.  Currently these two audiences represent slightly more than 10% of our user base (XP being about 5x larger than Vista).  By the end of 2017 the number is projected to decrease to about half of that amount.
Microsoft ceased security updates for XP a few years ago and plans to do the same for Vista in early 2017.  With users no longer getting operating system updates, it becomes increasingly difficult for those users to stay safe online.  Further, continuing to include XP and Vista for new Firefox releases comes at an opportunity cost (more on this below).  We are reaching the point where we feel it is better to direct our primary efforts to where the majority of users are (Windows 7+).

All of that said, we don't feel right about the prospect of leaving users, who in many instances may not have the option to purchase a new computer to update their OS, completely vulnerable due to lack of security updates.  That's why we feel that continuing to provide updates via ESR 52 (and possible extension afterwards) allows us to reduce the opportunity cost of supporting these OS versions, while keeping users as safe as we can.

Opportunity Cost of Supporting Windows XP/Vista

Continuing to release on Windows XP/Vista comes at an opportunity cost and is becoming increasingly difficult due to our inability to source older hardware.  By removing XP/Vista from new Firefox releases we can:

• Lower XP/Vista specific testing/operational costs
• Move to modern compilers
• Simplify the codebase by removing XP/Vista-specific code from mainline
• Avoid XP/Vista specific bugs/regressions
  

What does this mean for XP/Vista users?

• Windows XP and Vista users will continue to get the latest Firefox versions up to and including v52
• v52 will be the last version of Firefox available to XP/Vista users
• XP/Vista users will continue to get security updates into 2018, with the potential for the timeframe to be extended

How will this plan be communicated?

Subsequent to the collection of feedback from this email, I will be drafting a blog post describing the finalized plan.  In addition, we'll be using some form of in-product messaging to inform users.

If you have questions/comments, please reply to the firefox-dev mailing list.

Thanks,

Peter Dolanjski

Product Manager, Firefox

Mozilla

[Mike Hommey]

On Wed, Oct 12, 2016 at 03:16:08PM -0400, Peter Dolanjski wrote:
> By removing XP/Vista from new Firefox releases we can:
>

(...)
> - Move to modern compilers

This one is a non-argument: supporting XP/Vista isn't preventing us from
using the latest Visual Studio. It only prevents us from using some newer
win32 APIs.

Mike
_______________________________________________
firefox-dev mailing list
firef...@mozilla.org
https://mail.mozilla.org/listinfo/firefox-dev

[L. David Baron]

On Wednesday 2016-10-12 15:16 -0400, Peter Dolanjski wrote:
> - Monitor user numbers as ESR 52 reaches end of life (2018), continue

> security patches for XP/Vista users on ESR 52 branch if user numbers
> justify it (exact threshold TBD)

This sounds like the potential to be a commitment to support the
ESR52 branch for an indefinite amount of time.

As that commitment lengthens, it has the potential to be more of a
burden than continuing support for XP/Vista would have been (for
example, if we have security bugs that require major architectural
changes to fix). The cost of backporting security fixes increases
with the age of the branch (time), probably as a worse-than-linear
function of time.

Is there a way we can avoid this potential indefinite commitment?

-David

--
𝄞 L. David Baron http://dbaron.org/ 𝄂
𝄢 Mozilla https://www.mozilla.org/ 𝄂
Before I built a wall I'd ask to know
What I was walling in or walling out,
And to whom I was like to give offense.
- Robert Frost, Mending Wall (1914)

[Peter Dolanjski]

>    - Monitor user numbers as ESR 52 reaches end of life (2018), continue
>    security patches for XP/Vista users on ESR 52 branch if user numbers
>    justify it (exact threshold TBD)

This sounds like the potential to be a commitment to support the
ESR52 branch for an indefinite amount of time.

As that commitment lengthens, it has the potential to be more of a
burden than continuing support for XP/Vista would have been (for
example, if we have security bugs that require major architectural
changes to fix).  The cost of backporting security fixes increases
with the age of the branch (time), probably as a worse-than-linear
function of time.

Is there a way we can avoid this potential indefinite commitment?

What was discussed with release engineering was to take things on a case by case basis.  In reality this shouldn't be an indefinite time commitment.  By early 2018 user numbers will likely fall below the threshold where it becomes harder to justify continued investment. If a particular backporting issue is significant in nature, that may force a line in the sand to be drawn.  That's sort of a risk that we'll have to accept with this approach.

Peter

[Aaron Klotz]

On 10/12/2016 5:09 PM, Mike Hommey wrote:
>
> This one is a non-argument: supporting XP/Vista isn't preventing us from
> using the latest Visual Studio. It only prevents us from using some newer
> win32 APIs.
>
>

It does prevent us from using C++11 thread-safe "magic" statics in
VS2015 as they are not properly supported on Windows XP [1].

[1]
https://connect.microsoft.com/VisualStudio/feedback/details/1789709/visual-c-2015-runtime-broken-on-windows-server-2003-c-11-magic-statics

[Mike Hommey]

On Wed, Oct 12, 2016 at 08:23:12PM -0600, Aaron Klotz wrote:
> On 10/12/2016 5:09 PM, Mike Hommey wrote:
> >
> > This one is a non-argument: supporting XP/Vista isn't preventing us from
> > using the latest Visual Studio. It only prevents us from using some newer
> > win32 APIs.
> >
> >
>
> It does prevent us from using C++11 thread-safe "magic" statics in VS2015 as
> they are not properly supported on Windows XP [1].

That is related to DllMain, which is arguably a win32 detail.

Mike

[Gervase Markham]

On 13/10/16 00:17, L. David Baron wrote:
> This sounds like the potential to be a commitment to support the
> ESR52 branch for an indefinite amount of time.

No; it just means that the fate of XP/Vista is tied to how long we wish
to maintain ESR52. We can stop maintaining it any time we want, it's
just that with that, we will stop supporting those OSes.

We should put in place a plan to change the home page for XP/Vista
users, once support expires, to "this operating system is no longer
supported, has known security holes and is dangerous to use on the web
with any browser, not just Firefox. Please consider upgrading your
operating system or replacing your computer."

Gerv

[Peter Dolanjski]

We should put in place a plan to change the home page for XP/Vista
users, once support expires, to "this operating system is no longer
supported, has known security holes and is dangerous to use on the web
with any browser, not just Firefox. Please consider upgrading your
operating system or replacing your computer."

+1.  It's worth noting that Chrome put up a persistent banner notification with a similar message when they end of life'd. 
We actually ran a survey (N=819) of Chrome users on XP to see what their response was to Google's message.
About half of the sample planned to continue using Chrome on XP without support/updates.

It is also worth noting that about 14% of the sample said they planned to switch browsers.  We poured over our downloads and metrics data for XP users to see if there was any evidence of users migrating over to Firefox.  There is no evidence that this is happening.

We should obviously do our best to explain things to the user, but we definitely need to keep in mind that the average user will likely not take any action.

Peter

[Gervase Markham]

On 13/10/16 16:28, Peter Dolanjski wrote:
> We should obviously do our best to explain things to the user, but we
> definitely need to keep in mind that the average user will likely not
> take any action.

I think the principle from Ezekiel 33 applies very well here:

The word of the Lord came to [Ezekiel]: "Son of man, speak to your
people and tell them: Suppose I bring the sword against a land, and the
people of that land select a man from among them, appointing him as
their watchman, and he sees the sword coming against the land and blows
his trumpet to warn the people. Then, if anyone hears the sound of the
trumpet but ignores the warning, and the sword comes and takes him away,
his blood will be on his own head. Since he heard the sound of the
trumpet but ignored the warning, his blood is on his own hands. If he
had taken warning, he would have saved his life. However, if the
watchman sees the sword coming but doesn’t blow the trumpet, so that the
people aren’t warned, and the sword comes and takes away their lives,
then they have been taken away because of their iniquity, but I will
hold the watchman accountable for their blood."
https://www.biblegateway.com/passage/?search=ezekiel+33&version=HCSB

We are responsible for warning; users are responsible for taking action.

Gerv

[Chris Hutten-Czapski]

(( There may actually be weak evidence in support of Windows XP users coming to Firefox in recent months, but that almost doesn't matter. ))

Could we encourage the upgrade of users? We can tell, say, via an addon, whether the user's computer is completely unable to run anything beyond Windows XP. Can we offer to "just" ship them a new computer with Firefox installed? Partner with Linux user groups, computer access charities... we have a lot of friends, can we get enough of them together to make this positive difference on the Web?

Chris

[Mike Hoye]

On 2016-10-13 11:14 AM, Gervase Markham wrote:
> We should put in place a plan to change the home page for XP/Vista
> users, once support expires, to "this operating system is no longer
> supported, has known security holes and is dangerous to use on the web
> with any browser, not just Firefox. Please consider upgrading your
> operating system or replacing your computer."

I'm pretty confident that this is not actionable information. At this
point it's unlikely that anyone still on WinXP will be able to choose
either of those options, either because of their technical facility or
their budget or both.

If we go this route, I'd much prefer we say something like "Thank you
for using Firefox, this will be the last release for Windows XP/Vista,
we're continuing to build Firefox for Windows 7 and 10. Here's how you
can get Firefox for your phone, too, and it's super-easy to move your
bookmarks over to it like so."

- mhoye

[Gervase Markham]

On 13/10/16 16:41, Mike Hoye wrote:
> I'm pretty confident that this is not actionable information. At this
> point it's unlikely that anyone still on WinXP will be able to choose
> either of those options,

Upgrading to a newer version of Windows, perhaps not. But it's not the
only OS out there.

> either because of their technical facility or
> their budget or both.

Insecure devices are a pox on the Internet, as Brian Krebs is finding
out. One reason they are so prevalent is that people don't know they are
doing the internet equivalent of farting in a lift (elevator). This
won't change until people understand that a bit.

Gerv

[Mike Hoye]

On 2016-10-13 11:46 AM, Gervase Markham wrote:
>
> Insecure devices are a pox on the Internet, as Brian Krebs is finding
> out. One reason they are so prevalent is that people don't know they are
> doing the internet equivalent of farting in a lift (elevator). This
> won't change until people understand that a bit.

This isn't a fair burden to put on the user, even if they knew what
steps to take next.

Haha, I _wanted_ to talk about this in terms of a home user's
"capabilities gap" - what the military calls the gulf between having a
plan and having the resources you need to execute on that plan, but when
I went to look up a good example here:

https://acc.dau.mil/CommunityBrowser.aspx?id=204085

it turns out I can't show it to you, because that site has an invalid
security certificate. So, yeah, if the largest and best-funded security
apparatus in the world can't reliably get basic, table-stakes-infosec
stuff like certs right then nontechnical users who rely on a 15-year-old
computer they either can't afford or don't know how to upgrade have no
shot. None.

I say we thank them for their support, tell them they can get Firefox on
newer devices (including phones) ... maybe point them to a decent
antivirus if they don't already have one? I just don't think trying to
frighten people into doing something they likely can't do will help.



- mhoye

[Mike Kaply]

The fact that Firefox 52 is the last version that supports plugins might be another reason to consider supporting it longer than a normal ESR.

I know there are many companies/organizations that are still struggling to get rid of legacy plugin needs.

Mike

[Mike Kaply]

On Thu, Oct 13, 2016 at 11:07 AM, Mike Hoye <mh...@mozilla.com> wrote:

Haha, I _wanted_ to talk about this in terms of a home user's "capabilities gap" - what the military calls the gulf between having a plan and having the resources you need to execute on that plan, but when I went to look up a good example here:

https://acc.dau.mil/CommunityBrowser.aspx?id=204085

it turns out I can't show it to you, because that site has an invalid security certificate. So, yeah, if the largest and best-funded security apparatus in the world can't reliably get basic, table-stakes-infosec stuff like certs right then nontechnical users who rely on a 15-year-old computer they either can't afford or don't know how to upgrade have no shot. None.

To be fair, it works in every other browser. We're the only browser that doesn't download intermediate certificate chains.

https://bugzilla.mozilla.org/show_bug.cgi?id=399324

Mike Kaply

[Mike Hoye]

On 2016-10-13 11:35 AM, Mike Kaply wrote:
> The fact that Firefox 52 is the last version that supports plugins
> might be another reason to consider supporting it longer than a normal
> ESR.

Per https://mzl.la/2dMxTTH about four percent of our XP users use Sync.
I'm not sure what that says other than "I like data-driven decision
making, this is data, therefore something", but there it is.

- mhoye

[Gervase Markham]

On 13/10/16 17:13, Mike Kaply wrote:
> To be fair, it works in every other browser. We're the only browser that
> doesn't download intermediate certificate chains.
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=399324

Actually, no, AFAICS it doesn't work because we (the Mozilla root store)
don't trust the US Federal PKI. It also doesn't work in Opera on Windows
or Chromium on Linux. It works in IE, presumably because Microsoft's
store does trust the Federal PKI. So it would work in Chrome on Windows,
but not elsewhere. No idea about MacOS.

Gerv

[Peter Dolanjski]

We examined XP users across a bunch of metrics including their engagement ratio, their continued activity rate (probability that a user is active in last 7 day given that they were active in the 21 days prior to that) and their search rate.

Across the board, they use Firefox less frequently and search less often, but we're talking about a ~10% difference relative to the broader Firefox population, not an order of magnitude.

Peter

[Justin Dolske]

I think the concern is that it's "indefinite" in the sense that the plan is vague about when support for ESR52 would end. And so the risk is that different people have different understanding of how long that's likely to continue, and that we find ourselves painted into a corner with needing to support ESR52 far longer than anyone thought we would. (As an extreme example, what would happen if XP usage stopped declining?)

I'd suggest placing bounds on it, so we can all be on the same page and comfortable with what's being committed to. E.G. "We project that at current rates, XP usage is expected to drop to X% by the normal end of ESR52. If, at the normal end of ESR52, usage remains above X+e%, we will further extend the life of ESR52 for an additional Z months. After that Z months, ESR52 will be discontinued irrespective of usage."

Justin

[Chris Peterson]

On 10/13/2016 10:52 AM, Justin Dolske wrote:
> I think the concern is that it's "indefinite" in the sense that the
> plan is vague about when support for ESR52 would end. And so the risk
> is that different people have different understanding of how long
> that's likely to continue, and that we find ourselves painted into a
> corner with needing to support ESR52 far longer than anyone thought we
> would. (As an extreme example, what would happen if XP usage stopped
> declining?)

As for the increasing difficulty of backporting security fixes, how many
bugs are fixed in a typical ESR dot release? Does the number of fixes
per dot release go down over the ESR support cycle?

[Gervase Markham]

On 13/10/16 18:52, Justin Dolske wrote:
> I'd suggest placing bounds on it, so we can all be on the same page and
> comfortable with what's being committed to. E.G. "We project that at
> current rates, XP usage is expected to drop to X% by the normal end of
> ESR52. If, at the normal end of ESR52, usage remains above X+e%, we will
> further extend the life of ESR52 for an additional Z months. After that
> Z months, ESR52 will be discontinued irrespective of usage."

But any policy we put in place now is subject to "events". Why can't we
just assess the continued effort vs. the number of users when the actual
time comes, and make a call based on available resources and opportunity
cost?

It's not like we can accidentally keep supporting it for too long...

[L. David Baron]

On Thursday 2016-10-13 20:26 +0100, Gervase Markham wrote:
> On 13/10/16 18:52, Justin Dolske wrote:
> > I'd suggest placing bounds on it, so we can all be on the same page and
> > comfortable with what's being committed to. E.G. "We project that at
> > current rates, XP usage is expected to drop to X% by the normal end of
> > ESR52. If, at the normal end of ESR52, usage remains above X+e%, we will
> > further extend the life of ESR52 for an additional Z months. After that
> > Z months, ESR52 will be discontinued irrespective of usage."
>
> But any policy we put in place now is subject to "events". Why can't we
> just assess the continued effort vs. the number of users when the actual
> time comes, and make a call based on available resources and opportunity
> cost?
>
> It's not like we can accidentally keep supporting it for too long...

My argument is that it may well be less effort to continue
supporting XP and Vista on mozilla-central for another two years
(followed by ESR for a bit) than to support ESR52 for three years.

So if we're actually talking about possibly maintaining ESR52 for
three years, I think we'd be better off not doing this, and
continuing to support XP and Vista in our non-ESR releases.

If we're willing to make a firmer commitment to ending support, I
think the plan becomes more reasonable.

[Aaron Klotz]

On 10/13/2016 1:37 PM, L. David Baron wrote:
> So if we're actually talking about possibly maintaining ESR52 for
> three years, I think we'd be better off not doing this, and
> continuing to support XP and Vista in our non-ESR releases.
>
> If we're willing to make a firmer commitment to ending support, I
> think the plan becomes more reasonable.
>

Then we need to make that commitment. Windows XP is no longer a minor
inconvenience such that we can just hack around its missing APIs via
dynamic linking. It is different enough from modern Windows that XP has
now become an impediment to shipping new features such as a11y on e10s
and the newest sandboxing mitigations. Continuing support for XP on
non-ESR is an option that completely opposes the objective of building
core strength.

[Chris Peterson]

On 10/13/2016 12:37 PM, L. David Baron wrote:
> My argument is that it may well be less effort to continue
> supporting XP and Vista on mozilla-central for another two years
> (followed by ESR for a bit) than to support ESR52 for three years.
>
> So if we're actually talking about possibly maintaining ESR52 for
> three years, I think we'd be better off not doing this, and
> continuing to support XP and Vista in our non-ESR releases.

Another option is to continue supporting XP/Vista in mozilla-central,
but treat XP/Vista as its own platform. We can create separate XP/Vista
and Win 7+ builds that #ifdef features we don't want to support on
XP/Vista at compile-time. This would open the option to use different
compilers or compiler options for Win7+ builds.

[Peter Dolanjski]

On 13/10/16 18:52, Justin Dolske wrote:
> I'd suggest placing bounds on it, so we can all be on the same page and
> comfortable with what's being committed to. E.G. "We project that at
> current rates, XP usage is expected to drop to X% by the normal end of
> ESR52. If, at the normal end of ESR52, usage remains above X+e%, we will
> further extend the life of ESR52 for an additional Z months. After that
> Z months, ESR52 will be discontinued irrespective of usage."

We can do that if it is useful, beyond a commitment for a final end of life date.

On 10/13/2016 1:37 PM, L. David Baron wrote:

So if we're actually talking about possibly maintaining ESR52 for
three years, I think we'd be better off not doing this, and
continuing to support XP and Vista in our non-ESR releases.

If we're willing to make a firmer commitment to ending support, I
think the plan becomes more reasonable.

Then we need to make that commitment. 

Based on an upper and lower bound projection, the end of ESR52 (likely April-ish 2018) shows that WinXP should be between 2.5-4.2% our Firefox Average Monthly ADIs.  By January 1st, 2019, the range is 0.8-2.6% of monthly ADIs (though that's trending pretty far out). 

I think we can say with some certainty that the worst case scenario (in terms of length of support) is the end of 2018.

Peter 

[Mike Hommey]

On Fri, Oct 14, 2016 at 11:41:29AM -0400, Peter Dolanjski wrote:
> >
> > On 13/10/16 18:52, Justin Dolske wrote:
> > > I'd suggest placing bounds on it, so we can all be on the same page and
> > > comfortable with what's being committed to. E.G. "We project that at
> > > current rates, XP usage is expected to drop to X% by the normal end of
> > > ESR52. If, at the normal end of ESR52, usage remains above X+e%, we will
> > > further extend the life of ESR52 for an additional Z months. After that
> > > Z months, ESR52 will be discontinued irrespective of usage."
>
>
> We can do that if it is useful, beyond a commitment for a final end of life
> date.
>
> On 10/13/2016 1:37 PM, L. David Baron wrote:
> >
> >> So if we're actually talking about possibly maintaining ESR52 for
> >> three years, I think we'd be better off not doing this, and
> >> continuing to support XP and Vista in our non-ESR releases.
> >>
> >> If we're willing to make a firmer commitment to ending support, I
> >> think the plan becomes more reasonable.
> >>
> >> Then we need to make that commitment.
> >
>
> Based on an upper and lower bound projection, the end of ESR52 (likely
> April-ish 2018) shows that WinXP should be between 2.5-4.2% our Firefox
> Average Monthly ADIs. By January 1st, 2019, the range is 0.8-2.6% of
> monthly ADIs (though that's trending pretty far out).
> I think we can say with some certainty that the worst case scenario (in
> terms of length of support) is the end of 2018.

... assuming the ADIs go down as projected. What if they don't?

Mike

[Chris Cooper]

On Fri, Oct 14, 2016 at 5:28 PM, Mike Hommey <m...@glandium.org> wrote:
> ... assuming the ADIs go down as projected. What if they don't?

This is possible, maybe even likely.

It's worth noting that very few Chrome XP users actually did anything
when faced with pending abandonment by Google, i.e. they didn't switch
browsers and are continue to use the final, dead-end version of
Chrome. It seems unlikely that Firefox XP users would behave any
differently especially given the lack of other browser options to move
to.

If ADI is all we care about, these users will still be reporting as
"Firefox" but won't be on the latest-and-greatest.

cheers,
--
coop