Moving FirefoxOS into Tier 3 support

[Fabrice Desré]

Hi all,

With the smartphone activity shifting to a more exploratory status, we
have been discussing with the platform & releng people which setup would
allow us to keep improving the product and supporting our community of
users while minimizing impact on other parts of the organization.

The decision we ended up with is to move Firefox OS into Tier 3
support category (see
https://developer.mozilla.org/en-U/docs/Supported_build_configurations).
That means that other teams won't be backed out and yelled at by
sheriffs for breaking b2g tests, and that the FirefoxOS team will be
responsible for fixing such breakage. Landing code for FirefoxOS stays
the same as today - we don't fork.
We're also working on a solution to move the b2g builds & tests to
their own infrastructure from which we'll ship our builds & updates.

I want to thank all the people from various corners of the platform
that helped us so far. I guess I'll have to bribe you now!

Feel free to reach out to me if you need any more details on the subject,

--
Fabrice Desré
Connected Devices
Mozilla Corporation

[Ehsan Akhgari]

Hi Fabrice,

On 2016-01-25 11:00 AM, Fabrice Desré wrote:
> Hi all,
>
> With the smartphone activity shifting to a more exploratory status, we
> have been discussing with the platform & releng people which setup would
> allow us to keep improving the product and supporting our community of
> users while minimizing impact on other parts of the organization.
>
> The decision we ended up with is to move Firefox OS into Tier 3
> support category (see
> https://developer.mozilla.org/en-U/docs/Supported_build_configurations).
> That means that other teams won't be backed out and yelled at by
> sheriffs for breaking b2g tests, and that the FirefoxOS team will be
> responsible for fixing such breakage. Landing code for FirefoxOS stays
> the same as today - we don't fork.
> We're also working on a solution to move the b2g builds & tests to
> their own infrastructure from which we'll ship our builds & updates.

I have two questions about this:

1. What does this mean for Firefox OS specific code in Gecko which is
designed to keep some level of testing possible on the Firefox OS side
by adding hacks to Gecko? Another example is hacks that we have needed
around Firefox OS's toolchain limitations (the gcc version requirements,
for example.) Can we remove such hacks now?

2. What about patches that couldn't land because of Firefox OS tests
failing in ways the author couldn't figure out? I'm thinking of
examples such as bug 1043562. Can we land such patches now?

Thanks,
Ehsan

[doug....@gmail.com]

I think the answer to (1) is no. There might be specific code-removals which enable us to move faster on supporting Firefox. I believe blassey has specific details on this. But, in general, do no harm yet.

As for (2), land patches and don't think about FirefoxOS. It's tier three and that's the point.

[Fabrice Desré]

Hi Ehsan,

On 01/25/2016 08:38 AM, Ehsan Akhgari wrote:

> I have two questions about this:
>
> 1. What does this mean for Firefox OS specific code in Gecko which is
> designed to keep some level of testing possible on the Firefox OS side
> by adding hacks to Gecko? Another example is hacks that we have needed
> around Firefox OS's toolchain limitations (the gcc version requirements,
> for example.) Can we remove such hacks now?

First, what you call "now" will not happen until we setup the
infrastructure we need for b2g. I would appreciate you to send me bug
numbers for all these issues you're talking about.
Secondly, since I really don't want us to fork gecko it would be
unfortunate if people felt like they can just burn the house. Please use
your best judgment and talk to people before *knowingly* breaking us.

> 2. What about patches that couldn't land because of Firefox OS tests
> failing in ways the author couldn't figure out? I'm thinking of
> examples such as bug 1043562. Can we land such patches now?

Yes.

[Ehsan Akhgari]

On 2016-01-25 12:11 PM, Fabrice Desré wrote:
> Hi Ehsan,
>
> On 01/25/2016 08:38 AM, Ehsan Akhgari wrote:
>
>> I have two questions about this:
>>
>> 1. What does this mean for Firefox OS specific code in Gecko which is
>> designed to keep some level of testing possible on the Firefox OS side
>> by adding hacks to Gecko? Another example is hacks that we have needed
>> around Firefox OS's toolchain limitations (the gcc version requirements,
>> for example.) Can we remove such hacks now?
>
> First, what you call "now" will not happen until we setup the
> infrastructure we need for b2g. I would appreciate you to send me bug
> numbers for all these issues you're talking about.

Hmm, I'm not sure if there are bugs on file for all of these things.
But an example would be bug 1197379.

> Secondly, since I really don't want us to fork gecko it would be
> unfortunate if people felt like they can just burn the house. Please use
> your best judgment and talk to people before *knowingly* breaking us.

Of course. :-)

My point wasn't about code that is needed for basic b2g support. I'm
talking about situations that caused us to add hacks inside Gecko
because of either partner issues or b2g timeline/release pressure.

For example, for a long time b2g partners held back our minimum
supported gcc. Now that there are no such partner requirements, perhaps
we can consider bumping up the minimum to gcc 4.8? (bug 1175546)

Another example, last year because of 2.5 release pressure where people
were planning on using service workers in gaia app, we had to add a huge
hack for "supporting" app:// URIs for service worker interception. The
last time that this code bit me was earlier this morning. It would be
nice if I can remove this broken code now instead of worrying about how
to keep supporting it going forward (this makes fixing bug 1222008 more
complicated than it needs to be.)

I'm sure others have similar examples to fill in.

>> 2. What about patches that couldn't land because of Firefox OS tests
>> failing in ways the author couldn't figure out? I'm thinking of
>> examples such as bug 1043562. Can we land such patches now?
>
> Yes.

Fantastic!

Just to double check, does the above "now" apply to this as well? IOW,
can I land that patch right now, or should I wait for a future
announcement which goes out when the aforementioned separate build/test
infrastructure is ready?

Cheers,
Ehsan

[Nathan Froyd]

On Mon, Jan 25, 2016 at 12:30 PM, Ehsan Akhgari <ehsan....@gmail.com>
wrote:

> For example, for a long time b2g partners held back our minimum supported
> gcc. Now that there are no such partner requirements, perhaps we can
> consider bumping up the minimum to gcc 4.8? (bug 1175546)
>

> I'm sure others have similar examples to fill in.
>

One current example is b2g's reliance on stlport and changing the build to
support a modern C++ library like libc++.

-Nathan

[Eric Rescorla]

It would be great to be able to make this change.

-Ekr


>
> -Nathan
> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Fabrice Desré]

On 01/25/2016 09:30 AM, Ehsan Akhgari wrote:

> For example, for a long time b2g partners held back our minimum
> supported gcc. Now that there are no such partner requirements, perhaps
> we can consider bumping up the minimum to gcc 4.8? (bug 1175546)

We moved to 4.8 on b2g a year ago: see
https://bugzilla.mozilla.org/show_bug.cgi?id=1056337
Who's behind? :P

> Another example, last year because of 2.5 release pressure where people
> were planning on using service workers in gaia app, we had to add a huge
> hack for "supporting" app:// URIs for service worker interception. The
> last time that this code bit me was earlier this morning. It would be
> nice if I can remove this broken code now instead of worrying about how
> to keep supporting it going forward (this makes fixing bug 1222008 more
> complicated than it needs to be.)

I have no objections to remove this particular support.

[Joshua Cranmer 🐧]

On 1/25/2016 11:30 AM, Ehsan Akhgari wrote:
> For example, for a long time b2g partners held back our minimum
> supported gcc. Now that there are no such partner requirements,
> perhaps we can consider bumping up the minimum to gcc 4.8? (bug 1175546)

Strictly speaking, I would advocate for 4.8.1, since that gets us ref
qualifiers on methods (or will, once we get VS 2015 as the minimum
requirement on Windows).

--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

[Naoki Hirata]

It sounds basically like Gecko support is cut off for the most part. If
there's no decision to pull it out of teir 3 support, ie a direction for
the project needs to be decided...
the project will eventually fail unless we lock into a gecko. (60 % of
failures for smoke tests come from gecko).

How soon do you think we will pull out of tier 3 support?

On Mon, Jan 25, 2016 at 9:40 AM, Fabrice Desré <fab...@mozilla.com> wrote:

> On 01/25/2016 09:30 AM, Ehsan Akhgari wrote:
>
> > For example, for a long time b2g partners held back our minimum
> > supported gcc. Now that there are no such partner requirements, perhaps
> > we can consider bumping up the minimum to gcc 4.8? (bug 1175546)
>

> We moved to 4.8 on b2g a year ago: see
> https://bugzilla.mozilla.org/show_bug.cgi?id=1056337
> Who's behind? :P
>
> > Another example, last year because of 2.5 release pressure where people
> > were planning on using service workers in gaia app, we had to add a huge
> > hack for "supporting" app:// URIs for service worker interception. The
> > last time that this code bit me was earlier this morning. It would be
> > nice if I can remove this broken code now instead of worrying about how
> > to keep supporting it going forward (this makes fixing bug 1222008 more
> > complicated than it needs to be.)
>
> I have no objections to remove this particular support.
>
> --
> Fabrice Desré
> Connected Devices
> Mozilla Corporation

> _______________________________________________
> dev-fxos mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-fxos
>

[Steve Fink]

On 01/25/2016 09:40 AM, Fabrice Desré wrote:
> On 01/25/2016 09:30 AM, Ehsan Akhgari wrote:
>
>> For example, for a long time b2g partners held back our minimum
>> supported gcc. Now that there are no such partner requirements, perhaps
>> we can consider bumping up the minimum to gcc 4.8? (bug 1175546)
> We moved to 4.8 on b2g a year ago: see
> https://bugzilla.mozilla.org/show_bug.cgi?id=1056337
> Who's behind? :P

I am.

The b2g rooting hazard analysis build is still using gcc 4.7. I spent a
bunch of effort trying to upgrade it to gcc 4.9, but ran into a variety
of issues I didn't understand related to the b2g build system (both on
my local laptop and on a build slave), and finally gave up in despair.
But the b2g hazard build is also a mozharness tangle of mixins and weird
inheritance structure (as is the older b2g emulator build), and those
are being replaced with taskcluster-based builds.

In the last 2 weeks, I've been working on redoing the b2g hazard build
on top of taskcluster and mulet. Partly because it's mulet, and partly
because it uses Docker and simple shell scripts, it has been *way* way
way way way *way* easier and more pleasant to deal with. I have it
working locally, so I hope to have the b2g hazard builds upgraded to gcc
4.9 soonish.

But that's on top of mulet, which means it isn't compiling any of the
MOZ_B2G_RIL code, which means it has lost some coverage over the
original build. If I understand correctly, the "right" thing to do would
be to use an emulator build instead, but that means that the b2g build
system gets involved again, which is complex and slings around a lot
more data, so everything is far slower to work on.

With FxOS becoming tier 3, I am disinclined to even attempt the emulator
version. In theory, it would be a straightforward adaptation of the
mulet hazard build script. In practice, it would take a while to even
test out that theory.

[Andrew Halberstadt]

On 25/01/16 11:00 AM, Fabrice Desré wrote:
> We're also working on a solution to move the b2g builds & tests to
> their own infrastructure from which we'll ship our builds & updates.

What specifically does this mean? Do you mean infrastructure at the IT
level? Or at the continuous integration level (taskcluster,
mozharness, test harnesses, etc)?

If the latter, is there a detailed outline of the plans being proposed here?

-Andrew

[nhi...@mozilla.com]

I think the QC gonk layer had required 4.7 to lunch.

As far as I know you would have had to use the emulator from the get go if you wanted to test the ril? The ril is in the Gonk layer as far as I know still and the Gonk layer isn't in the mulet nor desktop builds...

[nhi...@mozilla.com]

More over : https://bugzilla.mozilla.org/show_bug.cgi?id=1239082 ; Aren't we
disabling all Buildbot Based b2g desktop "hazard" builds on all trees ?

[Steve Fink]

Sorry. To be clear: the (still existing) hazard builds *do* use the
emulator, which is at least half the reason why they're so difficult to
maintain and upgrade.

I don't know what the Gonk layer is exactly. I do know that there is
code in the gecko tree guarded by #ifdef MOZ_B2G_RIL, and I believe that
code is compiled by the same compiler that compiles the rest of gecko,
which is why the hazard analysis found bugs in it. There is other code
compiled when doing an emulator build that is compiled by a different
compiler, but it can't call the JSAPI directly or indirectly so I don't
care about it for the analysis. (The problems I ran into when I
attempted to change only the gecko compiler seemed to be related to more
stuff than I expected getting compiled with the gecko compiler, but
that's my confused impression of what was going on and doesn't really
matter here.)

As for bug 1239082, it started as bug 1236835 that originally planned to
turn off *all* buildbot based b2g desktop builds. When I stated that I
still needed the hazard builds, Callek reduced his changes so that he'd
turn off all but the hazard builds, then forked off bug 1239082 to
finish up the job after I migrated the hazard builds to taskcluster. I'm
working on that (that's the thing I said works locally on my laptop
now), but I did it as a mulet-based build so I wouldn't have to deal
with the b2g build system again, at least until we decided we needed the
ril coverage back.

Only now with b2g being tier-3, I'm thinking that if b2g needs that
coverage back, then a person who is not me can port my mulet build
changes over to the emulator build script, and I will congratulate
not-me on a job well done and breathe a sigh of relief that not-me is,
in fact, not me.

[Mike Hommey]

On Mon, Jan 25, 2016 at 11:57:40AM -0600, Joshua Cranmer ? wrote:
> On 1/25/2016 11:30 AM, Ehsan Akhgari wrote:
> >For example, for a long time b2g partners held back our minimum supported
> >gcc. Now that there are no such partner requirements, perhaps we can
> >consider bumping up the minimum to gcc 4.8? (bug 1175546)
>
> Strictly speaking, I would advocate for 4.8.1, since that gets us ref
> qualifiers on methods (or will, once we get VS 2015 as the minimum
> requirement on Windows).

IIRC, 4.8.x < 4.8.5 (or was it 4.8.3?) miscompiles gecko anyways.

Mike

[Juan Gómez]

Yeah, upgarding to gcc 4.8 was a bit tough but it established the basis for
less compllcated upgardes in the future. I could compile all FFOS flavors
(ICS, JB, KK and LL) using gcc-4.9 base toolchain without any remarcable
complication. My plan was to upgrade to gcc 4.9 [1] last year, but we had
to stop it becuase we needed the partner confirmation to move forward. Now
that we don't need that confirmation, we could resume the work and land
gcc-4.9 support.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1087161

On Mon, Jan 25, 2016 at 6:40 PM, Fabrice Desré <fab...@mozilla.com> wrote:

> On 01/25/2016 09:30 AM, Ehsan Akhgari wrote:
>
> > For example, for a long time b2g partners held back our minimum
> > supported gcc. Now that there are no such partner requirements, perhaps
> > we can consider bumping up the minimum to gcc 4.8? (bug 1175546)
>

> We moved to 4.8 on b2g a year ago: see
> https://bugzilla.mozilla.org/show_bug.cgi?id=1056337
> Who's behind? :P
>

> > Another example, last year because of 2.5 release pressure where people
> > were planning on using service workers in gaia app, we had to add a huge
> > hack for "supporting" app:// URIs for service worker interception. The
> > last time that this code bit me was earlier this morning. It would be
> > nice if I can remove this broken code now instead of worrying about how
> > to keep supporting it going forward (this makes fixing bug 1222008 more
> > complicated than it needs to be.)
>
> I have no objections to remove this particular support.
>
> --
> Fabrice Desré
> Connected Devices
> Mozilla Corporation

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

--
___________________
Juan Gómez Mosquera
Platform Engineer
Mozilla Corporation
_AtilA_

[Tim Guan-tin Chien]

On Tue, Jan 26, 2016 at 12:00 AM, Fabrice Desré <fab...@mozilla.com> wrote:
> Hi all,
>
> With the smartphone activity shifting to a more exploratory status, we
> have been discussing with the platform & releng people which setup would
> allow us to keep improving the product and supporting our community of
> users while minimizing impact on other parts of the organization.
>
> The decision we ended up with is to move Firefox OS into Tier 3
> support category (see
> https://developer.mozilla.org/en-U/docs/Supported_build_configurations).
> That means that other teams won't be backed out and yelled at by
> sheriffs for breaking b2g tests, and that the FirefoxOS team will be
> responsible for fixing such breakage. Landing code for FirefoxOS stays
> the same as today - we don't fork.

> We're also working on a solution to move the b2g builds & tests to
> their own infrastructure from which we'll ship our builds & updates.
>

> I want to thank all the people from various corners of the platform
> that helped us so far. I guess I'll have to bribe you now!
>
> Feel free to reach out to me if you need any more details on the subject,
>

What's the scope of "b2g tests"?

I can see obvious ones like any test testing Gaia and tests running on
Mulet or Emulator, but how about DOM mochitests on, say, Firefox
Desktop, for APIs that only B2G uses?

Is there a definitive list? Are we purposely not going to make a list
just to be flexible?


Tim

[Ehsan Akhgari]

On 2016-01-25 5:59 PM, Juan Gómez wrote:
> Yeah, upgarding to gcc 4.8 was a bit tough but it established the basis
> for less compllcated upgardes in the future. I could compile all FFOS
> flavors (ICS, JB, KK and LL) using gcc-4.9 base toolchain without any
> remarcable complication. My plan was to upgrade to gcc 4.9 [1] last
> year, but we had to stop it becuase we needed the partner confirmation
> to move forward. Now that we don't need that confirmation, we could
> resume the work and land gcc-4.9 support.

That would be wonderful!

[Fabrice Desré]

On 01/25/2016 05:15 PM, Tim Guan-tin Chien wrote:

> What's the scope of "b2g tests"?
>
> I can see obvious ones like any test testing Gaia and tests running on
> Mulet or Emulator, but how about DOM mochitests on, say, Firefox
> Desktop, for APIs that only B2G uses?
>
> Is there a definitive list? Are we purposely not going to make a list
> just to be flexible?

I expect all products built with MOZ_B2G to not be built and tested on
the main treeherder. This means device builds, emulator builds and
tests, mulet build and tests, and b2gdroid builds. You're right that for
apis that are turned on by prefs in tests that's a gray area. That will
be on the module owners to decide what to do.

[Shawn Huang]

Hi Fabrice,
Following this comment, I'm confused. Do we need to check-in into
b2g-inbound by hand? "checked-in" keyboard no longer supports FirefoxOS
project? Does this mean only people who have Level 3 access permission can
land code?


https://bugzilla.mozilla.org/show_bug.cgi?id=1201778#c102

"dropping the checkin needed keyword here, alphan with the new tier 3
change for b2g and the drop of sheriff support for tier 3 we do not do
checkin needed anymore for b2g-inbound - if this needs to land on
mozilla-central let me know"

On 26 January 2016 at 00:00, Fabrice Desré <fab...@mozilla.com> wrote:

> Hi all,
>
> With the smartphone activity shifting to a more exploratory status, we
> have been discussing with the platform & releng people which setup would
> allow us to keep improving the product and supporting our community of
> users while minimizing impact on other parts of the organization.
>
> The decision we ended up with is to move Firefox OS into Tier 3
> support category (see
> https://developer.mozilla.org/en-U/docs/Supported_build_configurations).
> That means that other teams won't be backed out and yelled at by
> sheriffs for breaking b2g tests, and that the FirefoxOS team will be
> responsible for fixing such breakage. Landing code for FirefoxOS stays
> the same as today - we don't fork.
> We're also working on a solution to move the b2g builds & tests to
> their own infrastructure from which we'll ship our builds & updates.
>
> I want to thank all the people from various corners of the platform
> that helped us so far. I guess I'll have to bribe you now!
>
> Feel free to reach out to me if you need any more details on the subject,
>

> --
> Fabrice Desré
> Connected Devices
> Mozilla Corporation

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>



--

Regards,
Shawn Huang

[Gabriele Svelto]

On 26/01/2016 10:19, Shawn Huang wrote:
> Hi Fabrice,
> Following this comment, I'm confused. Do we need to check-in into
> b2g-inbound by hand? "checked-in" keyboard no longer supports FirefoxOS
> project? Does this mean only people who have Level 3 access permission can
> land code?

Same here. I'm now unable to land the gecko dependencies for bug 1227980
[1] and that's annoying to say the least considering it took a month to
write and test that thing (not even mentioning the time spent by the
many reviewers and QA people involved). What are we supposed to do with
those kind of patches which are b2g-only? Shall we land them ourselves
on mozilla-central?

Gabriele

[1] [Dialer] Merge the callscreen app back into the dialer and remove
the associated system app hacks
https://bugzilla.mozilla.org/show_bug.cgi?id=1227980

[jma...@mozilla.com]

> Same here. I'm now unable to land the gecko dependencies for bug 1227980
> [1] and that's annoying to say the least considering it took a month to
> write and test that thing (not even mentioning the time spent by the
> many reviewers and QA people involved). What are we supposed to do with
> those kind of patches which are b2g-only? Shall we land them ourselves
> on mozilla-central?

I would assume all gecko patches would get landed on either mozilla-inbound or fx-team. If you use mozreview and take advantage of the autoland feature, then these specific patches will land on mozilla-inbound.

[Gabriele Svelto]

On 26/01/2016 10:51, jma...@mozilla.com wrote:
> I would assume all gecko patches would get landed on either mozilla-inbound or fx-team. If you use mozreview and take advantage of the autoland feature, then these specific patches will land on mozilla-inbound.

Thanks, I'll definitely have a look at mozreview/autoland.

Gabriele

[Adam Farden]

If we jump on the Marshmallow bandwagon we can drop stlport, Google already
did that too.

https://bugzilla.mozilla.org/show_bug.cgi?id=1213259
https://bugzilla.mozilla.org/show_bug.cgi?id=1218802 - specifically patch
[07]

On Mon, Jan 25, 2016 at 6:34 PM, Nathan Froyd <nfr...@mozilla.com> wrote:

> On Mon, Jan 25, 2016 at 12:30 PM, Ehsan Akhgari <ehsan....@gmail.com>

> wrote:
>
>> For example, for a long time b2g partners held back our minimum supported
>> gcc. Now that there are no such partner requirements, perhaps we can
>> consider bumping up the minimum to gcc 4.8? (bug 1175546)
>>

>> I'm sure others have similar examples to fill in.
>>
>
> One current example is b2g's reliance on stlport and changing the build to
> support a modern C++ library like libc++.
>

> -Nathan

[Andrew Halberstadt]

Hi, I'm on the engineering productivity team, and work a lot on
continuous integration and test harnesses. I stood up initial Gecko
unittests on B2G emulator, worked on B2G automation for several years up
until the divide, and still help nominally maintain the emulator and
mulet unittests. So that's the hat I'll be wearing for this post.

I'd like to just state a few of the implications that this, and the
following quote from a gecko-all mail have:
> This means that all engineering on Firefox OS by Platform
> Engineering and Platform Operations will stop.

The purpose of this post is simply to make sure everyone involved is
aware of the following ramifications:

1. Gecko based test harnesses (mochitest, xpcshell, reftest, etc) will
eventually break and their corresponding jobs will be disabled. At some
point, the test harness will need to undergo a large enough change that
it will require a non-trivial amount of effort to not break B2G
emulators and mulet. I'd estimate for most harnesses, this will happen
sooner rather than later (within a quarter or two). When this happens,
the jobs will be turned off completely so as not to waste money on our
AWS bill. To be crystal clear, this means no more mochitest, reftest or
xpcshell on B2G emulators. Mulet will likely last a little longer as it
is similar enough to Firefox desktop.

2. If at any point CD wishes to rejoin mainline development and run the
set of Gecko unittests once again, re-integration will be a long and
difficult process.

3. Gaia tests will still need a substantial effort to keep green. This
one is more obvious, but still worth stating. It's really hard to keep a
job green after the fact. In my experience, keeping jobs in Tier 3 for
any extended period of time is not sustainable. CD will likely need to
fork if they want to keep these jobs green.

I think a question worth asking, is should we bother with Tier 3 at all?
Or should we jump straight to disabling CD specific jobs. I guess it
doesn't hurt to leave them running while they last, but in some cases
this will be a very short time frame.

Cheers,
Andrew