Shutting down legacy Taskcluster deployment

114 views
Skip to first unread message

Chris Cooper

unread,
Jun 25, 2020, 5:13:32 PM6/25/20
to dev-platform, Mozilla Product Builds, firef...@mozilla.org, tools-taskcluster, Sheriffs, perf-s...@mozilla.com
Hey all,

You may recall that last fall we redeployed Taskcluster into two
distinct clusters, one to support Firefox CI and another to handle
Community projects[1]. We left the legacy deployment running to
provide access to CI artifacts that would only be reachable via the
old taskcluster.net interface.

It turns out that we (Mozilla) don't access the legacy data very
often: we are storing close to a petabyte of older artifacts in AWS
and yet we access less than 25 GB of it. It costs us thousands of
dollars every month for that storage, and several hundred dollars more
to continue running the services to access it, so we've made a
decision to decommission the legacy Taskcluster deployment a few
months earlier than planned.

We'll be shutting down the legacy services on July 3rd, 2020. That's
one week from today. A week later on July 10, we will begin deleting
the old artifacts.

If you are one of the few individuals who does access artifacts from
the legacy cluster and haven't already spoken to us about
alternatives, please reach out before next Thursday.

Note: this will *not* affect day-to-day CI on either the Firefox CI or
Community clusters at all.

Thanks,
--
Chris Cooper
Manager, Taskcluster @ Mozilla
<co...@mozilla.com>

1. https://groups.google.com/g/mozilla.dev.platform/c/QhZFDhlQjxU/m/ngk8z6j1AgAJ

Kartikaya Gupta

unread,
Jun 26, 2020, 6:42:08 AM6/26/20
to Chris Cooper, William Lachance, dev-platform, Mozilla Product Builds, firefox-dev-owner firefox-dev-owner, tools-taskcluster, Sheriffs, perf-s...@mozilla.com
Do these legacy artifacts include firefox builds that are used by
mozregression? It looks like there's still a reference to the old
taskcluster.net URL at
https://github.com/mozilla/mozregression/blob/3df977cf7b9c50bdbe8495cde17baeb6c5724be2/mozregression/config.py#L23
which gets used. It would be valuable to ensure mozregression can
continue bisecting old builds after this decommissioning happens.
> _______________________________________________
> firefox-dev mailing list
> firef...@mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev

Chris Cooper

unread,
Jun 26, 2020, 9:53:50 AM6/26/20
to dev-platform, Mozilla Product Builds, tools-taskcluster, firefox-dev-owner firefox-dev-owner, Sheriffs, perf-s...@mozilla.com
I've received multiple replies about mozregression, so I'll provide
some more context.

Taskcluster artifacts expire by default after one year. This means the
legacy cluster currently provides access to artifacts from June 26 to
Nov 9, 2019 (Nov 9 was the cutover date to the new deployment). By
shutting down the legacy cluster on July 3, we will be removing access
to just over 4 months of artifacts. Note that this won't affect
separate release archives like
http://archive.mozilla.org/pub/firefox/. The end result for
mozregression is that it will have fewer choices for intermediate
builds while bisecting if it needs to reach back into that 4-month
window and will need to rely on the nightlies.

I looked at the specific S3 stats for older artifacts. I can't break
mozregression usage out specifically, but over the past year we (all
parties) have only accessed 19.04 GB of the 971.77 TB that are older
than 180 days.

cheers,
--
coop

Kartikaya Gupta

unread,
Jun 26, 2020, 3:01:37 PM6/26/20
to Chris Cooper, dev-platform, Mozilla Product Builds, tools-taskcluster, firefox-dev-owner firefox-dev-owner, Sheriffs, perf-s...@mozilla.com
Thanks for clarifying. If nightlies are still available for that
period then I guess that's ok. It will be a slight inconvenience but
probably not worth the effort to and save those builds for a few
months.

Jeff Muizelaar

unread,
Jun 26, 2020, 3:14:11 PM6/26/20
to Kartikaya Gupta, Chris Cooper, Sheriffs, tools-taskcluster, Mozilla Product Builds, perf-s...@mozilla.com, firefox-dev-owner firefox-dev-owner, dev-platform
Losing 4 months of mozregression does sort of suck though. Being able
to bisect down to a specific push is a lot more helpful than a
specific nightly. I have it often enough that I'm annoyed by only
having 1 year of builds.

What percentage of the space used for artifacts is actually builds
that are used for mozregression vs other stuff (like debug builds)? Is
there a way that we could somehow have different expiraries for the
different kinds of data? 1 year for logs, debug symbols and debug
builds seems very generous but 8 months for builds for mozregression
seems stingy.

-Jeff
> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

Andrew Halberstadt

unread,
Jun 26, 2020, 3:34:47 PM6/26/20
to Jeff Muizelaar, Kartikaya Gupta, Sheriffs, tools-taskcluster, Mozilla Product Builds, perf-s...@mozilla.com, firefox-dev-owner firefox-dev-owner, dev-platform, Chris Cooper
On Fri, Jun 26, 2020 at 3:14 PM Jeff Muizelaar <jmuiz...@mozilla.com>
wrote:

> What percentage of the space used for artifacts is actually builds
> that are used for mozregression vs other stuff (like debug builds)? Is
> there a way that we could somehow have different expiraries for the
> different kinds of data? 1 year for logs, debug symbols and debug
> builds seems very generous but 8 months for builds for mozregression
> seems stingy.
>

Keep in mind this is only temporary, with every passing day we approach a
full year's worth of builds again. As a longer term solution to this
particular problem, I wonder if we can get mozregression to farm out
missing builds on try.. then there would be no limit to how far back it
could go.

Tom Ritter

unread,
Jun 26, 2020, 4:04:24 PM6/26/20
to Andrew Halberstadt, Jeff Muizelaar, Sheriffs, tools-taskcluster, Mozilla Product Builds, Kartikaya Gupta, perf-s...@mozilla.com, firefox-dev-owner firefox-dev-owner, dev-platform, Chris Cooper
On Fri, Jun 26, 2020 at 7:34 PM Andrew Halberstadt <ah...@mozilla.com> wrote:
>
> On Fri, Jun 26, 2020 at 3:14 PM Jeff Muizelaar <jmuiz...@mozilla.com> wrote:
>>
>> What percentage of the space used for artifacts is actually builds
>> that are used for mozregression vs other stuff (like debug builds)? Is
>> there a way that we could somehow have different expiraries for the
>> different kinds of data? 1 year for logs, debug symbols and debug
>> builds seems very generous but 8 months for builds for mozregression
>> seems stingy.
>
>
> Keep in mind this is only temporary, with every passing day we approach a full year's worth of builds again. As a longer term solution to this particular problem, I wonder if we can get mozregression to farm out missing builds on try.. then there would be no limit to how far back it could go.

I think that would be difficult. While taskcluster specifiesthe
toolchain configuration, I think changes to TC infrastructure mean
that old trees do not easily run there. (I think worker-types
disappear for example.) And locally, your toolchain can be too new to
build an old version of Firefox - I've had to switch back to older
versions of rust a few times in particular.

-tom

Mike Hommey

unread,
Jun 26, 2020, 4:36:01 PM6/26/20
to Tom Ritter, Andrew Halberstadt, Sheriffs, Kartikaya Gupta, Mozilla Product Builds, tools-taskcluster, perf-s...@mozilla.com, firefox-dev-owner firefox-dev-owner, Jeff Muizelaar, dev-platform, Chris Cooper
Heck, we need to land patches to the esr68 branch for taskcluster infra
changes. As a matter of fact, anything that predates the switch to the
new infra won't work on try. I'm sure there are other changes that
happened since the switch that require newer code from mozilla-central
(ISTR there were gecko-decision task failures just a few weeks ago that
needed newer m-c)

Now, to come back on the longer term solution, the only thing I can
think of would be to publish all the shippable builds from taskcluster
to the "FTP" server. Because there's no saying the taskcluster infra
won't migrate again in x years and we'd have the same problem as now.

Mike

wlac...@mozilla.com

unread,
Aug 17, 2020, 3:25:32 PM8/17/20
to
On Friday, June 26, 2020 at 4:36:01 p.m. UTC-4, Mike Hommey wrote:
> On Fri, Jun 26, 2020 at 08:04:01PM +0000, Tom Ritter wrote:
> > On Fri, Jun 26, 2020 at 7:34 PM Andrew Halberstadt <ah...@mozilla.com> wrote:
> > >
> > > On Fri, Jun 26, 2020 at 3:14 PM Jeff Muizelaar <jmuiz...@mozilla.com> wrote:
> > >>
> > >> What percentage of the space used for artifacts is actually builds
> > >> that are used for mozregression vs other stuff (like debug builds)? Is
> > >> there a way that we could somehow have different expiraries for the
> > >> different kinds of data? 1 year for logs, debug symbols and debug
> > >> builds seems very generous but 8 months for builds for mozregression
> > >> seems stingy.

Hi all, I recently added some telemetry to mozregression to measure how often people were requesting older builds to try and answer these questions more scientifically. I wrote up a little report on this, hope people find it helpful:

https://wlach.github.io/blog/2020/08/mozregression-and-older-builds/

tl;dr: Most people are requesting newer builds (like in the last 6 months) but there's a long tail of people requesting older stuff, which shouldn't be completely discounted.

Will

Reply all
Reply to author
Forward
0 new messages