Reclaiming space on stage.mozilla.org - revisiting bug 342972
Chris Cooper
Hash: SHA1
For those who want to skip to my specific proposals ? there are 6 ? for
reclaiming space on stage.mozilla.org, please skip ahead to ?Redux?, but
if you?re going to comment, please read the whole thing.
Everyday we produce up to 17G worth of new nightly builds for Firefox
across all branches. This includes all opt+debug builds in 75+ locales,
each on 4 operating systems (OSes), each on 9 different project
branches. We do reclaim much of this 17G as we retire l10n builds older
than 1 week, but we are still creating 1.3G of new nightly en-US builds
that we need to store (essentially) indefinitely. nthomas did some
cleanup recently as part of bug 562261[1] and that has bought us some
more time, but this inexorable increase will eventually overrun our disk
capacity on the staging server. If we add to this disk usage by
nightlies from other products which may have worse nightly hygiene
habits and the expected increase in space requirements every night as we
add 4 new OSes (adding Linux 64bit, Windows 64bit, OSX10.6 64bit and
Android), the problem is magnified.
To date, our solution to this problem has been to do periodic cleanups,
usually under duress like bug 562261[1] (which can be error-prone), or
to simply buy more disk. As Justin notes[2], while disk space may be
?cheap?, it is not an infinite resource, either in terms of upfront or
management cost. We need an actual policy to govern how long we keep
nightlies. We can then use that policy as a baseline to frame further
discussions about keeping things for longer periods in special cases.
The good news: there *is* space that can be reclaimed: There are two
types of wins to be had here:
1. One-time space recovery by deleting or archiving material that is
no longer useful, or that can be kept offline and spun up as required.
2. Codified policy changes for automatically expiring old content.
We can tackle #1 (one-time space recovery) for Firefox by:
a. moving no-longer-supported releases, i.e. anything prior to 3.0
(including firebird) to a true archive. This will free up about 125G of
space, and will have the added benefit of not housing unsupported builds
next to supported builds, making them a little more difficult for people
to stumble upon.
b. deleting nightly builds older than a certain date. The Firefox
nightly directories are conveniently broken down by year, so it makes it
easy to see how much disk space we could reclaim by deleting old nightlies:
2004 16G
2005 53G
2006 191G
2007 129G
2008 177G
2009 236G
2010 216G (so far)
At the risk of making others? arguments for them, both previous times we
attempted to come up with a stage cleanup policy (2006 & 2008) the major
concern raised was a need to keep builds in perpetuity to allow
regression detection via binary search.
So I say this: I?d like to hear from developers who have actually had to
perform binary searches of nightlies to let me know exactly how far back
in time they have had to go. In the absence of any other data, I?m going
to suggest we remove all nightlies prior to 2007 simply because that
corresponds with the start of the hg era (March 2007).
Please note, that when I say ?remove? in this context of this
discussion, I am advocating for ?delete? but understand that I may need
to settle for ?archive? in whatever form that makes sense to IT (slow
disk/tape/???).
Other non-Firefox projects will need to make their own decisions as to
how/when to archive older releases, but those projects could also
reclaim a lot of space by deleting their older nightlies. Here are the
aggregate disk usage number for nightly builds of Calendar (both Sunbird
& Lightning), Camino, Mozilla Suite (not even built since 2007), and
XulRunner, broken down by year:
2001 472M
2002 6.9G
2003 6.7G
2004 23.2G
2005 57.6G
2006 79G
2007 106G
2008 114G
Again, there are big space recovery wins to be had here, depending on
far back we want our accessible, online repository of nightly builds to be.
In terms of policy changes to curb accumulation, I propose implementing
three reforms, the first two of which come out of nthomas? work in bug
562261.
First, we?ll script the automatic expiry of mar files for nightly builds
older than 1 month. Only the most recent complete and partial MARs are
required, so this gives us a more-than-adequate buffer to detect and fix
problems with the nightly update system. We have proven steps from bug
562261[1] that can be easily cron-ed to run weekly on weekends or other
periods when the staging server is (relatively) idle. This can be done
across all products.
Second, the RelEng team will start purging the contents of old
candidates directories as part of our release process. For those who are
unaware, the candidates directory lives under the nightly directory and
holds all the various release files (builds, source, signatures, logs)
until we green-light the release. Once the release is official, the
important contents are sync-ed over to the releases/ subdir and the
candidates dir becomes mostly redundant, modulo a few important logging
artifacts. We?ll delete the builds for all but the two most recent
candidate dirs, but will preserve the text files/logs that tell us
important things like # of builds/changesets/build IDs.
Note that this won?t be an automated procedure: the release engineer
responsible for the current release will need to go in and look at the
candidates directories involved and make a judgment call as to what to
delete. Sometimes the release procedure goes awry or we try something
new, and it?s important to be able to keep those examples around
until we?ve learned what we can from them.
Other projects that currently use a candidates directory for releases
should also consider making this change.
Third, we should agree to revisit nightly storage on a yearly basis.
Specifically, we should commit to taking the oldest year?s worth of
nightly builds offline in January of each year, e.g. if we?re
comfortable with a 3-year online repository of nightly builds, in
January 2011 we would take the nightly builds for 2007 offline. As you
can see from the YTD numbers for 2010, this is unlikely to actually keep
up with increasing storage needs, but it is better than nothing.
Redux:
Here?s a brief summary of my proposals for reclaiming space on stage to
make it easier for people to respond *AFTER* reading the above:
1) Move Firefox releases that are no longer supported (< 3.0, including
firebird) to separate storage.
2) Remove Firefox nightlies prior to 2007, freeing 260G. These can be
deleted if they're not going to be used, or archived if we think they might.
3) Remove nightlies for products other than Firefox prior to 2007,
freeing 174G. Again, "remove" can mean either deletion or archiving.
4) Automate the deletion of nightly MAR files older than one month. Only
the most recent MAR files are required. This would be done across all
products.
5) Delete builds from older candidates directories after official
release. This will reclaim up to 13G per build attempt per release. This
will be a manual process.
6) For every new year going forward, remove the oldest remaining year of
nightlies, e.g. for a 3-year history of nightly builds, remove nightly
builds from 2007 in January 2011. This will be a manual process.
Feedback is appreciated, either in the newsgroups or in bug 342972[3].
Corresponding blog post is here:
http://coop.deadsquid.com/2010/05/reclaiming-space-on-stage-mozilla-org/
cheers,
- --
coop
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=562261
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=342972#c33
[3] https://bugzilla.mozilla.org/show_bug.cgi?id=342972
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkvrZY0ACgkQqdY6vdJz4M+c4ACfTBUC2QFej89rYfb+bVyAoTmI
jVoAnRhvRWwxVNBb48IkrZLwiiBOzycR
=nk4F
-----END PGP SIGNATURE-----
Boris Zbarsky
> So I say this: I?d like to hear from developers who have actually had to
> perform binary searches of nightlies to let me know exactly how far back
> in time they have had to go. In the absence of any other data, I?m going
> to suggest we remove all nightlies prior to 2007 simply because that
> corresponds with the start of the hg era (March 2007).
Probably about every 10th bug I end up doing a regression search on
nightlies on lands me in 2005 or earlier (Gecko 1.8 timeframe). I
haven't had to go earlier than 2004 in a good long while).
I assume the 260G for Firefox nightlies before 2007 is across all three
platforms but only one locale? If so, I'm just not quite sure where
that number is coming from, since my total archive of 2004-2006 mac
nightlies is about 8G. My archive of Linux nightlies for 2001-2006 is
about 21G. Where is the 260G number coming from?
> Third, we should agree to revisit nightly storage on a yearly basis.
> Specifically, we should commit to taking the oldest year?s worth of
> nightly builds offline in January of each year, e.g. if we?re
> comfortable with a 3-year online repository of nightly builds, in
> January 2011 we would take the nightly builds for 2007 offline. As you
> can see from the YTD numbers for 2010, this is unlikely to actually keep
> up with increasing storage needs, but it is better than nothing.
A question. How much do we save if we only purge the old non-en-US
nightlies? Or are we purging those already?
-Boris
Robert Strong
> On 5/12/10 9:21 PM, Boris Zbarsky wrote:
>> I assume the 260G for Firefox nightlies before 2007 is across all three
>> platforms but only one locale? If so, I'm just not quite sure where that
>> number is coming from, since my total archive of 2004-2006 mac nightlies
>> is about 8G. My archive of Linux nightlies for 2001-2006 is about 21G.
>> Where is the 260G number coming from?
>
>
> Take http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2006/05/ -
> it's Aviary 1.0.x nightlies, including a Windows installer and a
> Windows zip and some XPIs I don't even recognize (a 5MB browser.xpi? a
> .5MB xpcom.xpi? UninstallFirefox.zip? for a stub installer that didn't
> exist, maybe?
The xpi's and UninstallFirefox.zip are all contained inside of the
Firefox 1.0.x and 1.5.0.x installers which are 7-zip self-extracting
archives.
> ), and Aviary Linux installers and tar.gzs, and 1.5.0.x installers and
> zips and tar.gzs and dmgs and complete and partial mars on multiple
> platforms and xforms.xpi and 2.0.x installers and zips and trunk
> installers and packages and 1.5.0.4 candidates sometimes en-US and
> sometimes across all locales on multiple days that nobody could say
> anymore what the difference was.
and the same xpi's for Linux as there are for Windows.
For Firefox 1.0.x and 1.5.0.x the windows-xpi and linux-xpi dirs are
approximately the same as their respective platform archive size.
Windows and Linux had an installer (though we did stop building the
Linux installer sometime later) which are approximately the same as
their respective archive sizes again. For 1.5.0.x and greater there are
the update mar files with the complete mar file being approximately the
same size as each platform's archive size.
Even with keeping the archives for builds (e.g. zip, tar.gz, and dmg)
for regression searches I would think it would be ok to delete the
Windows / Linux installers, <platform>-xpi directories, and the mar
files as coop suggested.
Cheers,
Robert
Gervase Markham
> Here?s a brief summary of my proposals for reclaiming space on stage to
> make it easier for people to respond *AFTER* reading the above:
Thanks for the summary! :-)
> 1) Move Firefox releases that are no longer supported (< 3.0, including
> firebird) to separate storage.
Only yesterday, I was installing a spectrum of Mozilla releases back to
M3, released in 1999. Could you elaborate a bit more on what you mean by
"move to separate storage", and how it would look from the perspective
of someone downloading stuff?
My strong instinct is to oppose taking offline anything we have ever put
out as an official alpha, beta or final release.
> 2) Remove Firefox nightlies prior to 2007, freeing 260G. These can be
> deleted if they're not going to be used, or archived if we think they might.
As I think Boris said, can we distinguish between "en-US" and "other"? I
suspect that for almost all bugs, we only need en-US to do binary search.
Also, how badly affected would our binary-regression-search capability
be if we deleted every second day's set of builds? Is it noticeably
harder to look at 2 days worth of checkins rather than 1 day's to find a
problem?
The other stuff you propose seems fine to me.
Gerv
Jean-Marc Desperrier
> Also, how badly affected would our binary-regression-search capability
> be if we deleted every second day's set of builds? Is it noticeably
> harder to look at 2 days worth of checkins rather than 1 day's to find a
> problem?
It's noticeably harder but I believe we are talking about things that
really are seldom required.
So I think for really old release it would be OK to keep only a weekly
build, dividing by 7 the space required, but *not* fully removing them.
A point to note, the older the release, the less used the environment,
the fewer people needs the builds, *but also* the harder it is to
rebuild one. And it can get real hard, I remember it from some years ago
when having to special order the specific compiler version required for
Mac OS 9 builds (the Metrowerks guys were really helpful to provide us
that non-available in retail version, but I doubt they'd still be able
to do it today).
PS: I see Chris cross-posted this to several place, with a reply-to at
dev.builds, but discussion seems to only have really started on
dev.planning.
Boris Zbarsky
> Even with keeping the archives for builds (e.g. zip, tar.gz, and dmg)
> for regression searches I would think it would be ok to delete the
> Windows / Linux installers, <platform>-xpi directories, and the mar
> files as coop suggested.
Yes. To be clear, what's useful to me personally for binary searches is
only non-installer trunk en-US builds. Installer builds, branch
nightlies, other locales are not particularly needed.
Branch nightlies for the last 2 branches (the supported ones) are worth
it to try to identify branch regressions, of course. But our date
cutoff would leave those anyway.
-Boris
Boris Zbarsky
> Is it noticeably harder to look at 2 days worth of checkins rather than 1 day's to find a
> problem?
Probably takes about 2-4 times as long, so figure we save 20-100
developer minutes per search that ends up in that situation.
-Boris
Boris Zbarsky
> It's noticeably harder but I believe we are talking about things that
> really are seldom required.
It depends on the date range. Going back to 2005 I'd say happens about
every 2 weeks for me.
> A point to note, the older the release, the less used the environment,
> the fewer people needs the builds, *but also* the harder it is to
> rebuild one.
Yes. For example, current Linux distros can't build any of our
nightlies older than early 2009 or so as far as I can tell. At least
not any of the ones using pango. So it's not like you can
build-to-bisect if the range you find through nightlies is just too big
to make sense of.
-Boris
Robert Kaiser
> My strong instinct is to oppose taking offline anything we have ever put
> out as an official alpha, beta or final release.
I fully agree there, anything that landed in releases/ should be
available online in some way, even if it's on a separate
achive.mozilla.org that is not strongly mirrored (because it won't get a
lot of traffic).
Robert Kaiser
Wayne Mery
>> 2) Remove Firefox nightlies prior to 2007, freeing 260G. These can be
>> deleted if they're not going to be used, or archived if we think they
>> might.
>
> As I think Boris said, can we distinguish between "en-US" and "other"? I
> suspect that for almost all bugs, we only need en-US to do binary search.
>
> Also, how badly affected would our binary-regression-search capability
> be if we deleted every second day's set of builds? Is it noticeably
> harder to look at 2 days worth of checkins rather than 1 day's to find a
> problem?
An option of last, last resort perhaps. This would complicate further,
hunting during periods where trunk builds are intermitently missing. Or
broken, or crashy, like Thunderbird was in early 2006 iirc.
--
http://wiki.mozilla.org/Thunderbird:Testing
http://www.spreadthunderbird.com/aff/165/
Chris Cooper
Hash: SHA1
First, here are the nightly usage numbers for Thunderbird that I
inadvertently forgot to include in the original post:
thunderbird/nightly/
2003 108M
2004 17G
2005 47G
2006 87G
2007 88G
2008 60G
2009 73G
2010 159G
To summarize the feedback I've received so far:
* regression hunting happens a lot (more than I would have thought),
enough that I think *deleting* nightlies in untenable
* if we were to pick a date before which we could archive nightlies,
people seem to go back as far as 2005
* there is lots of space to be reclaimed if we remove "cruft" from old
nightlies, where cruft is anything that is not an actual build, e.g.
installers, xpis - let me gather the numbers for how much space this
would save per product and report back
* yes, we (Mozilla) will continue to try to grow the disk space on stage
as we go forward (there is not necessarily a hard cap on space), but
that doesn't absolve us of the responsibility of cleaning up after
ourselves.
* let me verify that we don't have any old l10n builds still lurking in
the archives. We only need en-US for regression hunting.
cheers,
- --
coop
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkvsEMwACgkQqdY6vdJz4M+u4gCgkoKQDCh+wGq0GKRXPnboZ5T9
uj8AoJrn+0d1+DUrednht6WN4NuY56dn
=eSOs
-----END PGP SIGNATURE-----
sayrer
>
> * yes, we (Mozilla) will continue to try to grow the disk space on stage
> as we go forward (there is not necessarily a hard cap on space), but
> that doesn't absolve us of the responsibility of cleaning up after
> ourselves.
I don't think this approach is satisfactory.
The numbers we're talking about (and wasting time on) in this thread
are small. Really small.
These builds are probably being stored on some high quality, expensive
hard drives. We should be storing older builds on clusters of
redundant consumer grade hardware.
- Rob
matthew zeier
>
> These builds are probably being stored on some high quality, expensive
> hard drives. We should be storing older builds on clusters of
> redundant consumer grade hardware.
Surprisingly this is not the case. However, it is on hardware that won't cause everyone to panic if a component fails (or cause IT to firefight).
sayrer
So... I still don't see why we are debating how to optimize a small
amount of hard disk space. That seems like trading engineering and
management time for a very, very small savings in capital spending. We
continually fail to deal with rather small demands on HD space because
we don't employ distributed file systems on cheap hardware. Those
don't cause everyone to panic if a component fails, either.
I'm all for not storing things we absolutely don't need, but we
shouldn't be making trade-offs on keeping the output of our work
accessible on the Web, for everyone. It's just not that much data.
- Rob
Justin Fitzhugh
That said, if we show value and reasons for keeping this data - I am all for it. An example where we did this was with socorro (storing 3 years of crash data) where we did exactly what you talk about (hdfs) - and while yes it is on cheap commodity hardware, that's *by far* the least expensive part of the solution (people, coding, datacenter, power are much more and reoccurring) - there is much more to supporting, serving and keeping the data online than the hard drive costs.
So to be clear, this isn't a "it costs too much" issue, it's a "let's look at the data, see if it's even relevant to keep on disk and come up with a policy that lets us get to some predictable and maintainable storage solution".
-Justin
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning
sayrer
> Quite the opposite, I think it *is* the correct approach. While the data may not be huge now, having no policy, not putting any cost on storage growth, and not revisiting storing data which may have little to zero value is exactly what the operations and build teams should be doing. Saying "we'll just keep buying more disk" for something that we don't have a value or use for isn't responsible.
Look at the start of the thread. There were two options (A and B)
presented in order to achieve one-time space savings. The suggested
remedy for Option A would free up 161GB of space, while the suggested
remedy for option B would save 260GB of space. As an organization,
that is a quantity of data that we should be able to handle without
blinking. So, the thread should be about what to save going forward.
>
> That said, if we show value and reasons for keeping this data - I am all for it.
It is the output of our work, as I said. It is worth saving. I agree
that we should definitely delete things we don't need. Old release
builds, release branch nightly builds, and trunk nightlies are things
that we need.
> So to be clear, this isn't a "it costs too much" issue, it's a "let's look at the data, see if it's even relevant to keep on disk and come up with a policy that lets us get to some predictable and maintainable storage solution".
Sure, I don't see how one could argue with that statement. I just
think you should expect the required amount of storage to continue
rising. We should do what we can to keep that growth manageable, of
course.
- Rob
Chris Cooper
Hash: SHA1
On 10-05-13 10:46 AM, Chris Cooper wrote:
> * yes, we (Mozilla) will continue to try to grow the disk space on
> stage as we go forward (there is not necessarily a hard cap on
> space), but that doesn't absolve us of the responsibility of cleaning
> up after ourselves.
I've talked to IT and lack of disk space is a straw man. They have lots
of space to give us should we need it. My emphasis now becomes getting
the policies in place to curb accumulation to make that space last as
long as possible while maintaining the builds/tools developers need.
> * there is lots of space to be reclaimed if we remove "cruft" from
> old nightlies, where cruft is anything that is not an actual build,
> e.g. installers, xpis - let me gather the numbers for how much space
> this would save per product and report back
...and now, the report. Some products get full marks for already keeping
their archives clean, others...not so much. Here's a list of products
and the space we could recover by removing unused "cruft", which in this
case specifically refers to old (2009 and earlier) installer files
(linux and windows) and xpis:
camino: 20M
firefox: 512G
lightning: 2.2G
mozilla: 25G
seamonkey: 156G
sunbird: 36G
thunderbird: 165G
xulrunner: 30M
Other than camino and xulrunner, those are substantial (potential)
one-time space recovery wins.
I need to talk to KaiRo about whether it is acceptable to delete the
xpis for SeaMonkey since he was still producing xpis for Windows up
until the end of February this year.
Note that realistically I don't intend to delete *anything* before the
end of May to give everyone lots of time to weigh-in. I need to contact
lead developers from every product on the list above to make sure they
are on board with the plan. They should feel free to opt out of this
cleanup if they have valid reasons for needing to keep those builds around.
> * let me verify that we don't have any old l10n builds still lurking
> in the archives. We only need en-US for regression hunting.
There are only a few errant l10n builds kicking around in the nightly,
all from older releases (i.e. 3.0-era and earlier) when builds would end
up in the nightly dirs before being sync-ed over to the releases/
subdir. There's nothing to be gained in keeping these builds, provided
they were properly sync-ed at release time.
Based on the feedback I've received and the numbers above, I have
revised my policy proposals:
1) Keep all releases for all products online and available. There's no
need to remove them.
2) Keep all en-US nightly builds for all products online and available.
There's no need to remove them.
3) Delete nightly artifacts for all products that are not useful in
regression hunting. Specifically, this means deleting installer files
(linux and windows) and xpis from 2009 and earlier. This could represent
a one-time space recovery of almost 900GB. Individual products can opt
out of this cleanup with sufficient cause.
4) Automate the deletion of nightly MAR files older than one month. Only
the most recent MAR files are required. This would be done across all
products. (unchanged)
5) Delete builds from older candidates directories after official
release. This will reclaim up to 13G per build attempt per release. This
will be a manual process. (unchanged)
6) Automate the removal of nightly artifacts older than 6 months for all
products that are not useful in regression hunting.
Feedback is still appreciated.
cheers,
- --
coop
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkvtjS8ACgkQqdY6vdJz4M9uLACeLy863aEiXKkPVysHq55qTK35
lQ4Anj/VhZgSyIWeDaW1yDgGgpk3HVUF
=9mgh
-----END PGP SIGNATURE-----
sayrer
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 10-05-13 10:46 AM, Chris Cooper wrote:
>
> > * yes, we (Mozilla) will continue to try to grow the disk space on
> > stage as we go forward (there is not necessarily a hard cap on
> > space), but that doesn't absolve us of the responsibility of cleaning
> > up after ourselves.
>
> I've talked to IT and lack of disk space is a straw man. They have lots
> of space to give us should we need it. My emphasis now becomes getting
> the policies in place to curb accumulation to make that space last as
> long as possible while maintaining the builds/tools developers need.
This plan looks better. But I don't think we'll ever get a smooth,
predictable growth rate here.
The space will grow slowly over time, but it will also occasionally be
bursty in the event that we add a platform, start shipping more
specialized builds (e.g. SSE4-Windows), or go through a period where
we're maintaining more maintenance branches than we would like. That
means these disks should never get very close to full capacity, so
testers and developers don't run into HD space problems when they've
done things like increase the number of builds we support.
If we /really/ want to optimize this problem, we should look at a file
system that can maintain each nightly build as a delta from the last.
That would be take some time though, and probably isn't worth it.
- Rob
Smokey Ardisson
> I've talked to IT and lack of disk space is a straw man. They have lots
> of space to give us should we need it. My emphasis now becomes getting
> the policies in place to curb accumulation to make that space last as
> long as possible while maintaining the builds/tools developers need.
I look forward to not having to see another "OMG, need to clean up
stage; let's start deleting old nightlies! / No, don't take my
nightlies! Just clean up other cruft instead" discussion again in six
months (or ever again) :-)
> Here's a list of products
> and the space we could recover by removing unused "cruft", which in this
> case specifically refers to old (2009 and earlier) installer files
> (linux and windows) and xpis:
>
> camino: 20M
Since we don't have any win/lin installers or xpis (and since 20MB is
about a .dmg and a quarter, anyway), I am curious what you've
identified as cruft here ;-) I took another look through our
hierarchy and didn't see anything that looked like it matched those
criteria (or any of my own criteria for "cruft").
> Based on the feedback I've received and the numbers above, I have
> revised my policy proposals:
>
> 1) Keep all releases for all products online and available. There's no
> need to remove them.
>
> 2) Keep all en-US nightly builds for all products online and available.
> There's no need to remove them.
>
> 3) Delete nightly artifacts for all products that are not useful in
> regression hunting. Specifically, this means deleting installer files
> (linux and windows) and xpis from 2009 and earlier. This could represent
> a one-time space recovery of almost 900GB. Individual products can opt
> out of this cleanup with sufficient cause.
>
> 4) Automate the deletion of nightly MAR files older than one month. Only
> the most recent MAR files are required. This would be done across all
> products. (unchanged)
>
> 5) Delete builds from older candidates directories after official
> release. This will reclaim up to 13G per build attempt per release. This
> will be a manual process. (unchanged)
>
> 6) Automate the removal of nightly artifacts older than 6 months for all
> products that are not useful in regression hunting.
>
> Feedback is still appreciated.
One other thing I'd add (or, really, repeat from the bug) is that if,
for some reason (speed, mirroring, whatever), it's desirable to keep
stage/ftp.m.o down at a much smaller size, the old ftp.m.o/archive.m.o
split worked well back when, and I think it would work OK again now.
Keep the current year's nightlies on stage/ftp, but move older
nightlies to archive. Obviously, if keeping the size of stage/ftp
significantly smaller is not a goal, having every year's nightlies in
a single place like it is now is better ;-)
Smokey
Robert Kaiser
> I need to talk to KaiRo about whether it is acceptable to delete the
> xpis for SeaMonkey since he was still producing xpis for Windows up
> until the end of February this year.
Should be OK, we just had to run SeaMonkey 1.x nightlies until we did
EOL that branch, and we were hooked on the old installer and tinderbox
clients with all that, happily producing those XPIs that the old XPFE
installer used. I'm not even sure the stub installers that actually were
supposed to use them did work, as we didn't announce them much any more.
Now that we have closed down and EOLed that branch, I'm fully OK with
removing the installers for old nightlies and only keeping the zips and
tarballs around, if that's general policy for everyone else as well.
Robert Kaiser
Robert Kaiser
> 1) Keep all releases for all products online and available. There's no
> need to remove them.
Keeping releases/ untouched is important, yes.
> 2) Keep all en-US nightly builds for all products online and available.
> There's no need to remove them.
You mean that as in "one package per platform", right, given that the
next point is removing installers and files for stub-installers? :)
> 3) Delete nightly artifacts for all products that are not useful in
> regression hunting. Specifically, this means deleting installer files
> (linux and windows) and xpis from 2009 and earlier.
Note that this means that regression hunting for installers will get
harder. Not sure if people are doing that anyhow, it just came to my mind.
> 5) Delete builds from older candidates directories after official
> release. This will reclaim up to 13G per build attempt per release. This
> will be a manual process. (unchanged)
I have removed most old candidates directories for SeaMonkey this week,
thanks for the reminder.
Robert Kaiser
Gervase Markham
> Based on the feedback I've received and the numbers above, I have
> revised my policy proposals:
The new plan looks great :-)
Gerv
Chris Cooper
Hash: SHA1
On 10-05-15 1:51 AM, Smokey Ardisson wrote:
>> camino: 20M
>
> Since we don't have any win/lin installers or xpis (and since 20MB is
> about a .dmg and a quarter, anyway), I am curious what you've
> identified as cruft here ;-) I took another look through our
> hierarchy and didn't see anything that looked like it matched those
> criteria (or any of my own criteria for "cruft").
That's probably the sum total of directory sizes themselves since I
didn't exclude that from the reporting.
cheers,
- --
coop
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkv8IIEACgkQqdY6vdJz4M83ywCfZHJ0I6PXJAKZsEoriUt8Xuyy
4k4An1qqvo1zxLZo6MgRxiCyXXhx9fB+
=7ZDt
-----END PGP SIGNATURE-----
Chris Cooper
Hash: SHA1
On 10-05-14 1:49 PM, Chris Cooper wrote:
> Based on the feedback I've received and the numbers above, I have
> revised my policy proposals:
Thanks to everyone who weighed-in on this longstanding issue.
The cleanup crontabs for all products are in place now. I've run all the
individual commands by hand once and collected logs, but haven't seen
anything I didn't expect.
We didn't reclaim as much space as originally estimated because we
preserved the contrib dirs largely intact. We did still manage to
recover almost 350G of space across all products, bringing our %used
number on stage down to 79%. It was at 95% yesterday.
I've documented the new policy in the wiki:
https://wiki.mozilla.org/ReleaseEngineering:StageCleanupPolicy
We'll still have to grow the disk eventually, or shuffle stuff around,
but at least we have *something* in place policy-wise now.
cheers,
- --
coop
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkwZgw8ACgkQqdY6vdJz4M8vAQCcCnNeFZ1hGBQzhYHmXykEsag3
ekEAn23CX3FTc2Pkjw/qzmuQX5Hk6Sdw
=3UvK
-----END PGP SIGNATURE-----
Robert Strong
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 10-05-14 1:49 PM, Chris Cooper wrote:
>> Based on the feedback I've received and the numbers above, I have
>> revised my policy proposals:
> Thanks to everyone who weighed-in on this longstanding issue.
>
> The cleanup crontabs for all products are in place now. I've run all the
> individual commands by hand once and collected logs, but haven't seen
> anything I didn't expect.
>
> We didn't reclaim as much space as originally estimated because we
> preserved the contrib dirs largely intact. We did still manage to
> recover almost 350G of space across all products, bringing our %used
> number on stage down to 79%. It was at 95% yesterday.
>
> I've documented the new policy in the wiki:
> https://wiki.mozilla.org/ReleaseEngineering:StageCleanupPolicy
For the following:
"Delete nightly artifacts for all products that are not useful in
regression hunting. Specifically, this means deleting installer files
(linux and windows) and xpis from 2009 and earlier. Individual products
can opt out of this cleanup with sufficient cause."
could you specify what this will mean for future years instead of
specifying 2009?
Thanks,
Robert
Chris Cooper
Hash: SHA1
On 10-06-16 10:14 PM, Robert Strong wrote:
> For the following:
> "Delete nightly artifacts for all products that are not useful in
> regression hunting. Specifically, this means deleting installer files
> (linux and windows) and xpis from 2009 and earlier. Individual products
> can opt out of this cleanup with sufficient cause."
>
> could you specify what this will mean for future years instead of
> specifying 2009?
I've setup the various crontabs to expire the less-useful nightly
artifacts after 6 months. We could probably be more aggressive there,
but since we're almost 6 months into 2010 now, 6 months seemed like a
good threshold.
cheers,
- --
coop
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkwZhwAACgkQqdY6vdJz4M9hHwCgl+k1w4Z3robDe2sLm6+5vzH/
8iQAoIe3FBKFUVbls90MiG3+BFCKPkaE
=EnLm
-----END PGP SIGNATURE-----