archive.mozilla.org - Ongoing cleanup

[Sylvestre Ledru]

Hello,

As part of a cleanup effort, we are planning to remove 1.3PB of data (1328 TB) hosted on archive.mozilla.org.

In details, we are removing two kind of files:

• Old artifacts that we don’t need anymore:

Description	Pattern	Size (TB)
Old CI artifacts	pub/%/tinderbox-builds/	406
HG bundle	*.bundle	1.46
Some big unused files	('pub/mozilla/VMs/b2g.box', 'pub/mozilla/VMs/CentOS5-ReferencePlatform.tar.bz2')	0.017
Test packages (https://bugzilla.mozilla.org/show_bug.cgi?id=1827513)	*tests.tar.gz	19.3
Old try build	pub/firefox/try-builds/	0.02

• Adjusting our storage rules to keep files less longer

Description	Pattern	Size (TB)
Candidate builds on beta after one year but keep RC builds indefinitely. Do not purge any candidates for release and ESR	pub/%/candidates/%0b%' and 'pub/%/candidates/archived/%'	192
fx & tb Nightly l10n artifacts after 4 weeks but keep English, French, Hebrew & Japanese indefinitely for mozregression on locales	pub/%/nightly/%-l10n% and not 'pub/%/nightly/%-l10n%/%.{fr,jp,he}.%'	710

Goals of this cleanup:

• Decrease the cost of storage / networking

• Reduce complexity in managing files and data

• Reduce our carbon footprint

First, to make sure we are not going to be break anything, we have been moving files into 

https://archive.mozilla.org/to-be-deleted/pub/ 

On July 15th, we will remove files for real from the storage.

If we missed some files or we are going too far, please let us know!

Jacob & Sylvestre