Standardizing on archive formats in CI

[Chris AtLee]

We currently use a mix of various archive formats in automation. For
example, we use:

* .tar.bz2 for linux builds
* .tar.gz for test archives
* .zip for jshell, crashreporter symbols, and mozharness.zip
* .dmg for macos builds
* .exe AND .zip for windows builds

Supporting this variety of different formats has several drawbacks:
* Complicates automation scripts that need to know what format to expect,
depending on the platform
* .tar.bz2 is slow to compress and decompress, and has poor compression
compared with modern compression methods
* .tar.gz is pretty fast, but we can also improve compression ratios with
modern methods
* .zip is similar to .tar.gz in terms of speed and compression ratio
* producing both .zips and .exes on Windows is a waste of time and space
* .dmgs are hard to work with on non-mac platforms

I'd like to propose a few changes for what archive formats we produce and
consume in CI:
1) Standardize on .tar.xz or .tar.zst for all build and test archives used
in CI. If this proposal has general support, I'd like to see a good
analysis of each of xz and zst so we can ideally pick one that works best
for our use cases.

2) Change our canonical linux packaging format to .tar.xz or .tar.zst.

3) Change Windows builds to produce only .tar.xz or .tar.zst in CI. We can
produce the installers from the tarballs in separate "repackage" tasks

4) Change the macOS builds to produce only .tar.xz or .tar.zst in CI. We
already are producing the .dmg files from tarballs in separate "repackage"
tasks

With these changes I hope we can simplify the code used to support
builds/tests in CI, and also achieve some cpu time and space improvements.

Thoughts?

Cheers,
Chris

[Axel Hecht]

Hi catlee,

responding here with the perspective of doing local (and automated) l10n
repacks.

How would this affect on what `./mach package` does, and how would it
affect l10n repacks? Both local and in automation?

Axel

Am 27.05.19 um 20:19 schrieb Chris AtLee:

[Geoffrey Brown]

Chris,

That sounds good to me.

You didn't mention Android explicitly; will Android be treated just like
Linux?

If you encounter a trade-off between time and space, how will you decide
which is "best"?

- Geoff

> _______________________________________________
> tools mailing list
> to...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/tools
>

[Justin Wood]

Does this including no longer shipping .zip for windows, or only for ci?

-Justin Wood (Callek)

> _______________________________________________
> release-engineering mailing list
> release-e...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/release-engineering
>

[Chris AtLee]

I'm not an expert on l10n by any means, so you and Callek in particular
probably have better ideas.

Naively, I would say that `./mach package` should always produce tar balls,
and that l10n repacks should consume and produce tarballs. Generation of
platform specific installers (.exe / .msi / .dmg) should be part of the
repackage tasks.

What do you think?

On Mon, 27 May 2019 at 14:43, Axel Hecht <ax...@mozilla.com> wrote:

> Hi catlee,
>
> responding here with the perspective of doing local (and automated) l10n
> repacks.
>
> How would this affect on what `./mach package` does, and how would it
> affect l10n repacks? Both local and in automation?
>
> Axel
>
> Am 27.05.19 um 20:19 schrieb Chris AtLee:

[Chris AtLee]

I wasn't sure what was best for Android. For testing, do we need to have
the builds in apk format? If so, then I don't think there's any point in
generating anything except APKs for Android.

I think we should heavily favour time over space. Storage is relatively
cheap compared to human or machine time waiting for archives to be produced
or extracted. Perhaps one caveat to this is anything that we end up
shipping to users. If we can spend more cpu time producing a smaller
archive for users, then I think we should.

In automation, perhaps our evaluation metric could be something like "m =
compression_time + N*(transfer_time + decompression_time)", where N is the
number of downstream tasks from a build. Further to this, maybe we should
assume a constant network speed, and then the only thing to compare would
be compression and decompression time on set of reference archives.

On Mon, 27 May 2019 at 15:51, Geoffrey Brown <gbr...@mozilla.com> wrote:

> Chris,
>
> That sounds good to me.
>
> You didn't mention Android explicitly; will Android be treated just like
> Linux?
>
> If you encounter a trade-off between time and space, how will you decide
> which is "best"?
>
> - Geoff
>
> On Mon, May 27, 2019 at 12:19 PM Chris AtLee <cat...@mozilla.com> wrote:
>

[Chris AtLee]

What are the use cases for shipping .zip files?

We already provide .exe and .msi files. You can use tools like 7zip to
extract the files from the .exe if you don't want to run it.

On Mon, 27 May 2019 at 20:51, Justin Wood <jw...@mozilla.com> wrote:

> Does this including no longer shipping .zip for windows, or only for ci?
>
> -Justin Wood (Callek)
>

[Justin Wood]

Comment inline..

On Wed, May 29, 2019 at 10:43 AM Chris AtLee <cat...@mozilla.com> wrote:

> I'm not an expert on l10n by any means, so you and Callek in particular
> probably have better ideas.
>
> Naively, I would say that `./mach package` should always produce tar balls,
> and that l10n repacks should consume and produce tarballs. Generation of
> platform specific installers (.exe / .msi / .dmg) should be part of the
> repackage tasks.
>
> What do you think?
>
>

I suspect we should treat it similar to builds. Specifically:

* In CI we only produce tarballs and repack based on tarballs.
* In Local Builds we do whatever a local build would do for en-US (e.g.
create an .exe for a user on windows)

I'm not sure how easily this concept translates to existing code offhand,
but I think that is the best high level of the what-should-we-do if we go
this route.

~Justin Wood (Callek)

> On Mon, 27 May 2019 at 14:43, Axel Hecht <ax...@mozilla.com> wrote:
>
> > Hi catlee,
> >
> > responding here with the perspective of doing local (and automated) l10n
> > repacks.
> >
> > How would this affect on what `./mach package` does, and how would it
> > affect l10n repacks? Both local and in automation?
> >
> > Axel
> >
> > Am 27.05.19 um 20:19 schrieb Chris AtLee:

[Justin Wood]

When I was younger I would stuff Firefox zip on a USB drive that I could
download on a public-but-locked-down computer, open and use Firefox, and
then move on....

I also have personally used, when testing on windows, .zip's as easy and
quick "download, unzip, run" cycles for things like regression hunting and
for testing a feature in an isolated environment after a first-run....

Even on linux I have downloaded windows .zip files for inspection of
archive contents (e.g. symbol stuff for xul.dll)

My own use case is not necessarily a good reflection of the users we care
about, and I personally rarely need .zip's and I do know how to use 7zip
and get the stuff I care about out, it just is more convenient for .zip.

~Justin Wood (Callek)

On Wed, May 29, 2019 at 10:54 AM Chris AtLee <cat...@mozilla.com> wrote:

> What are the use cases for shipping .zip files?
>
> We already provide .exe and .msi files. You can use tools like 7zip to
> extract the files from the .exe if you don't want to run it.
>
> On Mon, 27 May 2019 at 20:51, Justin Wood <jw...@mozilla.com> wrote:
>
>> Does this including no longer shipping .zip for windows, or only for ci?
>>
>> -Justin Wood (Callek)
>>
>> On Mon, May 27, 2019, 3:51 PM Geoffrey Brown <gbr...@mozilla.com> wrote:
>>
>>> Chris,
>>>
>>> That sounds good to me.
>>>
>>> You didn't mention Android explicitly; will Android be treated just like
>>> Linux?
>>>
>>> If you encounter a trade-off between time and space, how will you decide
>>> which is "best"?
>>>
>>> - Geoff
>>>
>>> On Mon, May 27, 2019 at 12:19 PM Chris AtLee <cat...@mozilla.com> wrote:
>>>

>>> > tools mailing list
>>> > to...@lists.mozilla.org
>>> > https://lists.mozilla.org/listinfo/tools
>>> >

[Geoffrey Brown]

Yes, Android builds need to be in apk format. Test archives should probably
use the same formats as desktop, or linux if there is a distinction.

On Wed, May 29, 2019 at 8:53 AM Chris AtLee <cat...@mozilla.com> wrote:

> I wasn't sure what was best for Android. For testing, do we need to have
> the builds in apk format? If so, then I don't think there's any point in
> generating anything except APKs for Android.
>
> I think we should heavily favour time over space. Storage is relatively
> cheap compared to human or machine time waiting for archives to be produced
> or extracted. Perhaps one caveat to this is anything that we end up
> shipping to users. If we can spend more cpu time producing a smaller
> archive for users, then I think we should.
>
> In automation, perhaps our evaluation metric could be something like "m =
> compression_time + N*(transfer_time + decompression_time)", where N is the
> number of downstream tasks from a build. Further to this, maybe we should
> assume a constant network speed, and then the only thing to compare would
> be compression and decompression time on set of reference archives.
>

[Axel Hecht]

Am 29.05.19 um 17:03 schrieb Justin Wood:

> Comment inline..
>
> On Wed, May 29, 2019 at 10:43 AM Chris AtLee <cat...@mozilla.com> wrote:
>
>> I'm not an expert on l10n by any means, so you and Callek in particular
>> probably have better ideas.
>>
>> Naively, I would say that `./mach package` should always produce tar balls,
>> and that l10n repacks should consume and produce tarballs. Generation of
>> platform specific installers (.exe / .msi / .dmg) should be part of the
>> repackage tasks.
>>
>> What do you think?
>>
>>
> I suspect we should treat it similar to builds. Specifically:
>
> * In CI we only produce tarballs and repack based on tarballs.
> * In Local Builds we do whatever a local build would do for en-US (e.g.
> create an .exe for a user on windows)

I'm wondering, ./mach build wget-en-US would continue to work the same
way? Also, artifact builds would continue to look for .exe/.dmg?

If that's the case, it's probably that ./mach package would continue to
create platform-dependent packages, and automation would override that
behavior to just create tar balls?

Axel

[Johan Lorenzo]

Re: APKs for Android.

I know we have some on-device tests. These require (signed - even with a
dummy key) APKs. I might be wrong, but I don't know any other way to
install an application without complying with Android's security model.

On Wed, May 29, 2019 at 4:53 PM Chris AtLee <cat...@mozilla.com> wrote:

> I wasn't sure what was best for Android. For testing, do we need to have
> the builds in apk format? If so, then I don't think there's any point in
> generating anything except APKs for Android.
>
> I think we should heavily favour time over space. Storage is relatively
> cheap compared to human or machine time waiting for archives to be produced
> or extracted. Perhaps one caveat to this is anything that we end up
> shipping to users. If we can spend more cpu time producing a smaller
> archive for users, then I think we should.
>
> In automation, perhaps our evaluation metric could be something like "m =
> compression_time + N*(transfer_time + decompression_time)", where N is the
> number of downstream tasks from a build. Further to this, maybe we should
> assume a constant network speed, and then the only thing to compare would
> be compression and decompression time on set of reference archives.
>
> On Mon, 27 May 2019 at 15:51, Geoffrey Brown <gbr...@mozilla.com> wrote:
>
> > Chris,
> >
> > That sounds good to me.
> >
> > You didn't mention Android explicitly; will Android be treated just like
> > Linux?
> >
> > If you encounter a trade-off between time and space, how will you decide
> > which is "best"?
> >
> > - Geoff
> >
> > On Mon, May 27, 2019 at 12:19 PM Chris AtLee <cat...@mozilla.com> wrote:
> >

> >> tools mailing list
> >> to...@lists.mozilla.org
> >> https://lists.mozilla.org/listinfo/tools
> >>
> >

[Botond Ballo]

On Wed, May 29, 2019 at 11:03 AM Justin Wood <jw...@mozilla.com> wrote:
> * In CI we only produce tarballs and repack based on tarballs.

One thing that may be worth keeping in mind, is that we sometimes ask
users to test Try builds. (For example, if a developer can't reproduce
a bug locally, they might ask the bug's reporter to try a speculative
fix pushed to Try, by downloading the relevant build artifact from
Treeherder.)

If the build artifact uses a compression format that's not supported
out of the box on the user's platform, that could introduce friction
into this process.

Cheers,
Botond

[Chris AtLee]

Could we address this by running the "repackage" tasks as required on Try,
either as part of the initial push, or added after via treeherder's "Add
New Jobs" feature?

[Axel Hecht]

Am 29.05.19 um 18:01 schrieb Chris AtLee:

> Could we address this by running the "repackage" tasks as required on Try,
> either as part of the initial push, or added after via treeherder's "Add
> New Jobs" feature

I wonder if we should give them a name that reflects their deliverable
"download and test this", more than it's purpose.

Or even just create a dummy task that doesn't do anything but echo the
artifact, under a human-readable task name?

Axel

[Chris AtLee]

To follow up on this, I've done some brief experimentation with different
compression methods and a variety of archives we use in automation. I've
uploaded the script I use here:
https://gist.github.com/catlee/80e4abae57d98180b755838ddbda6e04.

The results from my system are pasted below (lower scores are better). In
all cases zstd with the default setting is the preferred method. Are there
any other methods archives we should consider?

Cheers,
Chris

2019-05-30 11:50:55,333 - FITNESS for mozharness.zip:
2019-05-30 11:50:55,333 - bzip2 (default): 5.87
2019-05-30 11:50:55,333 - xz (max): 5.17
2019-05-30 11:50:55,333 - xz (default): 4.77
2019-05-30 11:50:55,333 - zstd (max): 2.44
2019-05-30 11:50:55,333 - gzip (max): 1.95
2019-05-30 11:50:55,333 - gzip (default): 1.78
2019-05-30 11:50:55,333 - zstd (default): 1.26
--
2019-05-30 11:59:43,806 - FITNESS for
firefox-69.0a1.en-US.win64.common.tests.tar.gz:
2019-05-30 11:59:43,806 - xz (max): 127.86
2019-05-30 11:59:43,806 - bzip2 (default): 89.52
2019-05-30 11:59:43,806 - xz (default): 51.55
2019-05-30 11:59:43,806 - zstd (max): 46.60
2019-05-30 11:59:43,806 - gzip (max): 40.55
2019-05-30 11:59:43,806 - gzip (default): 37.28
2019-05-30 11:59:43,806 - zstd (default): 28.78
--
2019-05-30 12:12:51,839 - FITNESS for
firefox-69.0a1.en-US.win64.installer.exe:
2019-05-30 12:12:51,839 - xz (max): 227.83
2019-05-30 12:12:51,839 - bzip2 (default): 173.62
2019-05-30 12:12:51,839 - xz (default): 119.74
2019-05-30 12:12:51,839 - gzip (max): 72.51
2019-05-30 12:12:51,839 - zstd (max): 71.87
2019-05-30 12:12:51,839 - gzip (default): 63.92
2019-05-30 12:12:51,839 - zstd (default): 37.57
--
2019-05-30 12:25:27,577 - FITNESS for
firefox-69.0a1.en-US.linux-x86_64.web-platform.tests.tar.gz:
2019-05-30 12:25:27,578 - xz (max): 206.53
2019-05-30 12:25:27,578 - bzip2 (default): 157.45
2019-05-30 12:25:27,578 - xz (default): 97.94
2019-05-30 12:25:27,578 - zstd (max): 74.08
2019-05-30 12:25:27,578 - gzip (max): 67.31
2019-05-30 12:25:27,578 - gzip (default): 64.80
2019-05-30 12:25:27,578 - zstd (default): 48.06
--
2019-05-30 12:52:52,874 - FITNESS for
firefox-69.0a1.en-US.mac.crashreporter-symbols.zip:
2019-05-30 12:52:52,874 - xz (max): 463.74
2019-05-30 12:52:52,874 - bzip2 (default): 140.90
2019-05-30 12:52:52,874 - xz (default): 114.19
2019-05-30 12:52:52,874 - zstd (max): 87.69
2019-05-30 12:52:52,874 - gzip (max): 62.37
2019-05-30 12:52:52,874 - gzip (default): 54.58
2019-05-30 12:52:52,874 - zstd (default): 33.59
--
2019-05-30 13:07:34,014 - FITNESS for firefox-69.0a1.en-US.mac.dmg:
2019-05-30 13:07:34,015 - xz (max): 241.71
2019-05-30 13:07:34,015 - bzip2 (default): 198.71
2019-05-30 13:07:34,015 - xz (default): 140.13
2019-05-30 13:07:34,015 - gzip (max): 84.63
2019-05-30 13:07:34,015 - zstd (max): 79.66
2019-05-30 13:07:34,015 - gzip (default): 73.84
2019-05-30 13:07:34,015 - zstd (default): 43.18
--
2019-05-30 13:21:15,153 - FITNESS for firefox-69.0a1.en-US.win64.zip:
2019-05-30 13:21:15,153 - xz (max): 236.04
2019-05-30 13:21:15,153 - bzip2 (default): 177.42
2019-05-30 13:21:15,154 - xz (default): 127.86
2019-05-30 13:21:15,154 - gzip (max): 76.46
2019-05-30 13:21:15,154 - zstd (max): 74.75
2019-05-30 13:21:15,154 - gzip (default): 63.98
2019-05-30 13:21:15,154 - zstd (default): 38.19
--
2019-05-30 13:36:42,315 - FITNESS for
firefox-69.0a1.en-US.linux-x86_64.tar.bz2:
2019-05-30 13:36:42,315 - xz (max): 247.77
2019-05-30 13:36:42,315 - bzip2 (default): 222.32
2019-05-30 13:36:42,315 - xz (default): 142.52
2019-05-30 13:36:42,315 - gzip (max): 88.78
2019-05-30 13:36:42,315 - zstd (max): 81.58
2019-05-30 13:36:42,315 - gzip (default): 70.58
2019-05-30 13:36:42,315 - zstd (default): 41.40

[Axel Hecht]

One caveat with zstd is that it's not supported in BSD tar, and in
gnu-tar only in >= 1.31, which was released this January.

So apart from the usual tar on mac, it doesn't seem to be on in stock
Ubuntu images yet, too.

Axel

Am 31.05.19 um 14:53 schrieb Chris AtLee:

[Chris AtLee]

It sounds like there's no serious opposition to this proposal, so let's
assume that we want to proceed with .tar.zst as our standard archive format.

I think the biggest barrier to making this happen is to ensure that we have
the zstd commandline tools and python module available everywhere in
automation.

Many developers won't have these tools available locally initially, so does
that mean we need to implement a fallback to .tar.gz perhaps? Put it behind
a configure flag? Require it via `mach bootstrap`?


On Mon, 3 Jun 2019 at 09:42, Ben Hearsum <bhea...@mozilla.com> wrote:

> My Ubuntu (19.04) appears to support zstd out of the box:
> ➜ tmp tar --help | grep zstd
> --zstd filter the archive through zstd

>
>
> On Sat, Jun 01, 2019 at 04:07:11PM +0200, Axel Hecht wrote:
> > One caveat with zstd is that it's not supported in BSD tar, and in
> gnu-tar
> > only in >= 1.31, which was released this January.
> >
> > So apart from the usual tar on mac, it doesn't seem to be on in stock
> Ubuntu
> > images yet, too.
> >
> > Axel
> >
> > Am 31.05.19 um 14:53 schrieb Chris AtLee:

> > --
> > You received this message because you are subscribed to the Google
> Groups "firefox-ci" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to firefox-ci+...@mozilla.com.
> > To post to this group, send email to firef...@mozilla.com.
> > To view this discussion on the web visit
> https://groups.google.com/a/mozilla.com/d/msgid/firefox-ci/5568dfd3-bbda-de79-d1a3-ad72218572fa%40mozilla.com
> .
>