change to Fennec, Firefox & XULRunner directories on ftp.m.o

16 views
Skip to first unread message

John O'Duinn

unread,
Jan 11, 2011, 2:22:00 PM1/11/11
to dev. planning, dev-tree-management
Summary:
========
Here�s a proposal to change the directory structure on ftp.m.o for new
Firefox, Fennec and XULrunner builds going forward. To reduce
disruption, existing builds would remain where they currently are, until
they are aged off as usual.

This fixes an intermittent problem we hit with
respins-of-nightly-builds, brings us one step closer to building cool
regression-hunting tools, and streamlines RelEng automation as we
consolidate Firefox+Fennec automation.


BIKESHED ALERT: There's lots of potential opinions here. To avoid
infinite loops, please read this entire doc, and the discussions in the
two bugs, before commenting. Also, I've cross-posted to a few groups, to
make sure this is widely seen. However, please respond here in
dev.planning, or if appropriate, in the related bugs:
https://bugzilla.mozilla.org/show_bug.cgi?id=449607
https://bugzilla.mozilla.org/show_bug.cgi?id=487036

Details:
========
On ftp.m.o, this proposal would only change files under
http://ftp.mozilla.org/pub/mozilla.org/firefox/,
http://ftp.mozilla.org/pub/mozilla.org/xulrunner/ and
http://ftp.mozilla.org/pub/mozilla.org/mobile/. Some concrete examples
would be helpful:

before: firefox/tinderbox-builds/{branchname}-{OS}/{seconds-since-epoch}/
after: firefox/tinderbox-builds/{branchname}/{YYYYMMDDHHMMSS}/{OS}

before: firefox/nightly/YYYY-MM-DD-HH-{branchname}
after: firefox/nightly/{branchname}/YYYYMMDDHHMMSS/{OS}

before: mobile/tinderbox-builds/{branchname}-{OS}/{seconds-since-epoch}/
after: mobile/tinderbox-builds/{branchname}/{YYYYMMDDHHMMSS}/{OS}

before: mobile/nightly/YYYY-MM-DD-HH-{branchname}
after: mobile/nightly/{branchname}/YYYYMMDDHHMMSS/{OS}

As an example, this would change from:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1283011618/
...to:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central/20100828160658/linux

...and change from:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2011-01-03-03-mozilla-central/firefox-4.0b9pre.en-US.win32.zip
...to:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/20110103035959-mozilla-central/win32/firefox-4.0b9pre.en-US.win32.zip


Why change?
===========
1) a common use case is when someone reports a problem with a buildID,
and we want to find that specific build on ftp.m.o. The current process,
of manually trying to find out approximately when the build was created,
and then converting to epoch, or manually eyeballing the timestamps on
files on ftp is inefficient. With this change, we would immediately be
able to find that build. We could later build tools that directly link
to the build on ftp.m.o.

2) Builds created with the same BuildID, for every OS, will be in the
same directory. We already do this for nightly builds.

3) This full BuildID corresponds to the full BuildID in the txt file we
already create alongside each build we post on ftp.m.o. For developers,
this txt file also includes the changeset info. For example:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2010-08-29-04-mozilla-central/firefox-4.0b5pre.en-US.win32.txt
contains:
20100829040614 414ff9016349

4) This avoids using changesets for unique directory identifier.
Changesets are unique, which is good. However, there are significant
drawbacks:
4a) changesets do not sort sequentially, which makes it harder to do a
binary divide on filesystem to find a regression.
4b) using changesets raises a different problem about how to handle
respin-of-same-changeset. Using BuildID handles respins. However, using
changesets would require an additional solution, like creating subdirs
numbered build1, build2, or subdirs numbered by BuildIDs/timestamp. That
seems even more complicated, and anyway still uses
BuildIDs/timestamp info. Even for cases where we do not respin, we'd
need to create this subdir anyway, to avoid having respin-logic need to
move files (and break links that point to the old location).
4c) using changesets is usually advocated by people trying to figure out
what changed between two specific builds. That is better resolved by
https://bugzilla.mozilla.org/show_bug.cgi?id=487036 (see below).

5) This helps fix a set of interconnected bugs
431905 Change build process to generate consistent BuildIDs
449607 change dated dirs on ftp.m.o to use new longer BuildID
496549 relbranch names should have a finer resolution than 1 day
487036 write tool to read buildbot db for BuildID+changesets of
nightlies, and then construct URL to feed to hg pushlog
538540 stop putting hour number in nightly directories
584178 list hourly tinderbox builds by changeset on ftp.mozilla.org

6) Semi-related, bug#570814 "Nightly builds should all use the same
revision" was fixed recently, so now all the builds for the same night
on the same branch get the same BuildID. This should further help tidy
up the build directories on ftp.m.o.

7) If RelEng is asked to respin a nightly, and we do so within the same
hour as the first nightly (rare but it has happened), the new nightly
overwrites the old. Not great, and causes problems for people getting
updates that needs manual RelEng repair work.

8) By using {OS} as a directory, it makes it easy to delete the
directory and recreate as part of posting the files of the build. This
fixes the recurring unhappiness whenever filenames change (like between
beta) and causes problems for nightly.m.o.

9) This makes the structure for Firefox, Fennec and XULrunner builds
consistent. This makes the structure for incremental builds and nightly
builds consistent. This consistency allows RelEng to further streamline
automation.


Open question:
===============
While we are doing this change, it seems like a good time to also rename
the directory "tinderbox-builds". We no longer using any tinderbox
clients to build/test, and we are almost complete with the switchover
from tinderbox-waterfall to TBPL, so this term no longer seems valid.
I'm suggesting "continuous" or maybe "continuous-builds" as a better
name to store all the incremental build-on-checkin work we do throughout
the day.

(Alternatives already suggested that I'd prefer to avoid:
"buildbot-builds" (in case we ever switch from buildbot), "builds" (too
vague/overloaded), "depend_build" (what happens if we do a clobber in
the day?) or "per_checkin_build" (what happens if we collapse build
queues to have multiple checkins per build?). What alternatives can you
come up with?)


Hope all that makes sense - there's a lot of background and details, so
if I missing something, do let me know. Also if you have comments or
concerns, please chime in.


thanks
John.

Jesse Ruderman

unread,
Jan 11, 2011, 4:02:15 PM1/11/11
to jod...@mozilla.com, dev. planning
> before: firefox/tinderbox-builds/{branchname}-{OS}/{seconds-since-epoch}/
> after: firefox/tinderbox-builds/{branchname}/{YYYYMMDDHHMMSS}/{OS}

Fuzzer jobs need to be able to find "the latest Mac32 build". Putting
the build ID before the OS makes it harder to find the latest build
for an OS.

Although I guess what I really want is "the Mac32 build for the most
recent changeset for which a build is available", and groveling around
FTP might not be the best way to find it ;)

> before: firefox/nightly/YYYY-MM-DD-HH-{branchname}
> after: firefox/nightly/{branchname}/YYYYMMDDHHMMSS/{OS}

I'd prefer keeping the dashes in the date.

I understand that you want the date representation in the directory
name to match the contents of the .txt file. Better to change the .txt
file to use the more readable representation than change the directory
name to use the less readable one, I'd think.

> 1) a common use case is when someone reports a problem with a buildID,
> and we want to find that specific build on ftp.m.o. The current process,
> of manually trying to find out approximately when the build was created,
> and then converting to epoch, or manually eyeballing the timestamps on
> files on ftp is inefficient. With this change, we would immediately be
> able to find that build. We could later build tools that directly link
> to the build on ftp.m.o.

I want nightly users to include changeset IDs, not build IDs, with
their bug reports.

Let's not assume that "people report problems with build IDs" will
continue to be a problem, or that looking at .txt files on FTP is the
best way to solve it.

> 2) Builds created with the same BuildID, for every OS, will be in the
> same directory. We already do this for nightly builds.

I'm not convinced this is an advantage.

Most people are only interested in one OS at a time. With OS/buildID/
they would be able to bookmark OS/ and make a single click. With
buildID/OS/ they have to make two clicks, or more if they're unlucky
and pick a buildID that only has Win64 builds.

> 4) This avoids using changesets for unique directory identifier.
> Changesets are unique, which is good. However, there are significant
> drawbacks:
> 4a) changesets do not sort sequentially, which makes it harder to do a
> binary divide on filesystem to find a regression.

Binary dividing on filesystem names is a trap. This should not be a
reason for preferring date-based directory names to changeset-based
directory names.

* Binary dividing on continuous builds gets confused by respins and
nightlies, which cause build ID order to not match the order of
pushes. I've already seen this cause confusion.

* Release minibranches, merges, and backouts make history nonlinear.

* Binary dividing on nightlies is inefficient because some nightlies
have more changes than others.

It will be easier to improve our regression-hunting tools if there's a
way to get a list of changeset IDs for available builds. Using the
changeset ID in the directory name is one way to do this.

Mike Shaver

unread,
Jan 11, 2011, 4:10:12 PM1/11/11
to Jesse Ruderman, dev. planning, jod...@mozilla.com
On Tue, Jan 11, 2011 at 1:02 PM, Jesse Ruderman <jrud...@gmail.com> wrote:
> Although I guess what I really want is "the Mac32 build for the most
> recent changeset for which a build is available", and groveling around
> FTP might not be the best way to find it ;)

You want pulse, or a pulse-based cache.

> I want nightly users to include changeset IDs, not build IDs, with
> their bug reports.

Yes, this.

> Binary dividing on filesystem names is a trap. This should not be a
> reason for preferring date-based directory names to changeset-based
> directory names.

Changesets don't sort, though, so bisection becomes a lot more onerous.

Mike

Steve Fink

unread,
Jan 11, 2011, 4:37:16 PM1/11/11
to dev-pl...@lists.mozilla.org
On 01/11/2011 11:22 AM, John O'Duinn wrote:
> BIKESHED ALERT: There's lots of potential opinions here. To avoid
> infinite loops, please read this entire doc, and the discussions in the
> two bugs, before commenting. Also, I've cross-posted to a few groups, to
> make sure this is widely seen. However, please respond here in
> dev.planning, or if appropriate, in the related bugs:
> https://bugzilla.mozilla.org/show_bug.cgi?id=449607
> https://bugzilla.mozilla.org/show_bug.cgi?id=487036
Ok, I did my homework.

It's a big bikeshed. Why not paint the outside one color, the inside
another, and maybe make some striped patterns on the roof?

Specifically, how about creating multiple views out of symlinks? Use
what you're proposing for the master, then create an {OS}-centric view
in an OS/ subdir, and perhaps paint stripes by making changeset-based
directories of symlinks under each. You can store your bike and eat it too!

John O'Duinn

unread,
Jan 11, 2011, 5:47:08 PM1/11/11
to dev. planning
hi;

On 1/11/11 11:22 AM, John O'Duinn wrote:
> Summary:
> ========
...


> before: firefox/nightly/YYYY-MM-DD-HH-{branchname}
> after: firefox/nightly/{branchname}/YYYYMMDDHHMMSS/{OS}

...

Sorry, there was a typo in the last example; it *should* read:

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/mozilla-central/20110103035959/win32/firefox-4.0b9pre.en-US.win32.zip

The template, and other examples, in the post were correct, this one
typo slipped through all the proof-reading by me and others. :-(


Sorry for any confusion.
John.

Boris Zbarsky

unread,
Jan 13, 2011, 12:27:08 PM1/13/11
to
On 1/11/11 2:22 PM, John O'Duinn wrote:
> before: firefox/nightly/YYYY-MM-DD-HH-{branchname}
> after: firefox/nightly/{branchname}/YYYYMMDDHHMMSS/{OS}

So I have a question, actually. I have a script that downloads nightly
builds so I can keep a local nightly archive. It does this by the
stupid expedient of looping over all possible values of HH for a given
YY-MM-DD and trying to look for the right filename in all 24 folders.

In the new setup, this isn't feasible, since I'd need to do 86400
network requests in the worst case, not 24. Is there a reasonable
approach I can take to make this script work with the new directory
structure?

> 3) This full BuildID corresponds to the full BuildID in the txt file we
> already create alongside each build we post on ftp.m.o. For developers,
> this txt file also includes the changeset info. For example:
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2010-08-29-04-mozilla-central/firefox-4.0b5pre.en-US.win32.txt
> contains:
> 20100829040614 414ff9016349

This txt file will now live in the 20100829040614 directory, right? So
I can't just look for it to figure out the right HHMMSS for a given date?

-Boris


John F

unread,
Jan 13, 2011, 12:52:12 PM1/13/11
to Boris Zbarsky, dev-pl...@lists.mozilla.org
Boris,

Instead of trying all values until you find one that is valid, couldn't
you download the directory listing from FTP/HTTP and parse it to figure
out which values are valid and haven't already been grabbed? This has
the advantage of getting you all copies of the nightly if there is more
than one on any given day. The Python FTP module makes these directory
listings pretty easy.

John

This script will list the nightly directory to stdout
=====================================================
#!/usr/bin/python

from ftplib import FTP
ftp=FTP('ftp.mozilla.org')
ftp.login()
ftp.cwd('pub/mozilla.org/firefox/nightly')
ftp.retrlines('LIST')
ftp.quit()
=====================================================

The following command will get the list of directories from HTTP

curl http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/ | grep
"\[DIR\]" | sed "s/^.*href=\"//" | sed "s,\">.*$,,"

-Boris


_______________________________________________
dev-planning mailing list
dev-pl...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-planning


Boris Zbarsky

unread,
Jan 13, 2011, 12:57:55 PM1/13/11
to
On 1/13/11 12:52 PM, John F wrote:
> Instead of trying all values until you find one that is valid, couldn't
> you download the directory listing from FTP/HTTP and parse it to figure
> out which values are valid and haven't already been grabbed?

I suppose I can try that; I just couldn't figure out a sane way to do
this (in particular doing that for FTP from scratch is rocket science).

> This script will list the nightly directory to stdout

Thanks! I'll give this a shot.

> The following command will get the list of directories from HTTP
>
> curl http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/ | grep
> "\[DIR\]" | sed "s/^.*href=\"//" | sed "s,\">.*$,,"

That depends on implemenentation details that I'd really rather not have
to depend on....

-Boris

Boris Zbarsky

unread,
Jan 13, 2011, 1:02:49 PM1/13/11
to
On 1/13/11 12:52 PM, John F wrote:
> This script will list the nightly directory to stdout

Ah, this is just the entire dir listing. It doesn't do the hard
"extract the filename" part.... So not as useful as I'd hoped. :(

-Boris

Christian Legnitto

unread,
Jan 13, 2011, 1:10:22 PM1/13/11
to Boris Zbarsky, dev-pl...@lists.mozilla.org

Christian Legnitto

unread,
Jan 13, 2011, 1:25:26 PM1/13/11
to Christian Legnitto, Boris Zbarsky, dev-pl...@lists.mozilla.org
Also, I've been thinking about a general system that would solve this FWIW, as I need a lot of this data in a release management system I am bringing up.

I wrote initial thoughts / spec at https://bugzilla.mozilla.org/show_bug.cgi?id=625417

Thanks,
Christian


On Jan 13, 2011, at 10:10 AM, Christian Legnitto wrote:

> You might want to take a look at http://hg.mozilla.org/users/clegnitto_mozilla.com/pulseshims/file/default/ftp_shim.py#l308
>
> Thanks,
> Christian
>
> On Jan 13, 2011, at 10:02 AM, Boris Zbarsky wrote:
>

Reply all
Reply to author
Forward
0 new messages