Summary: ======== Here s a proposal to change the directory structure on ftp.m.o for new Firefox, Fennec and XULrunner builds going forward. To reduce disruption, existing builds would remain where they currently are, until they are aged off as usual.
This fixes an intermittent problem we hit with respins-of-nightly-builds, brings us one step closer to building cool regression-hunting tools, and streamlines RelEng automation as we consolidate Firefox+Fennec automation.
BIKESHED ALERT: There's lots of potential opinions here. To avoid infinite loops, please read this entire doc, and the discussions in the two bugs, before commenting. Also, I've cross-posted to a few groups, to make sure this is widely seen. However, please respond here in dev.planning, or if appropriate, in the related bugs: https://bugzilla.mozilla.org/show_bug.cgi?id=449607 https://bugzilla.mozilla.org/show_bug.cgi?id=487036
Why change? =========== 1) a common use case is when someone reports a problem with a buildID, and we want to find that specific build on ftp.m.o. The current process, of manually trying to find out approximately when the build was created, and then converting to epoch, or manually eyeballing the timestamps on files on ftp is inefficient. With this change, we would immediately be able to find that build. We could later build tools that directly link to the build on ftp.m.o.
2) Builds created with the same BuildID, for every OS, will be in the same directory. We already do this for nightly builds.
3) This full BuildID corresponds to the full BuildID in the txt file we already create alongside each build we post on ftp.m.o. For developers, this txt file also includes the changeset info. For example: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2010-08-29-04-... contains: 20100829040614 414ff9016349
4) This avoids using changesets for unique directory identifier. Changesets are unique, which is good. However, there are significant drawbacks: 4a) changesets do not sort sequentially, which makes it harder to do a binary divide on filesystem to find a regression. 4b) using changesets raises a different problem about how to handle respin-of-same-changeset. Using BuildID handles respins. However, using changesets would require an additional solution, like creating subdirs numbered build1, build2, or subdirs numbered by BuildIDs/timestamp. That seems even more complicated, and anyway still uses BuildIDs/timestamp info. Even for cases where we do not respin, we'd need to create this subdir anyway, to avoid having respin-logic need to move files (and break links that point to the old location). 4c) using changesets is usually advocated by people trying to figure out what changed between two specific builds. That is better resolved by https://bugzilla.mozilla.org/show_bug.cgi?id=487036 (see below).
5) This helps fix a set of interconnected bugs 431905 Change build process to generate consistent BuildIDs 449607 change dated dirs on ftp.m.o to use new longer BuildID 496549 relbranch names should have a finer resolution than 1 day 487036 write tool to read buildbot db for BuildID+changesets of nightlies, and then construct URL to feed to hg pushlog 538540 stop putting hour number in nightly directories 584178 list hourly tinderbox builds by changeset on ftp.mozilla.org
6) Semi-related, bug#570814 "Nightly builds should all use the same revision" was fixed recently, so now all the builds for the same night on the same branch get the same BuildID. This should further help tidy up the build directories on ftp.m.o.
7) If RelEng is asked to respin a nightly, and we do so within the same hour as the first nightly (rare but it has happened), the new nightly overwrites the old. Not great, and causes problems for people getting updates that needs manual RelEng repair work.
8) By using {OS} as a directory, it makes it easy to delete the directory and recreate as part of posting the files of the build. This fixes the recurring unhappiness whenever filenames change (like between beta) and causes problems for nightly.m.o.
9) This makes the structure for Firefox, Fennec and XULrunner builds consistent. This makes the structure for incremental builds and nightly builds consistent. This consistency allows RelEng to further streamline automation.
Open question: =============== While we are doing this change, it seems like a good time to also rename the directory "tinderbox-builds". We no longer using any tinderbox clients to build/test, and we are almost complete with the switchover from tinderbox-waterfall to TBPL, so this term no longer seems valid. I'm suggesting "continuous" or maybe "continuous-builds" as a better name to store all the incremental build-on-checkin work we do throughout the day.
(Alternatives already suggested that I'd prefer to avoid: "buildbot-builds" (in case we ever switch from buildbot), "builds" (too vague/overloaded), "depend_build" (what happens if we do a clobber in the day?) or "per_checkin_build" (what happens if we collapse build queues to have multiple checkins per build?). What alternatives can you come up with?)
Hope all that makes sense - there's a lot of background and details, so if I missing something, do let me know. Also if you have comments or concerns, please chime in.
Fuzzer jobs need to be able to find "the latest Mac32 build". Putting the build ID before the OS makes it harder to find the latest build for an OS.
Although I guess what I really want is "the Mac32 build for the most recent changeset for which a build is available", and groveling around FTP might not be the best way to find it ;)
I understand that you want the date representation in the directory name to match the contents of the .txt file. Better to change the .txt file to use the more readable representation than change the directory name to use the less readable one, I'd think.
> 1) a common use case is when someone reports a problem with a buildID, > and we want to find that specific build on ftp.m.o. The current process, > of manually trying to find out approximately when the build was created, > and then converting to epoch, or manually eyeballing the timestamps on > files on ftp is inefficient. With this change, we would immediately be > able to find that build. We could later build tools that directly link > to the build on ftp.m.o.
I want nightly users to include changeset IDs, not build IDs, with their bug reports.
Let's not assume that "people report problems with build IDs" will continue to be a problem, or that looking at .txt files on FTP is the best way to solve it.
> 2) Builds created with the same BuildID, for every OS, will be in the > same directory. We already do this for nightly builds.
I'm not convinced this is an advantage.
Most people are only interested in one OS at a time. With OS/buildID/ they would be able to bookmark OS/ and make a single click. With buildID/OS/ they have to make two clicks, or more if they're unlucky and pick a buildID that only has Win64 builds.
> 4) This avoids using changesets for unique directory identifier. > Changesets are unique, which is good. However, there are significant > drawbacks: > 4a) changesets do not sort sequentially, which makes it harder to do a > binary divide on filesystem to find a regression.
Binary dividing on filesystem names is a trap. This should not be a reason for preferring date-based directory names to changeset-based directory names.
* Binary dividing on continuous builds gets confused by respins and nightlies, which cause build ID order to not match the order of pushes. I've already seen this cause confusion.
* Release minibranches, merges, and backouts make history nonlinear.
* Binary dividing on nightlies is inefficient because some nightlies have more changes than others.
It will be easier to improve our regression-hunting tools if there's a way to get a list of changeset IDs for available builds. Using the changeset ID in the directory name is one way to do this.
On Tue, Jan 11, 2011 at 1:02 PM, Jesse Ruderman <jruder...@gmail.com> wrote: > Although I guess what I really want is "the Mac32 build for the most > recent changeset for which a build is available", and groveling around > FTP might not be the best way to find it ;)
You want pulse, or a pulse-based cache.
> I want nightly users to include changeset IDs, not build IDs, with > their bug reports.
Yes, this.
> Binary dividing on filesystem names is a trap. This should not be a > reason for preferring date-based directory names to changeset-based > directory names.
Changesets don't sort, though, so bisection becomes a lot more onerous.
> BIKESHED ALERT: There's lots of potential opinions here. To avoid > infinite loops, please read this entire doc, and the discussions in the > two bugs, before commenting. Also, I've cross-posted to a few groups, to > make sure this is widely seen. However, please respond here in > dev.planning, or if appropriate, in the related bugs: > https://bugzilla.mozilla.org/show_bug.cgi?id=449607 > https://bugzilla.mozilla.org/show_bug.cgi?id=487036
Ok, I did my homework.
It's a big bikeshed. Why not paint the outside one color, the inside another, and maybe make some striped patterns on the roof?
Specifically, how about creating multiple views out of symlinks? Use what you're proposing for the master, then create an {OS}-centric view in an OS/ subdir, and perhaps paint stripes by making changeset-based directories of symlinks under each. You can store your bike and eat it too!
So I have a question, actually. I have a script that downloads nightly builds so I can keep a local nightly archive. It does this by the stupid expedient of looping over all possible values of HH for a given YY-MM-DD and trying to look for the right filename in all 24 folders.
In the new setup, this isn't feasible, since I'd need to do 86400 network requests in the worst case, not 24. Is there a reasonable approach I can take to make this script work with the new directory structure?
> 3) This full BuildID corresponds to the full BuildID in the txt file we > already create alongside each build we post on ftp.m.o. For developers, > this txt file also includes the changeset info. For example: > http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2010-08-29-04-... > contains: > 20100829040614 414ff9016349
This txt file will now live in the 20100829040614 directory, right? So I can't just look for it to figure out the right HHMMSS for a given date?
Instead of trying all values until you find one that is valid, couldn't you download the directory listing from FTP/HTTP and parse it to figure out which values are valid and haven't already been grabbed? This has the advantage of getting you all copies of the nightly if there is more than one on any given day. The Python FTP module makes these directory listings pretty easy.
John
This script will list the nightly directory to stdout ===================================================== #!/usr/bin/python
from ftplib import FTP ftp=FTP('ftp.mozilla.org') ftp.login() ftp.cwd('pub/mozilla.org/firefox/nightly') ftp.retrlines('LIST') ftp.quit() =====================================================
The following command will get the list of directories from HTTP
-----Original Message----- From: Boris Zbarsky <bzbar...@mit.edu> To: dev-plann...@lists.mozilla.org Subject: Re: change to Fennec, Firefox & XULRunner directories on
ftp.m.o Date: Thu, 13 Jan 2011 12:27:08 -0500 Newsgroups: mozilla.dev.planning
On 1/11/11 2:22 PM, John O'Duinn wrote: > before: firefox/nightly/YYYY-MM-DD-HH-{branchname} > after: firefox/nightly/{branchname}/YYYYMMDDHHMMSS/{OS}
So I have a question, actually. I have a script that downloads nightly builds so I can keep a local nightly archive. It does this by the stupid expedient of looping over all possible values of HH for a given YY-MM-DD and trying to look for the right filename in all 24 folders.
In the new setup, this isn't feasible, since I'd need to do 86400 network requests in the worst case, not 24. Is there a reasonable approach I can take to make this script work with the new directory structure?
> 3) This full BuildID corresponds to the full BuildID in the txt file we > already create alongside each build we post on ftp.m.o. For developers, > this txt file also includes the changeset info. For example: > http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2010-08-29-04-... > contains: > 20100829040614 414ff9016349
This txt file will now live in the 20100829040614 directory, right? So I can't just look for it to figure out the right HHMMSS for a given date?
> Instead of trying all values until you find one that is valid, couldn't > you download the directory listing from FTP/HTTP and parse it to figure > out which values are valid and haven't already been grabbed?
I suppose I can try that; I just couldn't figure out a sane way to do this (in particular doing that for FTP from scratch is rocket science).
> This script will list the nightly directory to stdout
Thanks! I'll give this a shot.
> The following command will get the list of directories from HTTP
Also, I've been thinking about a general system that would solve this FWIW, as I need a lot of this data in a release management system I am bringing up.
