How do we sign-off on Mercurial based releases' code+locales?

9 views
Skip to first unread message

Ben Hearsum

unread,
Aug 7, 2008, 2:54:45 PM8/7/08
to
Hi all,

With work ramping on HG release automation we need to decide what and
how we will sign-off on for Mercurial based releases.

In CVS, we had two repositories to deal with (/cvsroot and /l10n). For
each of these RelEng was given an official timestamp to pull. We
copied and pasted these to a configuration file for our automation.

In Mercurial, we have mozilla-central plus one more repository for
*each* locale that we ship (for Mobile reasons, we will have mobile-
browser, too). For a few reason we *cannot* use timestamps as a sign-
off in Mercurial. For these, we will need to use revision numbers. For
Firefox builds, we (RelEng) will need to receive a revision number for
mozilla-central, plus one for each locale we are shipping. Obviously
this is going to end up being a lot of different numbers. To avoid us
having to copy-paste these around (and possibly make a mistake in
doing so) I'd like to propose that QA/Dev check-in this information to
a yet-to-be-determined location. We would have our automation
automatically read these in to avoid the chance of a copy/paste, or
other human error.

If someone has a better idea, I'm totally open to it.

Thoughts/comments?

- Ben

Gavin Sharp

unread,
Aug 7, 2008, 2:59:55 PM8/7/08
to Ben Hearsum, dev-pl...@lists.mozilla.org
On Thu, Aug 7, 2008 at 11:54 AM, Ben Hearsum <bhea...@mozilla.com> wrote:
> In Mercurial, we have mozilla-central plus one more repository for
> *each* locale that we ship (for Mobile reasons, we will have mobile-
> browser, too). For a few reason we *cannot* use timestamps as a sign-
> off in Mercurial. For these, we will need to use revision numbers. For
> Firefox builds, we (RelEng) will need to receive a revision number for
> mozilla-central, plus one for each locale we are shipping. Obviously
> this is going to end up being a lot of different numbers. To avoid us
> having to copy-paste these around (and possibly make a mistake in
> doing so) I'd like to propose that QA/Dev check-in this information to
> a yet-to-be-determined location. We would have our automation
> automatically read these in to avoid the chance of a copy/paste, or
> other human error.

We could use tags for this, right? Release drivers could tag each repo
(m-c and all l10n repos) with a SHIRETOKO_3_0a1_SIGNOFF1 tag or
something? I guess that still means that there's no central list of
revisions for a given release, but I'm not sure that that's a problem.

Gavin

Ben Hearsum

unread,
Aug 7, 2008, 3:11:12 PM8/7/08
to Gavin Sharp, dev-pl...@lists.mozilla.org

This would work from an automation standpoint. It does feel important to
have a record of what revisions we built from. Tags can move to a new
revision BUT, Mercurial does create a changeset for each tag - so I
think we would have a record of tag movement.

Ben Hearsum

unread,
Aug 7, 2008, 3:11:12 PM8/7/08
to Gavin Sharp, dev-pl...@lists.mozilla.org

This would work from an automation standpoint. It does feel important to

Mike Beltzner

unread,
Aug 7, 2008, 3:34:48 PM8/7/08
to Ben Hearsum, dev-pl...@lists.mozilla.org, Gavin Sharp

This sounds promising. So the requirements on dev/l10n would be to
have each repository tagged with CODENAME_version_SIGNOFF_n for a
build? Can anyone move those tags?

Could we also build simple tools to show the changes which have been
merged *since* a given tag? That information was pretty important when
we were confirming with each other that the timestamp had caught the
latest changes required for a build.

Axel: will this work with your l10n dashboard? Presumably you could
use the tag as the root for the tests that are run in that dashboard,
thus being able to verify at a glance what the results of any l10n
build would be. The localizers could move the tags around as a way of
signing off on their locale.

cheers,
mike

Benjamin Smedberg

unread,
Aug 7, 2008, 3:52:21 PM8/7/08
to
Mike Beltzner wrote:

> This sounds promising. So the requirements on dev/l10n would be to have
> each repository tagged with CODENAME_version_SIGNOFF_n for a build? Can
> anyone move those tags?

Anyone can commit a new revision number for those tags. However, changes to
tags are versioned in Mercurial, so you'd know if a tag was changed by
looking at the commits. Why would you ever move the tag?

> Could we also build simple tools to show the changes which have been
> merged *since* a given tag? That information was pretty important when
> we were confirming with each other that the timestamp had caught the
> latest changes required for a build.

I'm not sure what this means. You mean, what changes happened "after" the
specified tag? That's fairly easy to get from hg: `hg log
-rCODENAME_version_SIGNOFF-n:default`

--BDS

Rob Helmer

unread,
Aug 7, 2008, 7:38:42 PM8/7/08
to
On Aug 7, 12:11 pm, Ben Hearsum <bhear...@mozilla.com> wrote:
> Gavin Sharp wrote:

This is similar but much better than the current CVS situation in
fact, where every file has it's own revision number and anyone can
move a tag without any record of that happening :) (malicious or
accidental moving of the tag could be detected by looking at the
source tarball that's generated as part of the release automation,
though).

The way it works for CVS is that the following tags are applied to all
applicable modules in the l10n repo, and mozilla module of the main
repo:

PRODUCT_x_y_z_RELEASE
PRODUCT_x_y_z_RCn

A timestamp/branch combination for each module is given to the release
team, and the tag is applied by the automated system. A source tarball
is immediately created from the _RELEASE tag (second step after
tagging, in parallel with en-US builds).

If more than one RC is required, the _RELEASE tag is moved a new _RCn
tag is applied. I think the only reason the _RELEASE tag is moved is
to create the source tarball, which is in fact a tarred-up anonymous
CVS checkout. This is hacky and always made me unhappy, and
fortunately I think not really at all applicable to the new hg
world... moving tags should not be necessary unless something has gone
wrong.

So, the thing missing in this thread is, who applies the tag, given a
set of revisions? This has historically been the build/release team,
but theoretically each repository could be tagged by whoever is in
charge there, before the release build process starts.. I think that'd
be better, though it's pretty different from how it's been done up to
this point.

Rob Helmer

unread,
Aug 7, 2008, 7:47:54 PM8/7/08
to
On Aug 7, 4:38 pm, Rob Helmer <rhel...@gmail.com> wrote:
> So, the thing missing in this thread is, who applies the tag, given a
> set of revisions? This has historically been the build/release team,
> but theoretically each repository could be tagged by whoever is in
> charge there, before the release build process starts.. I think that'd
> be better, though it's pretty different from how it's been done up to
> this point.

Er, sorry brainfreeze on that first sentence; that's exactly what is
being proposed by Gavin, and sounds reasonable :)

Does this mean that each l10n team tags their own repo, or do they
report a revision to the release driver? Former seems better..

Axel Hecht

unread,
Aug 8, 2008, 4:45:16 AM8/8/08
to

That doesn't look fairly easy when you'd try it for 60 repos, really.

I wonder if pushlog can help here. Getting the tip changeset for a
particular timestamp sounds like something that's fairly easy to do.

Here's a rough idea:

For a given a timestamp:

For all locales in all-locales (or shipped-locales?), ask pushlogs for
the tip for that timestamp, and all changesets after that. Aggregate
those by push time and send it out to a webpage.

The confirm the timestamp (if you want fancy, do a timeline app), and
get all changeset IDs that go up to that timestamp. Adjust individual
locales as necessary.

I don't see a good chance in getting this done by the localizers, but if
ted would agree to dump yet another feature (*) into pushlog, it sounds
like a descent job for a web app.

So far, I haven't gotten my dashboard up and running for the hg builds,
so I can't currently estimate if it's going to be of any help. It might
turn out to be good, but I don't know.


For the general archival for this data, I wouldn't be too sad if we
ended up with not having to store the locales information in a file
that's part of the en-US tagged-sources and tarball, we struggle with a
chicken and egg problem there. So having that info elsewhere would be
cool, something like an extended shipped-locales file in a build repo?

Axel

(*) That'd be "tip at timestamp and all following pushes". Slightly
different than "pushes between dates". We could make a different hook,
if preferred, too, it's just that the pushlog.db seems like the best
place for me to get that data.

Axel Hecht

unread,
Aug 8, 2008, 4:47:18 AM8/8/08
to

You want to make our release builds depend on 60+ non-techies tagging
their tree right? I don't see that working, in particular not with
stable minor releases.

More in my other reply, which thunderbird and I agree I sent last night,
but that apparently didn't make it.

Axel

Ben Hearsum

unread,
Aug 8, 2008, 8:12:23 AM8/8/08
to Benjamin Smedberg
Benjamin Smedberg wrote:
> Mike Beltzner wrote:
>
>> This sounds promising. So the requirements on dev/l10n would be to
>> have each repository tagged with CODENAME_version_SIGNOFF_n for a
>> build? Can anyone move those tags?
>
> Anyone can commit a new revision number for those tags. However, changes
> to tags are versioned in Mercurial, so you'd know if a tag was changed
> by looking at the commits. Why would you ever move the tag?
>

fwiw,
We tagged, eg, FIREFOX_3_1a1_RELEASE when we started 3.1a1build1. When
we respun for build2 we moved the _RELEASE tag to the appropriate place.

Ben Hearsum

unread,
Aug 8, 2008, 8:16:47 AM8/8/08
to Axel Hecht


Are you suggesting that this be done by l10n/dev, or by RelEng as part
of release automation?

This isn't something I feel we should rely on in automation. Automation
should be told _exactly_ what revision (by a changeset id or tag) to pull.

I'm totally not opposed to this for you guys collecting a list of
changesets, though.

Ben Hearsum

unread,
Aug 8, 2008, 8:19:13 AM8/8/08
to

I think it's important to have the _RELEASE tag to make it absolutely
clear what revision a given release was built from. Perhaps this is
something we could do as part of final release prep, though, to minimize
the chance of needing to move the tag.

Ben Hearsum

unread,
Aug 8, 2008, 8:20:40 AM8/8/08
to

Yes, and yes.

Axel Hecht

unread,
Aug 8, 2008, 10:09:13 AM8/8/08
to


Picking the right revisions for each shipping l10n repo is going to be
part of the release process.

Reality bites, there's going to be some manual tweaking involved to
separate out changes that we need to ship from those that we must not
ship, and we have tools available that can help us there.

One tool is that we can make this decision per locale.

Another tool is that we have pushlog, which gives us information which
locale got hit with which check-in.

Putting the two together should yield something that's operatable by
more than one individual, and that's one key to not fail miserably.

Who is doing the actual code sign-off for which release is something
that's part of the release team decisions for that release, and not a
tool question.

Axel

Justin Wood (Callek)

unread,
Aug 8, 2008, 2:18:16 PM8/8/08
to

IMO, if there is a script that gathers these revisions as suggested, I'd
still push for tags being applied to the code for those revisions.

RelEng could then either use the specific revision list or pull from
those tags.

And this would allow actual [human] users of the repo's to easily grab
(update to) certain releases as well, in order to better test for
regressions, use the tag as a jumping point in hg bisect, etc.

--
~Justin Wood (Callek)

Benjamin Smedberg

unread,
Aug 8, 2008, 3:12:38 PM8/8/08
to

Sure, although I no longer think that this is necessary, and we could apply
the _RELEASE tag only when we actually release. I was talking specifically
about the SIGNOFF_n tag; if for some reason you did a signoff and then
needed to take a new changeset, why not just use SIGNOFF_n+1 ?

--BDS

Rob Helmer

unread,
Aug 8, 2008, 3:22:49 PM8/8/08
to
On Aug 8, 1:47 am, Axel Hecht <l...@mozilla.com> wrote:
> Rob Helmer wrote:
> > On Aug 7, 4:38 pm, Rob Helmer <rhel...@gmail.com> wrote:
> >> So, the thing missing in this thread is, who applies the tag, given a
> >> set of revisions? This has historically been the build/release team,
> >> but theoretically each repository could be tagged by whoever is in
> >> charge there, before the release build process starts.. I think that'd
> >> be better, though it's pretty different from how it's been done up to
> >> this point.
>
> > Er, sorry brainfreeze on that first sentence; that's exactly what is
> > being proposed by Gavin, and sounds reasonable :)
>
> > Does this mean that each l10n team tags their own repo, or do they
> > report a revision to the release driver? Former seems better..
>
> You want to make our release builds depend on 60+ non-techies tagging
> their tree right? I don't see that working, in particular not with
> stable minor releases.


As I don't really have a stake in it, just making a suggestion :)
Seemed to me like that work could be distributed, although I agree
it's probably not ideal. You know the process better than I, maybe
it's easier for someone to scoop up revisions from Tinderbox output
and apply a tag to each?

It should be fine of course, just seems like a lot of paperwork.
Collecting all of this information upfront before starting any release
builds seems like a lot of wasted time and energy, lists to create,
typos to insert, etc.

What makes the most sense to me is to have production builds created
every time a checkin is done, and all of this information stored just
like for a nightly build (revision, time it was built, result, etc.
etc.). The release driver could come after-the-fact and apply the tag
to the correct revisions (would be better to have an automated system
do this, I think). See how releases work with Hudson, for instance.

Has anyone looked at how other large product releases do this? I know
that Java uses hg forests, for example.. Sun might have some war
stories.

Ben Hearsum

unread,
Aug 8, 2008, 3:23:45 PM8/8/08
to Justin Wood (Callek)

We're definitely not talking about getting rid of tags on l10n or code
repositories. What's in question here is how we know what revision to tag.

Ben Hearsum

unread,
Aug 8, 2008, 3:25:33 PM8/8/08
to Benjamin Smedberg

Ah. The unfortunate thing about that is that we'd have to track separate
tags for each repository. If we do end up going with a "signoff" tag I'd
be really nice to just tell automation "pull $PRODUCT_$VERSION_SIGNOFF"
from every repository, rather than pulling a possibly different tag from
each

Axel Hecht

unread,
Aug 11, 2008, 3:21:20 AM8/11/08
to
Rob Helmer wrote:
> On Aug 8, 1:47 am, Axel Hecht <l...@mozilla.com> wrote:
>> Rob Helmer wrote:
>>> On Aug 7, 4:38 pm, Rob Helmer <rhel...@gmail.com> wrote:
>>>> So, the thing missing in this thread is, who applies the tag, given a
>>>> set of revisions? This has historically been the build/release team,
>>>> but theoretically each repository could be tagged by whoever is in
>>>> charge there, before the release build process starts.. I think that'd
>>>> be better, though it's pretty different from how it's been done up to
>>>> this point.
>>> Er, sorry brainfreeze on that first sentence; that's exactly what is
>>> being proposed by Gavin, and sounds reasonable :)
>>> Does this mean that each l10n team tags their own repo, or do they
>>> report a revision to the release driver? Former seems better..
>> You want to make our release builds depend on 60+ non-techies tagging
>> their tree right? I don't see that working, in particular not with
>> stable minor releases.
>
>
> As I don't really have a stake in it, just making a suggestion :)
> Seemed to me like that work could be distributed, although I agree
> it's probably not ideal. You know the process better than I, maybe
> it's easier for someone to scoop up revisions from Tinderbox output
> and apply a tag to each?

Not that we know exactly what tinderbox will produce yet, but there's
more information going into the release decision that just successful
builds.

> It should be fine of course, just seems like a lot of paperwork.
> Collecting all of this information upfront before starting any release
> builds seems like a lot of wasted time and energy, lists to create,
> typos to insert, etc.

As shaver said, computers are good at that. No idea yet on how to make
something like this generate tagged output, but here's a static-for-now
ad-hoc attempt to get all pushes to l10n into a display,
http://l10n.mozilla.org/~axel/tmp-pushes/. Not that such thing would
need to export, pushlog has the data we need, it just needs to expose it.

> What makes the most sense to me is to have production builds created
> every time a checkin is done, and all of this information stored just
> like for a nightly build (revision, time it was built, result, etc.
> etc.). The release driver could come after-the-fact and apply the tag
> to the correct revisions (would be better to have an automated system
> do this, I think). See how releases work with Hudson, for instance.

I don't get this one. l10n release bits without en-US release bits?

> Has anyone looked at how other large product releases do this? I know
> that Java uses hg forests, for example.. Sun might have some war
> stories.
>

joduinn would be the one with connections, I suppose :-)

Axel

Mike Beltzner

unread,
Aug 11, 2008, 10:11:14 AM8/11/08
to Axel Hecht, dev-pl...@lists.mozilla.org
On 11-Aug-08, at 3:21 AM, Axel Hecht wrote:
>> What makes the most sense to me is to have production builds created
>> every time a checkin is done, and all of this information stored just
>> like for a nightly build (revision, time it was built, result, etc.
>> etc.). The release driver could come after-the-fact and apply the tag
>> to the correct revisions (would be better to have an automated system
>> do this, I think). See how releases work with Hudson, for instance.
>
> I don't get this one. l10n release bits without en-US release bits?

I might be putting words in someone else's mouth here, but I was
figuring that since the localizers already do their own checkins (with
your approval, natch) and since they know the state of their builds,
they'd be the one to put a signoff tag on their repos when they're in
the state needed for a milestone. Then you can check against those
tags and give your final approval through email, which would mean that
you assert the tags are in the same place.

So put you in the position of reviewer and approver, and give the
localizers responsibility of adding the tag. Ben tells me that it's
not hard to do.

cheers,
mike

Axel Hecht

unread,
Aug 11, 2008, 1:08:13 PM8/11/08
to

So instead of communicating one deadline and handling a handful of
exceptions I'd be reviewing some 60+ individual tagging processes? Oh,
and teaching them how to do that.

Let's not add 5 complex tasks for something that shouldn't be all that hard.

Related challenge, is there any feeling in this group about whether we'd
need sign-offs for Beta builds, or if green builds would be good enough?
(Plus search OK-ishness et al)

Axel

Mike Beltzner

unread,
Aug 11, 2008, 1:09:01 PM8/11/08
to Axel Hecht, dev-pl...@lists.mozilla.org
On 11-Aug-08, at 1:08 PM, Axel Hecht wrote:

> So instead of communicating one deadline and handling a handful of
> exceptions I'd be reviewing some 60+ individual tagging processes? Oh,
> and teaching them how to do that.

Perhaps this is naive of me (likely!) but I was envisioning:

- you continue to review l10n patches for entry into the tree
- you continue to communicate a deadline
- localizers communicate their adherence to a deadline by tagging
with the signoff tag
- your tools check for the presence of the signoff tag in addition
to the stuff they already check for
- you build shipped-locales based on the number of signoff tags &&
green lights you have

Teaching them how to do it, I'm told, is tantamount to teaching them
how to merge, but with slightly different syntax.

The alternate that I see is that you have to do all the tagging, which
I guess is more or less the same, except that it's a bunch of extra
work for you.

> Related challenge, is there any feeling in this group about whether
> we'd
> need sign-offs for Beta builds, or if green builds would be good
> enough?
> (Plus search OK-ishness et al)

I'd prefer to have some signal of "I'm done" from the localization
teams.

cheers,
mike

Axel Hecht

unread,
Aug 11, 2008, 1:32:01 PM8/11/08
to
Mike Beltzner wrote:
> On 11-Aug-08, at 1:08 PM, Axel Hecht wrote:
>
>> So instead of communicating one deadline and handling a handful of
>> exceptions I'd be reviewing some 60+ individual tagging processes? Oh,
>> and teaching them how to do that.
>
> Perhaps this is naive of me (likely!) but I was envisioning:
>
> - you continue to review l10n patches for entry into the tree

I don't until we freeze l10n, which is not before RC.

> - you continue to communicate a deadline
> - localizers communicate their adherence to a deadline by tagging with
> the signoff tag

Tagging their local tip, some revision, their last push? For something
that localizers hardly touched yet, I see too many variables. Yes, all
those should be the same thing if everything went well, but l10n-drivers
have a rather significant error margin in our planning with hg.

> - your tools check for the presence of the signoff tag in addition to
> the stuff they already check for

What do I do if there's no tag, or the tag isn't even close to tip or
something like that?

> - you build shipped-locales based on the number of signoff tags &&
> green lights you have
>
> Teaching them how to do it, I'm told, is tantamount to teaching them how
> to merge, but with slightly different syntax.

I'll need to teach them merging nevertheless, as we will continue to
land some patches for them. Also, as long as they do "hg as in cvs",
merging isn't as much of an issue.

> The alternate that I see is that you have to do all the tagging, which I
> guess is more or less the same, except that it's a bunch of extra work
> for you.

Which extra work? So far, I only see more chances for errors (and I've
seen weird ones on cvs already), with me doing the exactly same thing
after that. I'd probably end up writing a python script that reads
shipped-locales and calls into the hg api.

>> Related challenge, is there any feeling in this group about whether we'd
>> need sign-offs for Beta builds, or if green builds would be good enough?
>> (Plus search OK-ishness et al)
>
> I'd prefer to have some signal of "I'm done" from the localization teams.

OK.

Axel

Ben Hearsum

unread,
Aug 11, 2008, 2:13:33 PM8/11/08
to
Axel Hecht wrote:

> Mike Beltzner wrote:
>> - you continue to communicate a deadline
>> - localizers communicate their adherence to a deadline by tagging
>> with the signoff tag
>
> Tagging their local tip, some revision, their last push? For something
> that localizers hardly touched yet, I see too many variables. Yes, all
> those should be the same thing if everything went well, but l10n-drivers
> have a rather significant error margin in our planning with hg.
>

I don't understand what the difficulty is here. As I see it, localizers
don't even need to *know* the revision # to tag. When they've done their
last check-in for release X they 'hg tag ...' and push the tag back,
along with their final commit(s). Is there something about the process
I'm missing here?

If that's too complicated we could give the localizers a script called
'signoff.sh' that (maybe pulls) tags and pushes back.

I feel very strongly that Release Engineering not be responsible for
choosing which changesets to build (through automation or otherwise).
l10n is responsible for signing off on pull dates in CVS. What we're
talking about here is the equivalent for Mercurial. I don't think that
responsibility should move elsewhere because it is more difficult. I am
happy to help out here where I can, though.

Rob Helmer

unread,
Aug 11, 2008, 2:55:44 PM8/11/08
to

What I am trying to say is that there is no reason to have revisions
or tagging be part of the sign-off process. If you want to for some
reason that's fine, but I don't think the release process has to be
gated on any of this, and it'd save a lot of time (both wall-clock and
human).

Just wanted to clarify this point - why do you think l10n release bits
wouldn't have en-US bits? Ideally l10n builds should depend on en-US
build, no? And an en-US build should trigger rebuild of all locales.

What I am proposing is that en-US builds should be done on checkin
(which would presumably generate l10n repacks for all locales), and
individual locale builds should be rebuilt on checkin (against the
latest en-US). The whole release process could run, revisions for each
stored, and tags applied after-the-fact.

Right now the first step in the release process is to apply a tag,
based on a signed-off revision; that step is kicked off manually. I
don't think that needs to be the case, is all I'm saying.

That way, release drivers would be signing off on applying a tag to
source that produced the builds that QA has signed off on, instead of
doing all this work before build or QA can start.

Mike Beltzner

unread,
Aug 11, 2008, 3:25:19 PM8/11/08
to dev-pl...@lists.mozilla.org
On 11-Aug-08, at 2:54 PM, flod (Francesco Lodolo) wrote:

>> Is there something about the process I'm missing here?
>>

> Yes, that most of the localizers are not developers: it's already
> difficult to deal with versioning systems, don't add complexity to an
> already intricate procedure (have you ever tried to localize a Mozilla
> product? Try, it'll be fun ;-) ).

Thanks understood, Francesco, but as Ben has mentioned, tagging isn't
a lot of extra complexity, and scripts can be generated to make it
even easier than what needed to be done with CVS.

I think a discussion of better tools to make it easier to localize
products is important, but separate from this discussion.

My question to you would be: do you want to be the one who signals
"our localization is ready for this milestone" or do you think that
should be someone else? Previously, was it you who did that?

cheers,
mike

Mike Shaver

unread,
Aug 11, 2008, 3:26:13 PM8/11/08
to dev-pl...@lists.mozilla.org
On Mon, Aug 11, 2008 at 2:54 PM, flod (Francesco Lodolo)
<li...@lodolo.it> wrote:
>>
>> Is there something about the process I'm missing here?
>>
> Yes, that most of the localizers are not developers: it's already
> difficult to deal with versioning systems, don't add complexity to an
> already intricate procedure (have you ever tried to localize a Mozilla
> product? Try, it'll be fun ;-) ).

Yes, and to that end I think we should provide a script so that
localizers can run "./sign-off-locale en-CA 3.1b3" and have it do all
the right things for them, including yelling if there are unpushed
commits or uncommitted changes, or if the locale doesn't match their
pull, or if the version number isn't a known one, etc.

Mike

Axel Hecht

unread,
Aug 11, 2008, 4:40:38 PM8/11/08
to

Can you put that into a webapp behind ldap auth? I'd be game with that.

Shell scripts won't cut it, at least I'm not going to put my name down
for reliably shipping software based on that. We had loads of lengthy
discussions about tools and somesuch at the summit, and that'd just be a
large step backwards.

Axel

Mike Shaver

unread,
Aug 11, 2008, 5:04:22 PM8/11/08
to Axel Hecht, dev-pl...@lists.mozilla.org
On Mon, Aug 11, 2008 at 4:40 PM, Axel Hecht <l1...@mozilla.com> wrote:
> Can you put that into a webapp behind ldap auth? I'd be game with that.
>
> Shell scripts won't cut it, at least I'm not going to put my name down
> for reliably shipping software based on that. We had loads of lengthy
> discussions about tools and somesuch at the summit, and that'd just be a
> large step backwards.

Well, I think that we want the sign-off tool to be part of whatever
the localizers use to produce the builds they test and submit their
localization files. If they're using a web tool for that, then the
web-tool should let them say "this build I just tested is good, please
mark it for 3.1b3"; if they're committing and pushing to hg from the
command-line, I think we want the sign-off tool to be right there next
to the tree they were working in, so that it can catch "built with
these files but haven't committed them" or "committed but didn't
push".

We have yet to _ever_ reliably ship a piece of software without
something like a shell script being at the center of it, so I'm not
sure I understand that objection. If you'd feel better with it being
in python, sure, I don't care; I just thought it would be easier to
manipulate hg from a shell script. I doubt the localizers care what
it's implemented in, as long as it's well-tested and gives good error
messages. (Talking people through diagnosing their problems with a
shell script can actually be easier than with a python or other
program, IME, because of things like "sh -x".)

My point is really just that we should capture in software what the
steps are to declare something ship-ready, along with every bit of
sanity checking that we can imagine.

Mike

flod (Francesco Lodolo)

unread,
Aug 11, 2008, 7:12:23 PM8/11/08
to dev-pl...@lists.mozilla.org
>
> My question to you would be: do you want to be the one who signals
> "our localization is ready for this milestone" or do you think that
> should be someone else? Previously, was it you who did that?
Axel can be more detailed on this subject but, at least for my
experience with Firefox 3, we've always reasoned in terms of time, not
code versions.

For example:

1. tree is going to be closed for Firefox 3.1b1 on date X
2. before that limit we know that we have to update our localization,
go green (langpack builds) and check nightly builds (based upon
the current version of the locale in CVS) to ensure that localized
builds work as expected (with optional or mandatory litmus tests)
3. we opt-in for Firefox 3.1b1 using the l10n mailing list and
answering the specific opt-in thread

As far as I understand, you want to replace step 3 with tagging the
revision for the release. This seems an unnecessary step (at least from
my point of view), since on date X you just need to pick up the current
version (tip?) of the locale (and maybe tag this files if you need). If
I commit changes it's because they improve or fix my localization,
there's no reason to have a newer version and don't ship it.

Other concerns regard things like managing minor releases (do localizers
have to tag each minor release?), or what happen if I tag a version and
I need to make small updates before shipping.

Francesco

Asrail

unread,
Aug 11, 2008, 10:29:18 PM8/11/08
to
flod (Francesco Lodolo), 11-08-2008 20:12:

There are some limitations of CVS that could be avoided on hg and that
could be quite useful to the localizers, but roughly 95% of them
wouldn't require or be able to use that.

As Francesco said, usually if you have to make a change between a
deadline and the release, that's to fix something (even if that was
found by some user and doesn't depend on any change on en-US strings).

If someone can/want to make something different, he can use another
branch or a private repository for that.

Assuming that the changes after the freeze have to be reviewed, it would
be interesting to have the changes made on the head (master) and to use
a branch/tag to track the changes that will be made into the release.

Alternatively, since it's distributed, the release could pull from
another repositories that pull selectively the changes from the main
l10n repos.
Well... one pull would be quite similar to the tagging process and
someone reviewing the changes after a freeze would make the last
changeset before the release to be the right one to use - so it could be
tagged.

A tag on the repository would be quite similar if the localizer was told
to not push into the branch/tag.

That's easy for the ones that doesn't want to change and the ones that
know how to use hg could use their own branches/tags or their own - but
with 'stable' development on head (master) near releases.

John O'Duinn

unread,
Aug 11, 2008, 11:09:40 PM8/11/08
to Ben Hearsum, dev-pl...@lists.mozilla.org
hi;

Yikes - what a long thread! After reading *all* of the posts, I've a
comment.


On cvs, we (RelEng) rely on official signoff from Axel/Mic/Beltzner of
the exact code to tag. On hg, I think we should still have official
signoff of each locale, from the people who know the locales best, so
that RelEng know we are tagging exactly the right thing.


What if we tracked the changeset info for each locale in a file checked
into hg?

Instead of having only one person checkin to this file, each localizer
could add their locale and the "signoff" changeset for their locale to
the one central file, hence avoiding Axel-overload(!).

We (RelEng) would parse this file, extract the repo+changeset info and
use automation to tag all the repos listed using their specific
changeset. In the case of a locale respin, RelEng would simply re-parse
and re-tag based on the updated contents of the central file.

Right now, an enhanced shipped-locales file seems like a good location
for this, imho, but I'm open to alternatives. I also dont mind if the
file lives in mozilla-central repo or in buildtools repo or elsewhere;
I'm trying to avoid bikeshedding. :-)

(This avoids some concerns mentioned earlier about each localizer
applying SIGNOFF tags to their l10n repo, and dealing with retagging for
respins, etc. It also avoids concerns about RelEng programatically
trying to determine changesets from timestamps.)


Does that suggestion seem workable?


tc
John.
====


Ben Hearsum wrote:
> Hi all,
>
> With work ramping on HG release automation we need to decide what and
> how we will sign-off on for Mercurial based releases.
>
> In CVS, we had two repositories to deal with (/cvsroot and /l10n). For
> each of these RelEng was given an official timestamp to pull. We
> copied and pasted these to a configuration file for our automation.


>
> In Mercurial, we have mozilla-central plus one more repository for
> *each* locale that we ship (for Mobile reasons, we will have mobile-
> browser, too). For a few reason we *cannot* use timestamps as a sign-
> off in Mercurial. For these, we will need to use revision numbers. For
> Firefox builds, we (RelEng) will need to receive a revision number for
> mozilla-central, plus one for each locale we are shipping. Obviously
> this is going to end up being a lot of different numbers. To avoid us
> having to copy-paste these around (and possibly make a mistake in
> doing so) I'd like to propose that QA/Dev check-in this information to
> a yet-to-be-determined location. We would have our automation
> automatically read these in to avoid the chance of a copy/paste, or
> other human error.
>

> If someone has a better idea, I'm totally open to it.
>
> Thoughts/comments?
>
> - Ben
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

Jesper Kristensen

unread,
Aug 12, 2008, 4:47:25 AM8/12/08
to
Mike Shaver skrev:

> Well, I think that we want the sign-off tool to be part of whatever
> the localizers use to produce the builds they test and submit their
> localization files.

Then I guess this tool should be a webtool at tinderbox or ftp.

I think (but am not sure) that many localizers use a procedure like:

1: Update locale using favorite tool
2: Commit/push changes using command line or that favorite tool
3: Wait for tinderbox to build
4: Download build from ftp and test it

Justin Wood (Callek)

unread,
Aug 12, 2008, 8:14:55 AM8/12/08
to
Two things come to mind from this 'central file' suggestion.

If placed in m-c all localizers will need Hg access to m-c (which
currently is not the case, iirc).

Secondly, it begs the question of merges, can we expect/require all
localizers to understand merging well enough not to break someone else
when they are changing that file and someone else touches it?

Lastly, if in m-c do we want to burden most localizers with a full m-c
pull just to manage their locale?

--
~Justin Wood (Callek)

Ben Hearsum

unread,
Aug 12, 2008, 8:16:24 AM8/12/08
to Justin Wood (Callek)
Justin Wood (Callek) wrote:
> Two things come to mind from this 'central file' suggestion.
>
> If placed in m-c all localizers will need Hg access to m-c (which
> currently is not the case, iirc).
>
> Secondly, it begs the question of merges, can we expect/require all
> localizers to understand merging well enough not to break someone else
> when they are changing that file and someone else touches it?
>
> Lastly, if in m-c do we want to burden most localizers with a full m-c
> pull just to manage their locale?
>

Well, I think this theoretical file would end up living in a
hg.m.o/build/ repo, so a full m-c clone wouldn't be necessary.

With that said, my impression from Axel is that this is something we do
not want to put on localizers. Besides that, we've had better/easier
ideas suggested, such as a sign-off tag.

Ben Hearsum

unread,
Aug 12, 2008, 8:27:51 AM8/12/08
to
flod (Francesco Lodolo) wrote:
>>
>> My question to you would be: do you want to be the one who signals
>> "our localization is ready for this milestone" or do you think that
>> should be someone else? Previously, was it you who did that?
> Axel can be more detailed on this subject but, at least for my
> experience with Firefox 3, we've always reasoned in terms of time, not
> code versions.
>
> For example:
>
> 1. tree is going to be closed for Firefox 3.1b1 on date X
> 2. before that limit we know that we have to update our localization,
> go green (langpack builds) and check nightly builds (based upon
> the current version of the locale in CVS) to ensure that localized
> builds work as expected (with optional or mandatory litmus tests)
> 3. we opt-in for Firefox 3.1b1 using the l10n mailing list and
> answering the specific opt-in thread
>
> As far as I understand, you want to replace step 3 with tagging the
> revision for the release. This seems an unnecessary step (at least from
> my point of view), since on date X you just need to pick up the current
> version (tip?) of the locale (and maybe tag this files if you need). If
> I commit changes it's because they improve or fix my localization,
> there's no reason to have a newer version and don't ship it.
>

The problem here is that it Mercurial does not support "give me the
latest version on or before date XXX" properly. Additionally, Mercurial
has a repository wide revision number for each change. IMHO this is a
*much* more valuable piece of information to have. We've had mistakes
before where daylight savings time has caused us to build the wrong
code, or we've been off-by-one-second and built the wrong code. Having a
tag that says "build this code for release X" helps in both of the cases.

> Other concerns regard things like managing minor releases (do localizers
> have to tag each minor release?), or what happen if I tag a version and
> I need to make small updates before shipping.
>

For the latter case, all you need to do in move the tag to the new revision.

With regard to minor releases, that's a question...I certainly don't
think it would be great to make localizers re-tag code that hasn't
changed. We could probably have a separate script that is told which
locales haven't changed, and adds a new tag for the current release
appropriately.

Axel, what do you think?

Ben Hearsum

unread,
Aug 12, 2008, 8:31:25 AM8/12/08
to

Given that, can we all agree to the following:
1) locales being shipped for a release will have a
$PRODUCT_$VERSION_SIGNOFF tag in their repository
2) creation of these tags is the responsibility of l10n/localizers
(details TBD)

If we agree on this, we can move on and talk about the who and how of
implementing such a tool.

Justin Wood (Callek)

unread,
Aug 12, 2008, 8:42:16 AM8/12/08
to

If we want to automatically tag for locales whose minor release hasn't
changed at all, would a pushlog parser work here, or do we *only* care
about changes that affect Firefox rather than SeaMonkey (to use an example).

--
~Justin Wood (Callek)

Axel Hecht

unread,
Aug 12, 2008, 9:15:29 AM8/12/08
to

That's not going to work for stable releases. Those need to fly without
any l10n involvement at all.

Axel

Axel Hecht

unread,
Aug 12, 2008, 9:19:09 AM8/12/08
to

As I have mentioned before, pushlog.db does know about the changeset for
a particular timestamp, it's merely a small thing to actually expose
that knowledge externally.

>> Other concerns regard things like managing minor releases (do
>> localizers have to tag each minor release?), or what happen if I tag a
>> version and I need to make small updates before shipping.
>>
>
> For the latter case, all you need to do in move the tag to the new
> revision.
>
> With regard to minor releases, that's a question...I certainly don't
> think it would be great to make localizers re-tag code that hasn't
> changed. We could probably have a separate script that is told which
> locales haven't changed, and adds a new tag for the current release
> appropriately.
>
> Axel, what do you think?

I think we have all the information to convert a timestamp into a series
of changesets, and that we should use that instead of distributing
technical detail work.

Axel

Ben Hearsum

unread,
Aug 12, 2008, 9:30:42 AM8/12/08
to
Axel Hecht wrote:

> Ben Hearsum wrote:
>> The problem here is that it Mercurial does not support "give me the
>> latest version on or before date XXX" properly. Additionally,
>> Mercurial has a repository wide revision number for each change. IMHO
>> this is a *much* more valuable piece of information to have. We've had
>> mistakes before where daylight savings time has caused us to build the
>> wrong code, or we've been off-by-one-second and built the wrong code.
>> Having a tag that says "build this code for release X" helps in both
>> of the cases.
>
> As I have mentioned before, pushlog.db does know about the changeset for
> a particular timestamp, it's merely a small thing to actually expose
> that knowledge externally.
>> With regard to minor releases, that's a question...I certainly don't
>> think it would be great to make localizers re-tag code that hasn't
>> changed. We could probably have a separate script that is told which
>> locales haven't changed, and adds a new tag for the current release
>> appropriately.
>>
>> Axel, what do you think?
>
> I think we have all the information to convert a timestamp into a series
> of changesets, and that we should use that instead of distributing
> technical detail work.
>

I think this is perfectly fine as a way for l10n to generate sign-off
tags or collect revisions. I do not support this as part of the Release
Automation that RelEng runs after getting the "go".

Do we agree on this (ignoring implementation details)?

Ben Hearsum

unread,
Aug 12, 2008, 9:34:58 AM8/12/08
to
Axel Hecht wrote:
> That's not going to work for stable releases. Those need to fly without
> any l10n involvement at all.
>

I think what you're saying is "stable releases without a new locale need
to be do-able without l10n involvement. Is that correct? We can script
that, 'while (! tag for this version exists): try previous version'

John O'Duinn

unread,
Aug 12, 2008, 12:10:03 PM8/12/08
to Ben Hearsum, dev-pl...@lists.mozilla.org

Ben Hearsum wrote:
> Justin Wood (Callek) wrote:
>> Two things come to mind from this 'central file' suggestion.
>>
>> If placed in m-c all localizers will need Hg access to m-c (which
>> currently is not the case, iirc).
>>
>> Secondly, it begs the question of merges, can we expect/require all
>> localizers to understand merging well enough not to break someone else
>> when they are changing that file and someone else touches it?
>>
>> Lastly, if in m-c do we want to burden most localizers with a full m-c
>> pull just to manage their locale?
>
> Well, I think this theoretical file would end up living in a
> hg.m.o/build/ repo, so a full m-c clone wouldn't be necessary.

Absolutely correct, the file would likely not be in m-c, for exactly the
reasons Justin pointed out.

The main point here was: "have localizers store their changeset in a
file" as an alternative to "have localizers tag their repo".


>
> With that said, my impression from Axel is that this is something we do
> not want to put on localizers. Besides that, we've had better/easier
> ideas suggested, such as a sign-off tag.

From some of the other comments in this thread, it wasnt clear to me
that having localizers do tagging was something they were happy with.
However, checkins

tc
John.
=====

John O'Duinn

unread,
Aug 12, 2008, 12:10:56 PM8/12/08
to Ben Hearsum, dev-pl...@lists.mozilla.org

Ben Hearsum wrote:
> Justin Wood (Callek) wrote:
>> Two things come to mind from this 'central file' suggestion.
>>
>> If placed in m-c all localizers will need Hg access to m-c (which
>> currently is not the case, iirc).
>>
>> Secondly, it begs the question of merges, can we expect/require all
>> localizers to understand merging well enough not to break someone else
>> when they are changing that file and someone else touches it?
>>
>> Lastly, if in m-c do we want to burden most localizers with a full m-c
>> pull just to manage their locale?
>
> Well, I think this theoretical file would end up living in a
> hg.m.o/build/ repo, so a full m-c clone wouldn't be necessary.

Absolutely correct, the file would likely not be in m-c, for exactly the
reasons Justin pointed out.

The main point here was: "have localizers store their changeset in a
file" as an alternative to "have localizers tag their repo".


>

> With that said, my impression from Axel is that this is something we do
> not want to put on localizers. Besides that, we've had better/easier
> ideas suggested, such as a sign-off tag.

From some of the other comments in this thread, it wasnt clear to me

that having localizers do tagging was something they were happy with.

However, checkins and merges are "routine", hence safer under release
pressure...

tc
John.
=====

Ben Hearsum

unread,
Aug 13, 2008, 2:13:20 PM8/13/08
to
There was a lengthy discussion about this in #developers today. Out of
that, I propose the following:
* Release Automation will depend on a file that lists all of the locales
being shipped for a release and the changeset to pull for each of them.
This may or may not be shipped-locales (TBD).

* We (releng) will write a script that collects the current tip-most
changeset of a repository to generate the aforementioned file. This will
be run by l10n after all localizers are done checking in for a release.

Do we have agreement on this?

Axel Hecht

unread,
Aug 13, 2008, 2:38:56 PM8/13/08
to

Sounds cool to me.

Pseudo code courtesy of shaver and #hg:

for i in $alllocales; do hg -R $LOCALE_REPO_BASE$i tip --template
$i'\t{node}'; done > changeset-list

to be run at sign-off time on updated repos.

In particular for the early releases, I'd like to keep some flexibility
for search-bustage, code bustage, et al, so keeping that file separate
from shipped-locales might be a good idea.

Axel

Ben Hearsum

unread,
Aug 13, 2008, 3:00:51 PM8/13/08
to
Axel Hecht wrote:
> In particular for the early releases, I'd like to keep some flexibility
> for search-bustage, code bustage, et al, so keeping that file separate
> from shipped-locales might be a good idea.
>

For the sake of not breaking existing tools I support this as well. I
suggest putting it in either hg.m.o/build/tools, or
hg.m.o/build/buildbot-configs (alongside our yet-to-be-created release
automation configs). It would super duper good for release automation
not to have to clone a huge repository just to get locales+changesets.

What do you think?

Axel Hecht

unread,
Aug 13, 2008, 4:47:59 PM8/13/08
to

Yeah, and with the help of djc, Rob has a script in bug 449209 that uses
hg identify, which works on remote repos. Just for public FYI.

Axel

Robert Kaiser

unread,
Aug 16, 2008, 1:45:31 PM8/16/08
to
Axel Hecht wrote:
> Can you put that into a webapp behind ldap auth? I'd be game with that.

Actually, That would sound interesting, it could be coupled with only
allowing it if compare-locales succeeds for this changeset and we'd have
both automatic testing and localizer signoff.

Robert Kaiser

Reply all
Reply to author
Forward
0 new messages