Revisiting "suggested sign-off" on the dashboard

[Axel Hecht]

Hi,

I'm currently thinking about the "suggested sign-off" feature on the
sign-off pages. Right now, that's any push that's newer than the
accepted sign-off that's green.

I've come across a few cases where the latest status was actually red,
but had better translation coverage, and the old green one was just
green at the time.

I propose that we reduce the "suggested" one to only occur if the latest
compare-locales result is green.

That doesn't discourage the other revisions more than it used to, I just
think it's not encouraging sub-optimal revisions.

Opinions?

Axel

[Cédric Corazza]

Hi,
I would say "suggested sign-off" is OK for aurora and beta imo. For
releases, I think the sign-off should be mandatory: we'we got to be sure
the product has been tested by the localizers.
But maybe we should investigate why a green state has not the sign-off.
For my part, I often miss the deadline for sign-off because I forgot the
date, because deadlines can slip, etc.
We could have a smoother process and let everybody spares time
(especially l10n staff who have to ping every locale team without
sign-off), by adding the sign-off deadline for product and branch on the
dashboard itself, which is the place every localizer looks at regularly.
That wouldn't forbid to continue announcing deadlines on the newsgroups
though.

My two cents.

Cédric

[Axel Hecht]

To clarify, "suggested sign-off" is a visual indication on the sign-off
page what's technically looking good to sign-off on. One thing we're
currently working on is exposing that information on the teams page.

The way I see things happening is that localizers see there's something
to do, then they do stuff, land it, tree goes green, builds pop up,
folks test and based on their testing, sign off.

Thus, it's on purpose that green status isn't immediately signed off,
but we're looking for a more obvious status.

Seems also that I should clarify what we're using in builds:

On central and aurora, the latest status of the l10n repository is used
to create the depend and nightly builds.
On beta, you get depend builds for a push, but the weekly beta builds
are taken off of the signed-off revisions.

Does that help to scope the question?

Axel

[Pavel Franc]

>> For my part, I often miss the deadline for sign-off because I forgot the
>> date, because deadlines can slip, etc.
>> We could have a smoother process and let everybody spares time
>> (especially l10n staff who have to ping every locale team without
>> sign-off), by adding the sign-off deadline for product and branch on the
>> dashboard itself, which is the place every localizer looks at regularly.
>> That wouldn't forbid to continue announcing deadlines on the newsgroups
>> though.
>

Hi guys,
personally I have big trouble with signing.

I'm working on central, once upon a time (every six weeks) I move the
work to aurora, and I don't touch the beta at all. So, it is very easy
to forgot to sign the right revision in aurora/beta. Especially when the
dashboard is green, and everything is marked as signed (some revision
from past). Last time we were creating Firefox 10 beta builds from
revision of Firefox 9 !

Either some type of automatic signing, beta sign-off removing after
aurora->beta migration, or big warning in dashboard would be really
appreciate from my side.

Best,
Pavel Franc

[Justin Wood (Callek)]

Well if you do all your work in central/aurora you can signoff on aurora
and have that signoff carried forward to the beta release without having
to touch beta (you of course *can*)

We should also do a better job of notifying teams when there are
outstanding signoffs for a given cycle (Pike has a tool to make that
easier from us drivers -- at least in identifying teams, someone
could/should write up a tool to contact you).

Part of that is having responsive l10n contacts (that understand
english). And the method in which to get ahold of the teams can vary as
well. Which is how I view the effort to update the L10n:Teams pages on
wiki.m.o etc.

I for one haven't been contacting people who fall behind/forget as much
as I should, and I hope to get better at that.

--
~Justin Wood (Callek)

[Axel Hecht]

We're *so* close to having a team page with the interesting info right
there on one page.

The real downside right now is that it takes 13 seconds to load at least
for the french team. That's not cool, so I'm optimizing. And finding
what we're currently calling "suggested sign-off" on the sign-off page
is rather expensive, so I looked deeper, and remembered that I found
that suggestion to be wrong actually in a few cases. Thus I started the
discussion here.

Axel

[Mad Maks]

I have the same problem als Pavel, you don't see witch version is the
accepted signoff on the dashboard (Bug 669127).

but what i don't understand why is there a sign off for aurora, as far i
can see aurora is build an aviable even if you don't sign off. I think
it is much clearer when you only have to sign off beta.

Tim Maks

[flod]

Il 17/02/12 10.09, Mad Maks ha scritto:

> but what i don't understand why is there a sign off for aurora, as far
> i can see aurora is build an aviable even if you don't sign off. I
> think it is much clearer when you only have to sign off beta.

Working on central I usually sign-off on Aurora, while I request new
sign-offs on Beta only if there are problems that need a fix and can't
wait the next cycle.
Honestly I believed this was the most common workflow, but I've just
realized it's probably not.

Francesco

[Michael Bauer]

Would it not make sense to assume a signoff at the deadline date if the
locale is green at that point? Maybe add a "do not release" button or
something for the odd occasion when a locale is green but does not want
a release (not that I can think of one)?

Quite a few other projects seem to run along those lines.

Michael

16/02/2012 23:56, sgrìobh Pavel Franc:

[Axel Hecht]

On 17.02.12 11:48, Michael Bauer wrote:
> Would it not make sense to assume a signoff at the deadline date if the
> locale is green at that point? Maybe add a "do not release" button or
> something for the odd occasion when a locale is green but does not want
> a release (not that I can think of one)?
>
> Quite a few other projects seem to run along those lines.

I'm afraid you're more optimistic about the value of those tests than I am.

We're constantly trying to make those tests better and cover more
ground. You can tell by the number of updates I released to
compare-locales just recently. You can also tell from that number that
there has been ways to break localizations that we had yet to discover.

Also, the source-based checks only cover the most simplistic error cases.

The real criteria if we should ship a particular revision is the human
testing. "Sign-off" means that the localizer is confident that the state
of the tree is good. Not that you happen to close a window when you're
trying to copy some text from the scratchpad, for example. To give a
recent example.

Firefox is unlike most open-source projects. It's huge, both from the
amount of work, and in the impact that bugs have on humans around the
globe. We need to work to get better still, so that we keep having that
impact.

On the other hand, "sign-off" isn't all that uncommon in concept. It
actually is really close to pull requests on github. It has a different
name, and your pull request isn't on a repo that you fork, but you're
requesting an upstream maintainer to look at your contribution and ship
it. We're a tad better than what you get at github in that we have some
build and test automation for you, but from a conceptual point of view,
it's pretty similar to a pull request.

Axel

[Rimas Kudelis]

Hi,

2012.02.17 14:08, Axel Hecht rašė:
> On 17.02.12 11:48, Michael Bauer wrote:
>> Would it not make sense to assume a signoff at the deadline date if
>> the locale is green at that point? Maybe add a "do not release"
>> button or something for the odd occasion when a locale is green but
>> does not want a release (not that I can think of one)?
>>
>> Quite a few other projects seem to run along those lines.

I would want that too.

> I'm afraid you're more optimistic about the value of those tests than
> I am.
>
> We're constantly trying to make those tests better and cover more
> ground. You can tell by the number of updates I released to
> compare-locales just recently. You can also tell from that number
> that there has been ways to break localizations that we had yet to
> discover.
>
> Also, the source-based checks only cover the most simplistic error
> cases.
>
> The real criteria if we should ship a particular revision is the
> human testing. "Sign-off" means that the localizer is confident that
> the state of the tree is good. Not that you happen to close a window
> when you're trying to copy some text from the scratchpad, for
> example. To give a recent example.

This assumption that the localizer will test before signing up... it
simply can't be verified. I would guess that most of the teams who test
before signing off, would do that either way (and Michael's suggestion
does not even affect their workflow). On the other hand, the (small)
teams who don't have time or don't want to bother testing, they can sign
off untested builds anyway. So again, it's an assumption that has no
strong basis (and I've said it before).

Me, for example. I usually sign off as soon as my tree becomes green, if
I don't expect further changes (that is, if we're in a string freeze and
mobile team haven't announced that they don't care about it). And I
definately don't test everything I commit. I DO use Firefox daily, and
on the box where I'm localizing it, I have nightly builds of both
Firefox and Thunderbird running, and I do check out at least some of the
new strings (those that are unclear of otherwise interesting to me).
But, if I would happen to introduce that Scratchpad bug you mentioned as
an example, I don't think it's very likely that I would have caught it.
And yet, I would have signed that revision off, if it were green.

Signing off is good for those who do the QA homework. But for those who
don't, it's just a useless nuisance. I almost remember the "You have
checked, right?" message by heart, and yet, I can't say I have checked.
:P And by the way, either I am quite good at not making mistakes, or
compare-locales is quite good at catching them, or the users don't
really care, or there are too few users (I believe that at least the
first is true and the last is not) because I only get like one or two
bugs a year filed for my locale.



Regarding the actual subject of this thread: Axel, your proposal
definately makes sense, I wouldn't even ask if I were you. But what does
a suggested sign-off look like? Is it just the sentence saying "There is
a sourcestamp that technically looks good to sign off on.", or is it
something more?

Regards,
Rimas

[Axel Hecht]

Yeah, patterns are rather different.

Maybe it helps to detail on my basic thoughts:

1) all locales are equal
2) we can't just ship what's in hg.

1) isn't really true, but it's also true. In this context, it boils down
to me not feeling good to create some process for some locales, and a
different process for others.
2) worst case example would be the german locale breaking the google
plugin. doesn't even have to be malicious edits by the ferengi localizer
taking over that revenue stream, can just be accidental.

At this point, there's no resources at moco to just verify everthing
that lands. I'm also not sure that's well spent resources. We should
incorporate the community in doing so, and I'd be happy to see
suggestions on how we can make the source review tools more accessible,
or somesuch. Also, we'll probably discuss this differently once the new
team page is up.

> Regarding the actual subject of this thread: Axel, your proposal
> definately makes sense, I wouldn't even ask if I were you. But what does
> a suggested sign-off look like? Is it just the sentence saying "There is
> a sourcestamp that technically looks good to sign off on.", or is it
> something more?

I asked, but I didn't wait with the patch 'til I got all answers ;-).

"Suggested" doesn't really mean anything more than that. It doesn't even
discourage to sign-off on things that are not suggested. For those you
should probably just be more aware of why you want that particular
change in.

Axel

PS: I did file a documentation bug referencing this thread, in case
you're wondering :-)

[Armen Zambrano G.]

From release automation it would be helpful if:
* mozilla-beta would take automatically the "suggested sign-off" OR the
one signed off by the localizer

Would this be possible?

cheers,
Armen

[Rimas Kudelis]

Hi :)

Hehe, we keep debating this over and over... :)

For 1), my point is that if you gave two possibilities for all locales,
they would still remain equal, but could choose the process they feel
suits them better.

For 2), again, my point is that if I "sign off" whenever I feel like
I've made my last commit, then you still get to check whether or not I
have "accidently" tried to take over the revenue stream or broken
something else. You still perform the same checks, and use the same
resources you'd use checking the last good check-on. The only difference
is the sign-off itself. I (and Michael) would simply prefer the lack of
a sign-off to be treated as an implicit sign-off on the last green
check-in. It would be just a default, nothing more. The locales who
can't imagine signing off without a thorough QA process could still
sign-off explicitly. No-one would be hurt (well, IMO).

>> Regarding the actual subject of this thread: Axel, your proposal
>> definately makes sense, I wouldn't even ask if I were you. But what does
>> a suggested sign-off look like? Is it just the sentence saying "There is
>> a sourcestamp that technically looks good to sign off on.", or is it
>> something more?
>
> I asked, but I didn't wait with the patch 'til I got all answers ;-).
>
> "Suggested" doesn't really mean anything more than that. It doesn't even
> discourage to sign-off on things that are not suggested. For those you
> should probably just be more aware of why you want that particular
> change in.

I see. :)

> PS: I did file a documentation bug referencing this thread, in case
> you're wondering :-)

What bug?

Rimas

[Michael Bauer]

This may be a case of "my bad" but while I did *a lot* of testing during
the first few releases, I mainly see the dashboard as a means of
remining if I need to translate more stuff. Unless some dramatic changes
are in the pipeline that affect the layout, I can't see that changing.
Maybe not what dashboard is intended for but hey. I do use the beta on a
daily basis on my main system but it's been a long time since I caught
anything that wasn't a typo.

I take you're point about not wanting a two class system but perhaps
that's the wrong way of thinking of it. There are "mature" locales and
"new" locales to my mind and if you've been through several releases, I
personally can't see the harm.

But I'm not going to labour the point :)

Cheers,

Michael

17/02/2012 12:08, sgrìobh Axel Hecht:

[Axel Hecht]

My point is, I would offer two possibilities. I'd put some locales in
bucket one, and the rest in the other bucket. Locales in bucket one
could probably still opt out, but the other bucket couldn't get into
bucket one.

> For 2), again, my point is that if I "sign off" whenever I feel like
> I've made my last commit, then you still get to check whether or not I
> have "accidently" tried to take over the revenue stream or broken
> something else. You still perform the same checks, and use the same
> resources you'd use checking the last good check-on. The only difference
> is the sign-off itself. I (and Michael) would simply prefer the lack of
> a sign-off to be treated as an implicit sign-off on the last green
> check-in. It would be just a default, nothing more. The locales who
> can't imagine signing off without a thorough QA process could still
> sign-off explicitly. No-one would be hurt (well, IMO).

To be honest, you neglect how much time it takes to go through these.
Yes, I ask you guys to do some work. But that's because if you do, I
don't have to. If you feel that it's tedious that I ask you that,
imagine doing it 100 times.

>>> Regarding the actual subject of this thread: Axel, your proposal
>>> definately makes sense, I wouldn't even ask if I were you. But what does
>>> a suggested sign-off look like? Is it just the sentence saying "There is
>>> a sourcestamp that technically looks good to sign off on.", or is it
>>> something more?
>>
>> I asked, but I didn't wait with the patch 'til I got all answers ;-).
>>
>> "Suggested" doesn't really mean anything more than that. It doesn't even
>> discourage to sign-off on things that are not suggested. For those you
>> should probably just be more aware of why you want that particular
>> change in.
>
> I see. :)
>
>> PS: I did file a documentation bug referencing this thread, in case
>> you're wondering :-)
>
> What bug?

https://bugzilla.mozilla.org/show_bug.cgi?id=728066 would be the bug.

Axel

[Axel Hecht]

Why would this be useful for release automation?

Axel

[Staś Małolepszy]

(xposting to tools-elmo)

On 02/17/2012 09:38 AM, Axel Hecht wrote:

> We're *so* close to having a team page with the interesting info right
> there on one page.
>
> The real downside right now is that it takes 13 seconds to load at least
> for the french team. That's not cool, so I'm optimizing. And finding
> what we're currently calling "suggested sign-off" on the sign-off page
> is rather expensive, so I looked deeper, and remembered that I found
> that suggestion to be wrong actually in a few cases. Thus I started the
> discussion here.

I wonder if this means that we're hitting a bottleneck that's blocking
us from releasing the new version of the teams page with all its goodness.

I see three solutions:

1. release as-is, and optimize next. The teams page will feel a bit
slow in some cases, but the goodness is there for all to see and benefit
from.

2. rip off the 'suggested sign-off' feature and work on it next. The
page will be fast and it will still provide a lot of good data in a very
good-looking and organized form. Much better than it is right now, even
if without the suggested sign-off feature.

3. make 'suggested sign-off' async, maybe trigger it with a click?
try to find a solution that would combine the best of the two above.

-stas

[Axel Hecht]

.... or just land the branch that's requesting review from peterbe that
makes our perf awesome.
https://bugzilla.mozilla.org/show_bug.cgi?id=713878 for the curious.

Axel

[Armen Zambrano G.]

*sigh* I removed the 2nd point by mistake. I guess this is orthogonal.

If there was a "FF11 beta" milestone (we would not need the 6 beta
milestones) and we could query the "suggested sign-off"/"signed off
revision" milestone then we could simplify the process. I think.
I guess, I am trying to see if we need the same mozilla-release sign-off
process for betas.

I assume the process for mozilla-release would still be the same.

How does the creation of milestones and sign-offs work?
Does compare-locales fail on changes that otherwise L10n-drivers would
reject?

cheers and thanks in advance,
Armen

[Axel Hecht]

On 21.02.12 17:22, Armen Zambrano G. wrote:
> On 12-02-17 5:49 PM, Axel Hecht wrote:
>> On 17.02.12 20:44, Armen Zambrano G. wrote:
>>> From release automation it would be helpful if:
>>> * mozilla-beta would take automatically the "suggested sign-off" OR the
>>> one signed off by the localizer
>>>
>>> Would this be possible?
>>>
>>> cheers,
>>> Armen
>>
>> Why would this be useful for release automation?
>>
>> Axel
> *sigh* I removed the 2nd point by mistake. I guess this is orthogonal.
>
> If there was a "FF11 beta" milestone (we would not need the 6 beta
> milestones) and we could query the "suggested sign-off"/"signed off
> revision" milestone then we could simplify the process. I think.
> I guess, I am trying to see if we need the same mozilla-release sign-off
> process for betas.

The "11 beta" piece is an appversion, in our data model. milestone
actually keep a reference of what we shipped when, mostly because hg
tags are mutable, for one, and because tags in a 100 different repos
aren't any useful.

The rapid release process changes a good few assumptions that went into
the currently implemented design of the dashboard, and we're struggling
to get out of the hacks that we're using to keep afloat. I have a patch
that kind-of does the trick, but got stuck in data migration. That work
as been delayed by mobile, mobile, and then mobile, and then team page,
and then mobile. I hope to get back to it soon, but it'll be full of
merge conflicts.

> I assume the process for mozilla-release would still be the same.

We're actually not having any process for release anymore. You ship it
and forget about it.

> How does the creation of milestones and sign-offs work?
> Does compare-locales fail on changes that otherwise L10n-drivers would
> reject?

sign-offs get done by localizers, as I mentioned elsewhere you can
picture them as pull-requests into a repo that we don't have. That gets
accepted or not.

The accepted ones get 'tagged' onto another data structure during
release, the milestone.

The logic around that is too complicated these days, fixing it is very
complicated.

The fail/reject question was handled on #l10n, afaict.

Axel

[Staś Małolepszy]

On 02/21/2012 05:06 PM, Axel Hecht wrote:

> .... or just land the branch that's requesting review from peterbe that
> makes our perf awesome.
> https://bugzilla.mozilla.org/show_bug.cgi?id=713878 for the curious.

Oh sweet, always a step ahead :)

-stas