nightly testing coverage, what, why, how?
Axel Hecht
.planning about what we want to achieve with our nightly builds.
I'd like to selfishly focus that discussion around
- what do we want to get tested?
- where do we want to get it tested?
and based upon that,
- what do we expose to testers when?
The "what" is mostly which kind of gecko in which kind of
UI/application? central vs 1.9.2 vs 1.9.1 vs Lorentz vs UI experiments
or other project branches. Only builds with a real update channel? What
about those without?
The "where" is mostly which fragment of the internet do we want this
code to be tested against? I glanced at the stats for Firefox 3.7a1pre
nightlies, and there are about as many testers in russia as in the US.
Ukraine, Indonesia are follow ups, Germany and the UK are the only
countries in the top ten. In terms of localized builds, we're *very*
dominated by en-US builds, though.
I couldn't find data on SeaMonkey or thunderbird nightlies, so I can't
compare those numbers with fx.
Comparing geckos, it seems that we have twice as many testers on 1.9.2
than on 1.9.1, and 5 times as many on central, compared to 1.9.2. Given
that we can only compete on our testers, is that ratio good? (Neglecting
that we lost more on 4.0a1pre than we have on 3.6pre) If not, where
would we consider the ratio to be healthy? How many of those do we want
on Lorentz, too?
My selfish background is that I still owe an answer to releng on which
locales we should keep building on central, and just localizer's opt-in
isn't enough data for me to make that call. I'd like to know which
regions we want testing coverage on, and which localizations would be
good to get that, and then suck up to those localizers to work on yet
another tree. IMHO. I know that I don't want to bother our long tail
with keeping l10n-central up to date.
Axel
Axel Hecht
Looking at least at beltzner, and anyone that he'd be pointing fingers at.
Axel