Jeff Walden wrote:
> schrep wrote:
>> Gecko 1.9.0.x:
>> Release date: By end of June 2008
>> Status: Stable
>> Accepted Changes: Security, stability, performance, minor enhancements
>> that do not change *any* exposed APIs. All changes must pass all
>> regression tests and not regress performance.
> I'd go one further here: all changes must be accompanied with automated
> regression tests or must have manual litmus tests accompanied with an
> explanation why automated testing cannot be done. Barring unusual
> circumstances where a test can be written that works on 1.9.1 but
> doesn't work on 1.9.0 (thinking of damage-related tests that might rely
> on compositor, for example), the tests must land in 1.9.0 as part of the
> patch (after landing in 1.9.1, of course) and must run on tinderboxen on
> checkin.
Totally agreed. Our increase in automated regression testing has really
saved us time and time again - and our bar for test coverage (either
manual or automated) should continue to rise.
>> Gecko 1.9.1:
>> Release date: Beta stable summer 2008, production stable end of 2008
>> Status: In development
>> Accepted Changes: Anything that doesn't break frozen API's. Passing
>> unit tests and proof of no negative impact on perf from Talos required
>> before check-in
> How many tests (or reasons why tests cannot be written) can we require
> here? Branch obviously can't have a lower a bar than trunk, but that
> doesn't say anything about what should happen on trunk. I think we
> should set the bar for trunk fairly high and require a reviewed (even if
> it's just a sanity check) test be committed with each fix (or the
> explanation why it can't be tested and a litmus ? if appropriate). If
> it fails without the patch and succeeds with, that's good enough. On a
> case-by-case basis you'd want to require some bugs to have a greater
> number, depth, and concentration of tests, but even if each bug had just
> a single shallow test that'd be an improvement over where we are now.
++ same as above.
