Gardening status

[Stephen White]

• Win bots were suffering from indefinite hangs; dpranke@ and cmp@ managed to clean it up (we think)note that the Win DEPS bots will still be hanging until we roll past dpranke@'s change at r133076
• - Luckily no major build breakages while Win bots were out of commission
• - MountainLion bots (which are not core yet) are very noisy and tend to make garden-o-matic hard to read; dpranke@ is working on cleaning them up
• - ChromeOS trybots seem verrrry slow and flaky
• 
  
• - got a try in for 133073 which looks good (will put it in the CQ before I leave)
•   - unfortunately this includes r133044 that introduces http, isolatedWorlds test crashiness
•   - Adam reverted in r133078, so once we roll past that it should clear up
• - fischman@'s change http://trac.webkit.org/changeset/133069 to clear cache in DRT between tests doesn't seem to have caused major badness (yay)
•   - exposed some flakiness in accessibility tests:  https://bugs.webkit.org/show_bug.cgi?id=100887

Overall, disappointed that I didn't get more rolls in, but at least we're not too far behind ToT.

Stephen

[Dirk Pranke]

Note that the roll to r133044 caused issues in browser_tests in debug
mode as well, so we ended up reverting that roll.

-- Dirk

> --
>
>

[Adam Barth]

Sorry for the bad patch. :(

Hopefully rolling past the revert will work.

Adam

[Stephen White]

Ongoing issues:

• Builder Win7 (dbg) has been consistently failing 10-15 browser_tests.  This seems to be bot-specific; will ping a trooper for a reboot.

Resolved issues:

• Please join me in congratulating aboxhall@ on a very successful first day gardening:  several successful rolls, and many test suppressions and cleanups.
• tc@ fixed a number of flaky tests exposed by the cacheing change (http://trac.webkit.org/changeset/133069).
• I landed a speculative fix for flakiness in a filter blur test (http://trac.webkit.org/changeset/133399) which ended up de-flaking about five compositing tests.

Some observations about the long tail of flakiness (mostly from staring at the flakiness dashboard):

• Linux seems to be the least flaky of all the platforms right now.  Once you get past the first 30 or so tests, the rest have very low flake.
• The #1 culprit on MacOS Debug seems to be https://bugs.webkit.org/show_bug.cgi?id=81803, causing many CSS selectors failures
• The #1 culprit on Windows seems to be HTTP tests, many of which have intermittent timeouts (and many of which are filed under different bugs)
• Assuming they each have a single root cause, nailing either of the above two bugs would go a long way to cleaning up a lot of gardening noise

Stephen

[Ojan Vafai]

On Mon, Nov 5, 2012 at 5:13 AM, Stephen White <senor...@google.com> wrote:

• The #1 culprit on Windows seems to be HTTP tests, many of which have intermittent timeouts (and many of which are filed under different bugs)

This has been true forever. When we switched the Mac/Linux bots back to using Apache, the HTTP tests got considerably less flaky. Filed https://bugs.webkit.org/show_bug.cgi?id=101373 for doing the same on Windows.

Thanks for looking into actually fixing flakiness issues!