help me identify perf regressions

[Ojan Vafai]

I'm testing out gasper alerts for the webkit canary bots. So far today it's been ~40 false positives + 1 true regression (that got rolled back). That's actually a good enough hit rate that I think it's much easier than manually monitoring the perf graphs.

That said, I have 0 insight into whether it's missing any actual regressions. Over the next few weeks, could the active gardeners just ping/cc me anytime they encounter a perf regression?

If there are no missing regressions, then I think we could make it the gardeners job to monitor the gasper emails instead of having to look at the graphs. It would be considerably less work and would catch perf regressions much faster.

Thanks,

Ojan

[Dirk Pranke]

This sounds great!

-- Dirk

> --
>
>

[Annie Sullivan]

FYI, I think we can probably manage to reduce the false positives if we get some feedback on why they're not helpful. Some things that could possibly be problems:

* Monitoring too many graphs

* Alerts too sensitive

* Getting emails on improvements as well as regressions

* % change calculation incorrect

* Actual problems with GASP

Any Googler who wants to take a look at the alerts can see a full archive at https://groups.google.com/a/google.com/forum/?fromgroups#!forum/webkit-gasper

If anybody has time to make a list of good/bad alerts, it would be helpful to have the "Debug this alert in Gasper" link, the rev the reported regression happens at, and a quick explanation of why it's not helpful. You don't need to look at all the alerts; a small sample would be useful. I made a doc here for feedback: http://go/webkit-gasper-feedback

Thanks!

-Annie

[Ojan Vafai]

I've been assuming the problem is that the tests are too noisy, no? When you look at the graph after a few runs, you can see where there was something that looked like a regression but wasn't. Other than that, I haven't noticed concrete problems. I haven't taken too much time to look into any specific case though.

I think getting emails on improvements will be nice eventually. Eventually, when there are improvements, we should lower our bar for when the bots turn red. Right now, we rarely ever do that, so performance improvements get lost.

[Ojan Vafai]

Getting emails on improvements is definitely worth it. The gasper emails helped me file the following two bugs about actual regressions: https://bugs.webkit.org/show_bug.cgi?id=100872 and https://bugs.webkit.org/show_bug.cgi?id=100873. Today the false positive rate has been somewhere around ~50%. That's totally acceptable IMO.