How do you use the Flakiness Dashboard without swamping Chrome?

[Michael Giuffrida]

I want to use the flakiness dashboard to identify whether a failing test on my CL is a flake. However, I've always found the dashboard very hard to use, and lately, simply unusable. I know people use this dashboard successfully on a daily basis, so I must be doing something wrong.

For me, the page has over 1 million DOM nodes. Its tab uses over 2 GB of RAM according to Chrome's task manager. Trying to Ctrl-F a test name is very, very, very slow.

I know there are filters, but they don't seem to work as I expect. Basically, the page first loads alllllll the tests. Then, if I enter a test name, it *appends* those tests' results to the end of the page, after the always-present gigantic block of all tests. How do I narrow the results down to just the ones I'm interested in?

Here's the bottom of the flakiness dashboard, at 25% zoom. Note the scrollbar size -- the test I've filtered for is at the bottom, after 8,000 rows of tests I don't care about. How do I prevent the dashboard from loading all rows in the first place?

[Demetrios Papadopoulos]

> Then, if I enter a test name, it *appends* those tests' results to the end of the page, after the always-present gigantic block of all tests. 

Could this be a recent regression? I recall having used the dashboard in the pasts and only seeing search results, without the gigantic list. Although now I am able to reproduce what you are describing.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/e43babfd-49a7-47ac-83e8-edefa3bb1a51%40chromium.org.

[Karan Bhatia]

FWIW I find the dashboard to be quite unusable as well. However, the omnibox generally autocompletes some existing link for a test and then I modify the url to only load the results for a particular test, which is reasonably fast. And yeah, I have been lazy enough to not file a bug till now!

[Anita Woodruff]

First load of https://test-results.appspot.com/dashboards/flakiness_dashboard.html does seem slower than usual for me - in the tens of seconds - but I was able to successfully just see test results for OfflineBackgroundTaskTest.java just now by

a. Waiting for the page to load

b. Typing OfflineBackgroundTaskTest into the search box

c. Selecting Test type 'chrome_junit_tests' - upon which it reloads pretty quick.

I ended up at https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=chrome_junit_tests&tests=OfflineBackgroundTaskTest so maybe just try modifying that query if you don't want to wait for the initial page to load?

I have also observed the issue where it appends search results instead of replacing existing ones, but I think re-selecting Test type forces a reload.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/8075b641-2fe6-4a9c-90a8-025d3b13b809%40chromium.org.

[Dirk Pranke]

I didn't see a reply from any infra folks, so here's something ...

The dashboard is indeed known to be nigh-unusable these days (and has been for a long time). Unfortunately the folks that were working on improvements to it were recently pulled off to work on more important performance improvements to Monorail. The good news is that they've largely completed those improvements now, and so I'm hoping to get some cycles back from them to see if we can find some low-hanging fruit to pick, though large-scale improvements probably require a significant rearchitecting of the underlying databases.

In particular, I think the defaults that the page has (to load all of the results for one test suite on one bot) don't make sense these days, and we should change them. 

If you use a test filter, it *should* just list those files, and I thought it did that when I was sheriff last week, but I haven't tested this too exhaustively.

It would be interesting to know which features people really want to see in this app. For myself, I generally one of the following use cases:

* the results for a single test (or tests) across a few builders, because I'm trying to look at a set of failures from sheriff-o-matic.

* the results for a single test (or tests) across a few builders because I'm trying to see the actual flakiness history of the a test (or some tests), from a bug.

In both those cases, I might sometimes want the results from all CI builders, but I almost never want the tryserver results mixed in with the CI builders, and it would be nice if the UI kept them more separate (i.e., optional and/or not displayed by default).

I don't think I've ever wanted to see *all* of the failures for a given test suite on a builder, but I think that's actually what loads by default. I expect most of our test suites have too many failures / flakes / tests over time for this to be actually interesting, but I could be wrong. Do others often use that view?

-- Dirk

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAFObWWbO%3DPiQCDUa-Gv-k_MZ3SdiB5mbxqUYkSe%3DS2rvzSr%2BqQ%40mail.gmail.com.

[Harald Alvestrand]

The question I want answered from a flakiness dashboard is:

"Is this test on this builder failing on my CL because I broke it, or because it's flaky?"

That means that the most important view for me is one builder, one test, and getting to that *fast*.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTDgWkM1hyG-r1AjQ2CrKMW6MgZ9Lg435mVf3t0dXbeoFg%40mail.gmail.com.

[Sean McCullough]

We are aware that the flakiness dashboard is nearly unusable a lot of the time these days, and in Q1 we worked on a design for a replacement that works much better for Chromium's current needs. But as Dirk noted we'd placed those plans on hold in Q2 while Monorail's performance issues were more urgent. I am very interested in resurrecting this design in Q3 but don't have any committed plans yet to do so.

However: If you are comfortable writing SQL, we do have some BigQuery tables (test-results-hrd.events.test_results) that contain most of the data you see in the flakiness dashboard UI (though I'm afraid they're only available to @google.com users at this time). There's a Cloud Console interface and a CLI to access these, and they won't crash chrome.

Using SQL is much less turn-key than a web UI obviously, but it may get you your answers faster than waiting for the flakiness dashboard replacement to be available.  Please let me know if you're interested in using these BQ tables in the mean time.

[just returning from vacation, sorry for the delayed response!]

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.