Do we still need automated_ui_tests and reliablity_tests?

[John Abd-El-Malek]

These are build on all try and build runs, but aren't used there as far as I can tell. These depend on browser since they're ui_tests, so take a while to link.

reliablity_tests is for ChromeBot, but I don't know if this is still useful anymore now that we have lot of users and different channels? Do any developers/testers look at it?

automated_ui_tests: afaik this was like reliablity_tests but wasn't just browsing to a site, but also did some basic actions to the page. Is this actually used now?

[Mark Larson (Google)]

I think we can kill automated_ui_tests: definitely not used anymore.

I'll follow up with the guy working on getting the new chromebot/reliability tests running to see if that depends on reliability_tests and let you know.

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Nirnimesh]

reliability_tests will continue to be used by the new chromebot setup.

--
../NiR

[Darin Fisher]

We should be able to greatly speed up the link time though.  Tests that primarily drive Chrome over IPC shouldn't need to link against browser.lib.  If they do, it is probably just for some constants or other small bits.

-Darin

On Tue, May 15, 2012 at 5:45 PM, John Abd-El-Malek <j...@chromium.org> wrote:

--

[John Abd-El-Malek]

Agreed. I just don't know if it hasn't been done yet because there's some trickery involved or if it's just really low-hanging and no one tried :) I'll give it a shot.

[Darin Fisher]

I tried last year, but got distracted.  It should be fairly easy for that test.  I was looking at ui_tests.exe, which had a lot more dependencies on browser.lib.

-Darin

[John Abd-El-Malek]

Looking at this now, I'm not sure this is possible. reliability_tests itself depends on PrefService which is part of 'browser'. Even if prefs and all that it depended on was a separate library, test_support_common has _many_ dependencies on browser. We could try to split this up into two parts, one of which depends on browser and one which doesn't. I'm not sure if that would actually be possible or not until it's tried though, and if that'll be worth it (i.e. this still depends on test_support_content which is also large, and many other targets).

[John Abd-El-Malek]

I'm not sure anymore that this isn't used if ChromeBot is still planned on being used.

I thought ChromeBot both loaded pages, and also ran random commands. The code for the latter seems to be in automated_ui_tests target. See http://code.google.com/searchframe#OAMlx_jo-ck/src/chrome/test/reliability/automated_ui_test_base.h&exact_package=chromium&ct=rc&cd=1&q=automated_ui_test_base.h&sq=

+Chris

On Tue, May 15, 2012 at 7:25 PM, Mark Larson (Google) <m...@chromium.org> wrote:

[Darin Fisher]

Does reliability_tests really use prefs, or does it just refer to pref names?  I figure it mostly just issues IPC messages.  One idea might be to create a browser_constants library.

-Darin

[John Abd-El-Malek]

It uses it, see http://code.google.com/searchframe#OAMlx_jo-ck/src/chrome_frame/test/reliability/page_load_test.cc&exact_package=chromium&q=page_load_Test&type=cs&l=439

[Anthony LaForge]

With the number of persistent renderer stability regressions, ChromeBot will need to play a more prominent role in the quarters to come.

Kind Regards,

Anthony Laforge
Technical Program Manager
Mountain View, CA

[Nico Weber]

Was there a consensus on the fate of reliability_tests reached, here or elsewhere?

http://crbug.com/288383 was filed about removing reliability_tests (and other targets) with the text "Clean up test targets (…, reliability_tests, …), they are unused. If they are used, then they will not be removed." and https://codereview.chromium.org/23536051/ deleted them.

The reliability_tests target happened to be used by the memory bots though, and the reaction to it being used was then to post a CL to remove the memory bot using them ( https://codereview.chromium.org/24299003/ ), which in my eyes contradicts the "if they are used, they will not be removed" bit.

It's well possible that removing that bot is the right thing to do, but I couldn't find any discussion about the fate of reliability_tests newer than this thread.

Thanks,

Nico

[John Abd-El-Malek]

reliablity_tests used the automation framework, which has been deprecated for over a year. So in that sense, anyone that is depending on tests that use it have had ample time to convert to other frameworks (i.e. browser_tests, telemetry). There comes a point when we just have to bite the bullet and start removing that code.

[Nico Weber]

On Tue, Sep 24, 2013 at 12:38 PM, John Abd-El-Malek <jabde...@google.com> wrote:

reliablity_tests used the automation framework, which has been deprecated for over a year. So in that sense, anyone that is depending on tests that use it have had ample time to convert to other frameworks (i.e. browser_tests, telemetry). There comes a point when we just have to bite the bullet and start removing that code.

Thanks for pointing out that this is part of the automation framework deprecation, that wasn't clear to me.

Do we have some new test that's comparable to reliablity_tests that the memory bots should run instead?

[John Abd-El-Malek]

On Tue, Sep 24, 2013 at 12:51 PM, Nico Weber <tha...@chromium.org> wrote:

On Tue, Sep 24, 2013 at 12:38 PM, John Abd-El-Malek <jabde...@google.com> wrote:

reliablity_tests used the automation framework, which has been deprecated for over a year. So in that sense, anyone that is depending on tests that use it have had ample time to convert to other frameworks (i.e. browser_tests, telemetry). There comes a point when we just have to bite the bullet and start removing that code.

Thanks for pointing out that this is part of the automation framework deprecation, that wasn't clear to me.

Do we have some new test that's comparable to reliablity_tests that the memory bots should run instead?

It's hard to say because I don't know which parts of reliablity_tests were most useful in catching memory errors that weren't caught by other tests. Once that's known, then it shouldn't be hard to write a specific test for that in either browser tests harness or telemetry.