Interactive Tests (dbg) builder is too slow

1 view
Skip to first unread message

John Bates

unread,
Apr 19, 2012, 7:10:17 PM4/19/12
to chromium-dev
Look at the run times of these interactive_ui_tests:

Even the green runs are too close to the timeout cutoff, causing random timeouts across many tests. Why is Interactive Tests (dbg) so slow?

Nicolas Sylvain

unread,
Apr 19, 2012, 7:32:17 PM4/19/12
to jba...@google.com, chromium-dev
I'm actually planning to shard this test (on multiple machines) this week.  I haven't had a chance to do it yet though.
When that's done it should be less of a pain for the sheriffs/developers.

If anyone knows how to make the test itself faster (instead of throwing hardware at the problem) this would be great as well :)

Nicolas
 

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

John Abd-El-Malek

unread,
Apr 19, 2012, 7:53:26 PM4/19/12
to nsyl...@chromium.org, jba...@google.com, chromium-dev
On Thu, Apr 19, 2012 at 4:32 PM, Nicolas Sylvain <nsyl...@chromium.org> wrote:


On Thu, Apr 19, 2012 at 4:10 PM, John Bates <jba...@google.com> wrote:
Look at the run times of these interactive_ui_tests:
 
Even the green runs are too close to the timeout cutoff, causing random timeouts across many tests. Why is Interactive Tests (dbg) so slow?
I'm actually planning to shard this test (on multiple machines) this week.  I haven't had a chance to do it yet though.
When that's done it should be less of a pain for the sheriffs/developers.

If anyone knows how to make the test itself faster (instead of throwing hardware at the problem) this would be great as well :)

I don't think you'll be able to shard interactive_ui_tests, since as is many tests fake mouse/keyboard events by bringing the browser to the foreground and telling the OS to simulate a keypress.

I've converted a few interactive_ui_tests to browser_tests during my test conversion, and was planning on finishing the rest of interactive_ui_tests once I'm done with ui_tests. For keyboard/mouse events, I'm just faking them at the RenderWidgetHost level instead.

Nicolas Sylvain

unread,
Apr 19, 2012, 8:01:27 PM4/19/12
to John Abd-El-Malek, jba...@google.com, chromium-dev
On Thu, Apr 19, 2012 at 4:53 PM, John Abd-El-Malek <j...@chromium.org> wrote:


On Thu, Apr 19, 2012 at 4:32 PM, Nicolas Sylvain <nsyl...@chromium.org> wrote:


On Thu, Apr 19, 2012 at 4:10 PM, John Bates <jba...@google.com> wrote:
Look at the run times of these interactive_ui_tests:
 
Even the green runs are too close to the timeout cutoff, causing random timeouts across many tests. Why is Interactive Tests (dbg) so slow?
I'm actually planning to shard this test (on multiple machines) this week.  I haven't had a chance to do it yet though.
When that's done it should be less of a pain for the sheriffs/developers.

If anyone knows how to make the test itself faster (instead of throwing hardware at the problem) this would be great as well :)

I don't think you'll be able to shard interactive_ui_tests, since as is many tests fake mouse/keyboard events by bringing the browser to the foreground and telling the OS to simulate a keypress.
The "(on multiple machines)" was important ;)  I meant to say that I will create a new column on the waterfall, like we did with dbg(1) to dbg(6). And I'll have 2 machines running the interactive tests. The first half on the first one, and the second half on the second one.

Nicolas

John Abd-El-Malek

unread,
Apr 19, 2012, 8:56:48 PM4/19/12
to Nicolas Sylvain, jba...@google.com, chromium-dev
On Thu, Apr 19, 2012 at 5:01 PM, Nicolas Sylvain <nsyl...@chromium.org> wrote:


On Thu, Apr 19, 2012 at 4:53 PM, John Abd-El-Malek <j...@chromium.org> wrote:


On Thu, Apr 19, 2012 at 4:32 PM, Nicolas Sylvain <nsyl...@chromium.org> wrote:


On Thu, Apr 19, 2012 at 4:10 PM, John Bates <jba...@google.com> wrote:
Look at the run times of these interactive_ui_tests:
 
Even the green runs are too close to the timeout cutoff, causing random timeouts across many tests. Why is Interactive Tests (dbg) so slow?
I'm actually planning to shard this test (on multiple machines) this week.  I haven't had a chance to do it yet though.
When that's done it should be less of a pain for the sheriffs/developers.

If anyone knows how to make the test itself faster (instead of throwing hardware at the problem) this would be great as well :)

I don't think you'll be able to shard interactive_ui_tests, since as is many tests fake mouse/keyboard events by bringing the browser to the foreground and telling the OS to simulate a keypress.
The "(on multiple machines)" was important ;)  I meant to say that I will create a new column on the waterfall, like we did with dbg(1) to dbg(6). And I'll have 2 machines running the interactive tests. The first half on the first one, and the second half on the second one.

doh, missed that. I'm sure I won't have gotten to interactive_ui_tests by then, so if it's blocking people, this is fine to do. Otherwise if it can wait a few weeks, the whole binary might just be gone..

John Abd-El-Malek

unread,
May 16, 2012, 1:48:18 AM5/16/12
to Nicolas Sylvain, jba...@google.com, chromium-dev
On Thu, Apr 19, 2012 at 5:56 PM, John Abd-El-Malek <j...@chromium.org> wrote:


On Thu, Apr 19, 2012 at 5:01 PM, Nicolas Sylvain <nsyl...@chromium.org> wrote:


On Thu, Apr 19, 2012 at 4:53 PM, John Abd-El-Malek <j...@chromium.org> wrote:


On Thu, Apr 19, 2012 at 4:32 PM, Nicolas Sylvain <nsyl...@chromium.org> wrote:


On Thu, Apr 19, 2012 at 4:10 PM, John Bates <jba...@google.com> wrote:
Look at the run times of these interactive_ui_tests:
 
Even the green runs are too close to the timeout cutoff, causing random timeouts across many tests. Why is Interactive Tests (dbg) so slow?
I'm actually planning to shard this test (on multiple machines) this week.  I haven't had a chance to do it yet though.
When that's done it should be less of a pain for the sheriffs/developers.

If anyone knows how to make the test itself faster (instead of throwing hardware at the problem) this would be great as well :)

I don't think you'll be able to shard interactive_ui_tests, since as is many tests fake mouse/keyboard events by bringing the browser to the foreground and telling the OS to simulate a keypress.
The "(on multiple machines)" was important ;)  I meant to say that I will create a new column on the waterfall, like we did with dbg(1) to dbg(6). And I'll have 2 machines running the interactive tests. The first half on the first one, and the second half on the second one.

doh, missed that. I'm sure I won't have gotten to interactive_ui_tests by then, so if it's blocking people, this is fine to do. Otherwise if it can wait a few weeks, the whole binary might just be gone..


Nicolas
 

I've converted a few interactive_ui_tests to browser_tests during my test conversion, and was planning on finishing the rest of interactive_ui_tests once I'm done with ui_tests. For keyboard/mouse events, I'm just faking them at the RenderWidgetHost level instead.

btw I've finished converting as many of the interactive_ui_tests to browser_tests or unit_tests as I can. The running time is now 17 minutes. Unfortunately I don't know how much better it is since I didn't keep track of the initial duration. Do you remember?

John Abd-El-Malek

unread,
May 16, 2012, 1:59:57 AM5/16/12
to Nicolas Sylvain, jba...@google.com, chromium-dev
On Tue, May 15, 2012 at 10:48 PM, John Abd-El-Malek <j...@chromium.org> wrote:


On Thu, Apr 19, 2012 at 5:56 PM, John Abd-El-Malek <j...@chromium.org> wrote:


On Thu, Apr 19, 2012 at 5:01 PM, Nicolas Sylvain <nsyl...@chromium.org> wrote:


On Thu, Apr 19, 2012 at 4:53 PM, John Abd-El-Malek <j...@chromium.org> wrote:


On Thu, Apr 19, 2012 at 4:32 PM, Nicolas Sylvain <nsyl...@chromium.org> wrote:


On Thu, Apr 19, 2012 at 4:10 PM, John Bates <jba...@google.com> wrote:
Look at the run times of these interactive_ui_tests:
 
Even the green runs are too close to the timeout cutoff, causing random timeouts across many tests. Why is Interactive Tests (dbg) so slow?
I'm actually planning to shard this test (on multiple machines) this week.  I haven't had a chance to do it yet though.
When that's done it should be less of a pain for the sheriffs/developers.

If anyone knows how to make the test itself faster (instead of throwing hardware at the problem) this would be great as well :)

I don't think you'll be able to shard interactive_ui_tests, since as is many tests fake mouse/keyboard events by bringing the browser to the foreground and telling the OS to simulate a keypress.
The "(on multiple machines)" was important ;)  I meant to say that I will create a new column on the waterfall, like we did with dbg(1) to dbg(6). And I'll have 2 machines running the interactive tests. The first half on the first one, and the second half on the second one.

doh, missed that. I'm sure I won't have gotten to interactive_ui_tests by then, so if it's blocking people, this is fine to do. Otherwise if it can wait a few weeks, the whole binary might just be gone..


Nicolas
 

I've converted a few interactive_ui_tests to browser_tests during my test conversion, and was planning on finishing the rest of interactive_ui_tests once I'm done with ui_tests. For keyboard/mouse events, I'm just faking them at the RenderWidgetHost level instead.

btw I've finished converting as many of the interactive_ui_tests to browser_tests or unit_tests as I can. The running time is now 17 minutes. Unfortunately I don't know how much better it is since I didn't keep track of the initial duration. Do you remember?

I had thought that the build logs don't go that far since they got lost after master restart (maybe I'm mistaken?), but I picked older build numbers and found that April 19 is around 22300. Total cycle time range from 40-60 minutes. The total now is 22 minutes. So I think no need to split across two machines?
Reply all
Reply to author
Forward
0 new messages