PSA: Enabling full swarming on linux_android_rel_ng tomorrow night

[Mike Stipicevic]

Hello!

linux_android_rel_ng, your friendly Android trybot, will soon be faster! (If all goes well tomorrow.)

As of today, all of linux_android_rel_ng's Android tests have run successfully on its fully-swarmed counterpart, Android Swarm Builder. Tomorrow night at 6pm Pacific, we will convert linux_android_rel_ng to full swarming as well. This will significantly reduce the cycle time of the bot. The tree will be closed for approximately 1 hour while we restart masters and move Android testers to swarming.

Please contact me if you have any questions!

- Mike

[Mike Stipicevic]

(of course, this is pending the current Swarming outage being fixed in time)

[Daniel Murphy]

Woooo! Thanks for all the hard work, Stip!

[Mike Stipicevic]

Reminder: this is happening in a little over an hour.

[Mike Stipicevic]

The transition has completed. We are watching the first builds to make sure everything is working alright before we open the tree.

[Mike Stipicevic]

The transition has gone smoothly, although we're tracking elevated failures in telemetry_perf_unittests in crbug.com/616865. Opening the tree, will monitor throughout tomorrow to see how things are working.

[Elliott Sprehn]

Seeing failures: https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/81358

Is that related ?

[Mike Stipicevic]

The android_webview_test_apk failures look like real flakes, especially because content_browsertests has been swarmed for weeks now. I'll investigate if it's been happening more now that swarming is enabled.

[Mike Stipicevic]

I filed a bug for the android_webview_test_apk flakiness here: https://bugs.chromium.org/p/chromium/issues/detail?id=617012.

[Mike Stipicevic]

Looks like we've survived one day of full load. We'll monitor it throughout next week, but I think we can declare this an (early) success. After the transition, linux_android_rel_ng is twice as fast as before: the median execution has dropped from 52 minutes to 27 and the 90th percentile has dropped from 80 minutes to 41.

Thanks to all who helped make this happen, especially (alphabetically): bpastene@, dpranke@, friedman@, jbudorick@, martiniss@, maruel@, sergeyberezin@.

- Mike

[Kenneth Russell]

Great work Mike and everybody! Having Swarming available for Android is a huge step forward.