[PSA] Starting in early Q4, trying out AutoVM for Suite scheduler and release testing

[Vivian Lam]

Summary (TL;DR)
In early Q4, we are piloting AutoVM, a system that automatically shifts HW testing to run on virtual machines (VMs) for tests tagged with 'HW_Agnostic' on a probabilistic basis starting with 50%. Test owners will be able to opt-out if they prefer not to have their tests swapped to VMs.

Why is this happening?

To optimize the utilization of our testing resources for upcoming AL needs, as a pilot we are going to execute tests tagged as "HW_Agnostic" on VMs. This change is expected to reduce test demand by approximately 300-400 DUTs/day. This transition will streamline our testing processes, free up hardware resources for other critical testing needs, and improve overall testing efficiency to ensure upcoming AL needs will be met within our current lab infrastructure. It's important to note that this will only affect test suites that are scheduled through the suite scheduler configs and release testing. Tests that are run through other scheduling mechanisms will not be impacted. Our initial plan is to use VMs with a 50% probability. Over time, this probability would gradually increase to 90%. We believe that this optimization will significantly enhance our testing capabilities and enable us to deliver higher quality products more efficiently and enable a smoother rollout for AL testing needs. We will closely monitor the transition process and its impact to ensure a seamless transition and address any issues that may arise.

We need your input!

Shifting tests to VMs automatically means you may not find that your test results have seemingly gone missing. We are evaluating how to best present the VM results in Testhaus with two different options. Please take a look and vote for your preferred option using the poll at the top of the below doc. If you have any additional feedback, we encourage you to leave a comment in the document. This will help us know who to follow up with for further discussion. [Voting Open] AutoVM-Testhaus Options

Want to learn more or get in touch?

For any questions, please reach out to: Varun Srivastav Derek Beckett David Haddock Steve Lin Vivian Lam

(This email is bcc’ed to: chromeos-eng, OWNERS in Susch config, and google group owners of hw_agnostic tagged tests)