Confusing flakiness results from WPT bot

[Ian Clelland]

Hey folks,

I'm looking at a WPT export failure on this Chromium CL, and trying to understand the results.

As far as I can tell, neither the Chromium changes nor the WPT changes in that CL should have affected the tests that the bot is complaining about. The change relates to reporting in permissions policy, and while the three flagged tests are related to permissions policy, a reporting string change should not have affected them at all.

I'm also not sure what the CI test is doing -- does it run its current version of Chrome with the tests from the CL (in which case those should *definitely* not have been affected, since they aren't modified in that CL) or does it also recompile Chrome, and run the updated tests there (which would make more sense from a CI perspective, but probably wouldn't run on GitHub in that case?)

I'd appreciate any pointers; whether this is likely to indicate an actual issue with the CL, or if this just serendipitously exposed some pre-existing flakes in those tests, which we should just file bugs for.

Thanks!

Ian

[Stephen Mcgruer]

Hey Ian,

Thanks for reaching out! 

> As far as I can tell, neither the Chromium changes nor the WPT changes in that CL should have affected the tests that the bot is complaining about. The change relates to reporting in permissions policy, and while the three flagged tests are related to permissions policy, a reporting string change should not have affected them at all.

That CL changes feature-policy/resources/featurepolicy.js, which those flaky tests include, hence they are detected as affected.

> I'm also not sure what the CI test is doing -- does it run its current version of Chrome with the tests from the CL (in which case those should *definitely* not have been affected, since they aren't modified in that CL) or does it also recompile Chrome, and run the updated tests there (which would make more sense from a CI perspective, but probably wouldn't run on GitHub in that case?)

It runs the current dev Chrome release with the changes from the CL.

> I'd appreciate any pointers; whether this is likely to indicate an actual issue with the CL, or if this just serendipitously exposed some pre-existing flakes in those tests, which we should just file bugs for.

Looks like those tests are just pre-existing flake; I'll happily admin-merge your exported PR once it lands on the Chromium side :)

Thanks,
Stephen

--
You received this message because you are subscribed to the Google Groups "ecosystem-infra" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ecosystem-inf...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ecosystem-infra/CAK_TSXJvpyqVPWMBtVRSi%3DLX_RTRh-gjV_s_qYTp1foQp3bT2w%40mail.gmail.com.

[Ian Clelland]

Thanks, Stephen!

FYI, that CL has now landed; I don't know if you need to take any action on your end to merge, or if it is just something that will come up next time it flakes.

Ian

[Stephen Mcgruer]

Thanks for the heads up. I've admin-merged the upstream WPT PR.