PSA: New recommended way to rebaseline layout tests

[Quinten Yearsley]

For new CLs which require rebaselining layout tests, you can now download new baselines for all platforms based on try job results. This means you can now commit new baselines along with your CL that changes the test results.

To do this:

1. Upload your CL.
2. Trigger Blink try jobs.
• The try bots to use are the blink try bots that end in blink_rel. This can be done via the Rietveld UI or git cl try.
3. Wait for the try jobs to finish.
4. Run webkit-patch rebaseline-cl to fetch new baselines.
5. Commit the new baselines and upload a new patch.

For updated documentation, see how to rebaseline.

This is currently still experimental -- for bug reports, please include the component Blink>Infra.

[Emil A Eklund]

This is awesome, thank you so much for doing this! Bye bye merge
conflicts in TestExpectations, tests effectively being skipped whilst
waiting for the bots, and mutli-CL reverts!

Can't wait to try the new workflow.

[Dirk Pranke]

This will hopefully be a big step forward for us. Thanks for working on it!

-- Dirk

--
You received this message because you are subscribed to the Google Groups "blink-infra" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-infra+unsubscribe@chromium.org.
To post to this group, send email to blink...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-infra/CADu_oUBF7kw0cY4XtDt2hv4uqb-mTeAUYZ88172M4vMmosdS%2BQ%40mail.gmail.com.

[Christian Biesinger]

Hi Quinten,

thanks a lot for working on this. One question though, you say I need the blink_rel bots. But CQ dry run runs win_chromium_rel_ng, etc, and those run the layout tests. Does rebaseline-cl not work with those -- do I really need to manually trigger the blink bots?

Thanks!

-Christian

[Stephen Chenney]

I've run the CQ dry run for a rebaseline and it definitely doesn't produce the required results. I'm running again manually specifying all the specific platforms' _blink_rel bots.

The easiest way to address this may be to have another single-button option to "Select Rebaseline Try Jobs".

Stephen.

[Quinten Yearsley]

Aye, that's correct -- right now webkit-patch rebaseline-cl only looks at the (non-CQ) blink try bots, so you have to manually trigger and wait for those :-/

The reason for using this separate set of try bots instead of {linux,mac,win}_chromium_rel_ng is to support cases where different versions of the same OS have different baselines; the CQ bots cover one version per OS currently.

A button to select rebaseline try jobs sounds good. Although, then you still have to wait for the try jobs to finish. I was thinking one possible next step would be to add a command-line script to trigger the try jobs and wait.

Note, webkit-patch rebaseline-cl now assumes that a baseline is correct only for the specific platform it came from (e.g. Mac 10.10, Linux Precise), and then after downloading each baseline, it de-dupes them, so it currently relies on having results for every single platform -- it may be nicer if it assumes that one linux result is valid for all linux if only one linux result is available, for example.

[Christian Biesinger]

I see... in this case, can you add an option to git cl upload similar to the current --cq-dry-run that uses these new bots? My current workflow relies on such a commandline option :)

-Christian

[Robert Iannucci]

I believe that git cl try supports an option to trigger additional bots (https://cs.chromium.org/chromium/tools/depot_tools/git_cl.py?q=CMDtry&sq=package:chromium&l=1) but I'm on my phone at the moment and codesearch seems to hang chrome on this phone :/. Maybe webkit-patch rebaseline-cl could use that to trigger the jobs if it can't find them? So you basically run it once, it triggers the jobs, then running it later consumes the results?

R

[Quinten Yearsley]

Yep -- git cl try will try the release blink try bots for rebaselining with with:

git cl try -b linux_precise_blink_rel -b linux_trusty_blink_rel \

  -b mac10.9_blink_rel -b mac10.10_blink_rel \

  -b mac10.11_blink_rel -b mac10.11_retina_blink_rel \

  -b win7_blink_rel -b win10_blink_rel

This is unwieldy to type out, but if you add it to a shell script or function or alias after uploading, then this should work.

But anyway, I agree that easier way(s) to trigger try jobs will be helpful; I filed crbug.com/639533. I really like Robbie's idea of making webkit-patch rebaseline-cl trigger any jobs that are necessary, if there aren't currently try job results for the latest patchset of a CL.

[qyea...@google.com]

Update:

Since this was posted, some fixes have been made to this new tool based on initial feedback. So if you haven't tried it, feel free to give it a try next time you have a change that requires rebaselining layout tests, especially if you suspect that your change may affect platform-specific baselines, and you want to verify those baselines before committing:

1. Upload your CL
2. Run Tools/Scripts/webkit-patch rebaseline-cl
3. Wait for all try jobs to finish
4. Run Tools/Scripts/webkit-patch rebaseline-cl again
5. Upload again and inspect

If you encounter issues, please let me know by filing an issue with component Blink>Infra, or reach out to me directly. (Although I may not respond this coming week due to Thanksgiving.)

P.S. besides this, there are still a couple other supported ways to rebaseline:

1. If you know the baselines should not be platform-specific:

1. Build the target blink_tests
2. Run Tools/Scripts/run-webkit-tests --new-test-results <my-test>
   
3. Upload and inspect

2. If you suspect the new baselines are platform-specific but you don't want to inspect the new baselines and want to commit as quickly as possible:

1. Edit LayoutTest/TestExpectations to mark <my-test> as [ NeedsRebaseline ]
2. Wait and inspect the CL created by rebaseline-o-matic at some point in the future.

[Stephen Chenney]

Please note that using the rebaseline-cl method is preferred over the rebaseline-o-matic approach because the latter makes both reverting changes harder and it less likely that test regressions will be caught in review.

Stephen.