Dogfood and feedback for "webkit-patch rebaseline-cl"

27 views
Skip to first unread message

Quinten Yearsley

unread,
May 23, 2017, 7:37:30 PM5/23/17
to layout-dev
Hi layout-dev!

I heard that as part of LayoutNG work, many layout test baselines may have to be updated soon, and I was thinking this might be a good time to ask for feedback about "webkit-patch rebaseline-cl", the new tool to download baselines from try jobs.

Now, I'm a little bit concerned that the tool isn't quite ready (in terms of usability, bugs in edge cases, etc.), so I asked paint-dev if they'd like to dogfood it and pdr@ suggested asking here too :-)

Let me know if you've tried this way of rebaselining and have any feedback!

Possible benefits of "webkit-patch rebaseline-cl" for large rebaseline CLs:
  • The old rebaseline-o-matic uses Rietveld, which can't handle very large CLs. With rebaseline-cl you can use Gerrit, which does handle large files and large CLs.
  • Since layout tests are tried before committing new baselines, there may be more confidence that the new baselines are correct.
How to use the tool:
  1. Upload your CL.
  2. Trigger Blink try jobs for try bots on tryserver.blink. This can be done by running Tools/Scripts/webkit-patch rebaseline-cl with no arguments.
  3. Wait for all try jobs to finish.
  4. Run Tools/Scripts/webkit-patch rebaseline-cl again to fetch the new baselines generated in the try jobs. (You can also --help to see more options.)
  5. Commit and upload your new patchset that includes baselines.

Steve Kobes

unread,
May 23, 2017, 8:19:49 PM5/23/17
to Quinten Yearsley, layout-dev
I tried to use this today, on http://crrev.com/2900973003.  When I ran it the second time, instead of fetching any baselines it started a new set of try jobs.  Then I tried it with --no-trigger-jobs and it printed a bunch of errors like:

Got 404 response from:
Failed to fetch results for Build(builder_name=u'android_blink_rel', build_number=2555)
Retry job by running: git cl try -b android_blink_rel

Is there a way to specify a particular build to fetch baselines from (namely https://build.chromium.org/p/tryserver.chromium.android/builders/android_blink_rel/builds/2554 which has finished)?

--
You received this message because you are subscribed to the Google Groups "layout-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to layout-dev+unsubscribe@chromium.org.
To post to this group, send email to layou...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/layout-dev/CAFp4ESkJGc3hGLTYU7E7Z6MASZJ38xyA%2BFs_LwURSeHfHbxnCA%40mail.gmail.com.

Morten Stenshorne

unread,
May 24, 2017, 8:16:43 AM5/24/17
to Quinten Yearsley, layout-dev
On Wed, May 24 2017 at 01:37, Quinten Yearsley <qyea...@chromium.org> wrote:

> *Possible benefits of "webkit-patch rebaseline-cl" for large rebaseline CLs*
> :
>
> - The old rebaseline-o-matic uses Rietveld, which can't handle very
> large CLs. With rebaseline-cl you can use Gerrit, which does handle large
> files and large CLs.
> - Since layout tests are tried before committing new baselines, there
> may be more confidence that the new baselines are correct.

I like rebaseline-cl a lot! One problem, though:

> *How to use the tool*:
>
> 1. Upload your CL.
> 2. Trigger Blink try jobs for try bots on tryserver.blink
> <https://build.chromium.org/p/tryserver.blink/builders>. This can be
> done by running Tools/Scripts/webkit-patch rebaseline-cl with no
> arguments.

$ ./Tools/Scripts/webkit-patch rebaseline-cl
No issue number for current branch.

This is the first time I try it with gerrit. I uploaded the initial
patch set with "git cl upload --gerrit", then I did "git cl try", got
failures, fixed something and did another "git cl upload".

I've got this error occasionally previously too, when using rietveld.

Branch info from .git/config:
[branch "tmp"]
base = a44af8f558766ab8c6b65fff8be4b1ef1a5f903b
base-upstream = refs/remotes/origin/master
gerritissue = 512824
gerritserver = https://chromium-review.googlesource.com
gerritsquashhash = 5bc15411c105dd94b171589ca462133541e4cffc
last-upload-hash = 044038f17a3ce63593d15cef9c6568f26db04aee

--
---- Morten Stenshorne, developer, Opera Software ----
------------------ http://www.opera.com/ -------------

Robert Hogan

unread,
May 24, 2017, 1:31:30 PM5/24/17
to Steve Kobes, Quinten Yearsley, layout-dev
On Wed, May 24, 2017 at 1:19 AM Steve Kobes <sko...@chromium.org> wrote:
I tried to use this today, on http://crrev.com/2900973003.  When I ran it the second time, instead of fetching any baselines it started a new set of try jobs.  Then I tried it with --no-trigger-jobs and it printed a bunch of errors like:

Got 404 response from:
Failed to fetch results for Build(builder_name=u'android_blink_rel', build_number=2555)
Retry job by running: git cl try -b android_blink_rel

 I've been getting this the last few days too, though prior to that the tool was working great for me and I was starting to make regular use of it. When it works its very slick and does a good job of ignoring bogus rebaselines. 

Quinten Yearsley

unread,
May 24, 2017, 2:57:27 PM5/24/17
to Steve Kobes, layout-dev
On Tue, May 23, 2017 at 5:19 PM, Steve Kobes <sko...@chromium.org> wrote:
I tried to use this today, on http://crrev.com/2900973003.  When I ran it the second time, instead of fetching any baselines it started a new set of try jobs.  Then I tried it with --no-trigger-jobs and it printed a bunch of errors like:

Got 404 response from:
Failed to fetch results for Build(builder_name=u'android_blink_rel', build_number=2555)
Retry job by running: git cl try -b android_blink_rel

Is there a way to specify a particular build to fetch baselines from (namely https://build.chromium.org/p/tryserver.chromium.android/builders/android_blink_rel/builds/2554 which has finished)?

In general, this indicates that that build failed (android build 2555). In most cases, if the android bot fails it doesn't affect the new baselines, since most tests aren't run on android. Right now, the way to ignore such failures and assume that there are no platform-specific baselines is to run:

./webkit-patch rebaseline-cl --fill-missing --no-trigger-jobs

But in this particular case, where the test is run on Android and you specifically want to download a new baseline for just one platform and just one test, there is another (somewhat secret) way!

./webkit-patch rebaseline-test-internal --test=TEST --builder=BUILDER --build-number=BUILD_NUMBER

"rebaseline-cl" actually works by checking what tests are failing for what platforms, and then invoking "rebaseline-test-internal" lots of times.

Quinten Yearsley

unread,
May 24, 2017, 2:59:22 PM5/24/17
to Morten Stenshorne, layout-dev
On Wed, May 24, 2017 at 5:16 AM, Morten Stenshorne <mste...@opera.com> wrote:

$ ./Tools/Scripts/webkit-patch rebaseline-cl
No issue number for current branch.

This is the first time I try it with gerrit. I uploaded the initial
patch set with "git cl upload --gerrit", then I did "git cl try", got
failures, fixed something and did another "git cl upload".

I've got this error occasionally previously too, when using rietveld.

Ah, a new bug which I hadn't seen before!

The tool gets the issue number by running "git cl issue"; when you see "No issue number for current branch", does "git cl issue" still give the right output?

Morten Stenshorne

unread,
May 24, 2017, 4:18:17 PM5/24/17
to Quinten Yearsley, layout-dev
$ git cl issue
Using git-cl wrapper /home/mstensho/misc/git-cl
Issue number: 512824 (https://chromium-review.googlesource.com/512824)

Yes, it does! But AHA! I'm using a wrapper script for git-cl that just
unsets DISPLAY and invokes the original git-cl in depot_tools. I've had
some performance problems when working remotely (and it turned out to be
X traffic - don't know why), and I wanted to remind myself that I had
this wrapper script lying around, and hence the "Using git-cl wrapper"
line. That seems to confuse the output parser for rebaseline-cl. I just
removed that additional line of text, and now it works fine. Thanks!
Reply all
Reply to author
Forward
0 new messages