Running external/wpt tests on Windows?

[Raymond Toy]

On linux and mac, I can say 

run-webkit-tests external/wpt/webaudio

to run the WPT webaudio tests.  But on windows, this doesn't work.  I get an error:

# ret> 1

Traceback (most recent call last):

  File "C:\src\chromium\src\third_party\WebKit\Tools\Scripts\webkitpy\thirdparty\wpt\wpt\wpt", line 5, in <module>

    wpt.main()

  File "C:\src\chromium\src\third_party\WebKit\Tools\Scripts\webkitpy\thirdparty\wpt\wpt\tools\wpt\wpt.py", line 132, in

 main

    rv = script(*args, **kwargs)

  File "C:\src\chromium\src\third_party\WebKit\Tools\Scripts\webkitpy\thirdparty\wpt\wpt\tools\manifest\update.py", line

 96, in run

    update_from_cli(**kwargs)

  File "C:\src\chromium\src\third_party\WebKit\Tools\Scripts\webkitpy\thirdparty\wpt\wpt\tools\manifest\update.py", line

 37, in update_from_cli

    download_from_github(path, tests_root)

  File "C:\src\chromium\src\third_party\WebKit\Tools\Scripts\webkitpy\thirdparty\wpt\wpt\tools\manifest\download.py", li

ne 124, in download_from_github

    force=force)

  File "C:\src\chromium\src\third_party\WebKit\Tools\Scripts\webkitpy\thirdparty\wpt\wpt\tools\manifest\download.py", li

ne 73, in download_manifest

    commits = commits_func()

  File "C:\src\chromium\src\third_party\WebKit\Tools\Scripts\webkitpy\thirdparty\wpt\wpt\tools\manifest\download.py", li

ne 123, in <lambda>

    return download_manifest(path, lambda: git_commits(tests_root), github_url,

  File "C:\src\chromium\src\third_party\WebKit\Tools\Scripts\webkitpy\thirdparty\wpt\wpt\tools\manifest\download.py", li

ne 38, in git_commits

    return [item for item in git("log", "--format=%H", "-n50").split("\n") if item]

  File "C:\src\chromium\src\third_party\WebKit\Tools\Scripts\webkitpy\thirdparty\wpt\wpt\tools\manifest\vcs.py", line 17

, in git

    return subprocess.check_output(full_cmd, cwd=repo_path, stderr=subprocess.STDOUT)

  File "C:\python_27_amd64\files\lib\subprocess.py", line 537, in check_output

    process = Popen(stdout=PIPE, *popenargs, **kwargs)

  File "C:\python_27_amd64\files\lib\subprocess.py", line 679, in __init__

    errread, errwrite)

  File "C:\python_27_amd64\files\lib\subprocess.py", line 896, in _execute_child

    startupinfo)

WindowsError: [Error 2] The system cannot find the file specified

What do I need to run the wpt tests on windows?

[Dirk Pranke]

From that stack trace, it looks more like you're hitting an issue launching git/git.exe; is that in your path?

-- Dirk

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAE3TgXGdXfZzRFZu6sPu5VfjQb-RKg5MdhK21S_RUrY7bsHhcQ%40mail.gmail.com.

[Scott Graham]

It's not using shell=True, so I don't imagine it's been tried on Windows.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEoffTDYdRqQhQBiCG_x3bgn230J_kHXa9pEuRSHGu4fz8nVEw%40mail.gmail.com.

[Chris Nardi]

Scott is correct; I looked into this (briefly) when I ran into it, and it's due to git having the path of git.bat in depot_tools on Windows, which shell=False would not expand and thus the script cannot find git. I'm not sure why this works on the bots and not locally (perhaps they have another form of git installed?).

Some scripts have a workaround for this (https://cs.chromium.org/search/?q=git.bat&sq=package:chromium&type=cs), but since this code comes from upstream WPT a solution should probably be proposed there, and then Tools/Scripts/webkitpy/thirdparty/wpt rolled.

[Raymond Toy]

Thanks for all the info.

It's not a show-stopper for me, but we're (slowly) moving as many of our layout tests to wpt, so eventually this is going to be a problem when I can't easily run the tests locally on windows.  But they run on the bots so I can do work that way.  Not great for me or the bots, but it's a way to do it if necessary.

[Daniel Bratell]

Is this operation critical or could the error be turned into a warning?

/Daniel

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAE3TgXFE7%2BD6FHggHiZMX7WKyO8tZt4E4hY_X%2BptSYZZes75YQ%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

[Raymond Toy]

For me it's not critical so a warning is fine instead of an error.

But others must be running into this issue too.  How are they running wpt tests on their windows machines?

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAE3TgXFE7%2BD6FHggHiZMX7WKyO8tZt4E4hY_X%2BptSYZZes75YQ%40mail.gmail.com.

[Chris Nardi]

This should now be fixed after https://chromium-review.googlesource.com/c/chromium/src/+/944720, which just landed a couple hours ago. I just checked with my local Windows machine, and everything worked fine.

[Philip Jägenstedt]

That's awesome, thanks Chris! https://github.com/w3c/web-platform-tests/pull/9652 was the fix, I presume.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAE3TgXFE7%2BD6FHggHiZMX7WKyO8tZt4E4hY_X%2BptSYZZes75YQ%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKXZydeD83VuN9f6-9fTYJ4ykju7xinrAxeT2E_5Zcmzgf6wVA%40mail.gmail.com.

[Raymond Toy]

Super excited to see this, but didn't get around to trying it out until now.  From a ToT from yesterday, I rebuilt as normal and ran:

PS C:\src\chromium\src> .\third_party\WebKit\Tools\Scripts\run-webkit-tests.bat --details --no-retry --skipped=ignore external/wpt/webaudio

No errors now, but no tests are run:

Test configuration: <win10, x86, release>

View the test results at file://C:\src\chromium\src\out\Release\layout-test-results/results.html

Using random order with seed: 1520360716

Baseline search path: win -> generic

Using Release build

Pixel tests enabled

Regular timeout: 6000, slow test timeout: 30000

Command line: C:\src\chromium\src\out\Release\content_shell.exe --run-layout-test --ignore-certificate-errors-spki-list=Nxvaj3+bY3oVrTc+Jp7m3E3sB1n3lXtnMDCyBsqEXiY= --user-data-dir --enable-direct-write --enable-crash-reporter --crash-dumps

-dir=C:\src\chromium\src\out\Release\crash-dumps\reports -

Found 0 tests; running 0, skipping 0.

No tests to run.

This command is what I use to run the tests on linux and it works.

I must be doing something wrong, but I don't know what.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAE3TgXFE7%2BD6FHggHiZMX7WKyO8tZt4E4hY_X%2BptSYZZes75YQ%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

[Chris Nardi]

I ran into that as well, and I think that's because glob patterns aren't supported for running tests in WPT. The docs could probably be more explicit about that, but I believe that's the limitation you're running into. Because of this, I think you can only run single tests (or a list of tests) at a time.

[Raymond Toy]

Ah, ok.  Oddly, this runs all of the wpt tests that I want on linux.  I guess it's an issue on windows for whatever reason.

[Robert Ma]

By not supporting "glob" we mean things like "external/foo/bar-*"

"external/wpt/webaudio" should work, but since Windows gets less attention in webkitpy, there might be some bug here. Have you tried using the other slash, "external\wpt\webaudio"? What about non-WPT layout tests? (e.g. fast/dom, or fast\dom)

[Raymond Toy]

I already tried backslashes and that didn't make a difference.  Specifying fast\dom works fine.  1627 tests are run.  (Hmm. They all crash for me, but that's a different issue.)

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/f284c100-54e2-4a8d-960b-af298b3c38ef%40chromium.org.

[Chris Nardi]

https://crrev.com/542166 just landed, which should fix running entire folders in external/wpt on Windows. Using either forward or backslashes should work (both "external/wpt/webaudio" and "external\wpt\webaudio". Hopefully this fixes the major issues with using WPT locally on Windows.

[Robert Ma]

And I'd like to mention that CL also ends up fixing https://crbug.com/796424, bring back 4000 virtual tests that were missing on Windows :)

Thank you, Chris! Nice job!

[Philip Jägenstedt]

Is the lack of support for glob patterns a nuisance that many have run into? And what kinds of workarounds do you then use?

A minor wrinkle if we tried to fix this is multi-global tests where the test name is not the filename, but I believe that's pretty confusing with or without glob matching.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAE3TgXFE7%2BD6FHggHiZMX7WKyO8tZt4E4hY_X%2BptSYZZes75YQ%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKXZydeD83VuN9f6-9fTYJ4ykju7xinrAxeT2E_5Zcmzgf6wVA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Robert Ma]

On Sat, Mar 10, 2018 at 9:09 PM, Philip Jägenstedt <foo...@google.com> wrote:

Is the lack of support for glob patterns a nuisance that many have run into? And what kinds of workarounds do you then use?

A minor wrinkle if we tried to fix this is multi-global tests where the test name is not the filename, but I believe that's pretty confusing with or without glob matching.

This is exactly the reason globs are not supported in r-w-t, as a tradeoff for simpler implementation to support WPT.

[Raymond Toy]

This works great!  Thanks for fixing this.  Everything works just like on my linux and mac boxes.

I don't normally use globbing for the wpt tests, so not having that on windows doesn't bother me.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAE3TgXFE7%2BD6FHggHiZMX7WKyO8tZt4E4hY_X%2BptSYZZes75YQ%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKXZydeD83VuN9f6-9fTYJ4ykju7xinrAxeT2E_5Zcmzgf6wVA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.