Running webtransport wpt tests

[Jake Holland]

Hi webtransport-dev,

I started looking at the current webtransport state by trying to run wpt, and I'm seeing a few odd things.  I wanted to check where things stand and what's currently expected.  Is this the right forum?

I don't think this is blocking me on what I'm looking into, but is there someone looking at the wpt behavior, if so is a report on this of interest to you, and do you need more information about the setup?  Should I open a ticket for this, and if so against wpt or webtransport?

What I'm seeing:

I've got the server running in one ubuntu VM and the client running in another VM on the same host system.  If I manually run chrome 98.0.4758.9, I see some quic traffic and some passed and one failed test in webtransport/connect.https.any.html:

KEY=$(cat tools/wptrunner/wptrunner/browsers/chrome_spki_certs.py | \
  grep -E "^WPT_FINGERPRINT = " | \
  sed -e "s/WPT_FINGERPRINT = '\([^']*\)'/\1/")
/usr/bin/google-chrome  --ignore-certificate-errors-spki-list=${KEY} \
 --origin-to-force-quic-on=web-platform.test:0  \
 https://web-platform.test:8443/webtransport/connect.https.any.html

Passing --headless means no quic traffic goes to the server.  Not sure if that's expected, and not sure what test results come out.

If I try running wpt, I see no UDP traffic to the server.  I tried both with and without the "--origin-to-force-quic-on" passed with --binary-arg, and neither way got any traffic to the wpt server.

One time it claimed they all passed, but usually it claims there's one failed test, different from the one that fails from manually running:

./wpt run --binary=/usr/bin/google-chrome --binary-arg="--origin-to-force-quic-on=web-platform.test:0" --webdriver-binary=/usr/bin/chromedriver chrome webtransport/connect.https.any.html

Not sure why these all pass when no traffic comes to the server's vm, which is set up with the /etc/hosts entries from ./wpt make-host-file $SERVERIP on both the server and client, that seems unexpected to me.

Thanks,

Jake

[Yutaka Hirano]

Hi,

Thank you for reaching out to us.

We expect all test cases in connect.https.any.html to pass. According to wpt.fyi they're passing.

I'm not sure why they're not working in your environment. Is it possible that VM settings affect the test results?

--
You received this message because you are subscribed to the Google Groups "web-transport-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web-transport-...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/web-transport-dev/f6c4014d-1d3d-447f-b841-60f991bbf4d8n%40chromium.org.

[Jake Holland]

Hi Yutaka,

Thanks for responding!

I guess VM settings could play a role in the differences here, but as far as I can tell my setup matches with the instructions at https://web-platform-tests.org/running-tests/from-local-system.html#system-setup.  Perhaps there are some undocumented requirements?

But I'm also suspecting there might be at least one problem with the test harness, especially because when running with "wpt run", several of the tests claim to pass even when tcpdump sees no QUIC traffic to the server.  I don't see how this can be consistent with a properly functioning test that checks whether webtransport connections succeeded.

Does anyone have a local wpt setup where they do see traffic sent to their local server from a wpt run?

If so, maybe we could compare some of the possibly relevant environment, like the the command we're using to invoke wpt run, or the chrome version (mine is 98.0.4758.9) or the aoiquic commit (9e360a57d6b8cf78844fba814d9968cd5f602648)?

-Jake

[Yutaka Hirano]

Some tests expect that WebTransport session establishment to be blocked. These tests will pass when the network is down.