Deno WPT results on wpt.fyi

[Luca Casonato]

Hey 👋

Deno runs WPT tests during CI to verify that our implementations of Web APIs are conformant with relevant specs. We publish results to a dashboard at https://wpt.deno.land. Philip (@foolip) reached out on Twitter to suggest that we could push our WPT results to https://wpt.fyi. This would be great, as that would allow contributors and spec maintainers to use the existing and familiar wpt.fyi infrastructure for analyzing results and comparing engine behaviour.

As for how to technically do this, from what I understand it boils down to pushing a wptreport style test output to the wpt.fyi API. It would then handle processing and display on the page. We record wptreport compatible result outputs for each commit now (example: https://dl.deno.land/wpt/d6f6e157bddc02a64beaa1c96fcda8d8ea27aa6d-wptreport.json). The remaining step would then be to POST these reports to https://wpt.fyi/api/results/upload with relevant credentials.

My main remaining question is how often we can and should push up data. We collect data for each commit, but I don't know if it makes sense to push it all up every commit. We could also just push once a day. Our wptreport is relatively small, because we only run a very limited subset of tests.

Luca

[Philip Jägenstedt]

Hi Luca!

This is very cool, let's try to make it work! I have created a "deno" uploader account on staging.wpt.fyi and shared the password with you in a secure manner. In addition to that I expect we'll quickly find that some wpt.fyi assumptions will fail, and we'll need to add deno as a product both on the backend and in the UI, and I'll need to ask @Kyle Ju for help with that.

To prove out the concept, however, we can fake that the report comes from an already supported browser, it's OK if we do that on the staging instance. Let's work on the WPT Matrix channel to see how far we can get!

If we can get just one report working, there's the question of how often to push data. If you'd find it useful to have commits for every Deno commit that shouldn't be a problem, but if the goal is to compare to with other results then using the epochs/* branches in WPT which trigger runs every 3 hours or slower might be good. Exactly how to wire this up depends on which repo is running the tests, of course. If you have a copy of WPT in Deno, then I guess it becomes a question of updating that copy on a reasonable cadence.

Again, cool project, let's figure it out!

Best regards,

Philip

--
You received this message because you are subscribed to the Google Groups "ecosystem-infra" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ecosystem-inf...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ecosystem-infra/CAOwE1h3ZLsrq6Q2ZV4J6yQYm-uHQWGXtzeZsAoVsOaGg%3DGFhYg%40mail.gmail.com.