Blockers to using web-platform-tests?

[Philip Jägenstedt]

Hi all,

Good progress is being made towards web-platform-tests being the default way of testing, well summarized by James Graham in This year in web-platform-tests on Mozilla's dev-platform list.

Then there's also Next year in web-platform-tests, and I'd like to ask the same question to blink-dev as James asked of dev-platform:

What are your blockers to using web-platform-tests more, or exclusively?

Relevant OKRs (public):

• Ecosystem Infra 2018 OKRs
  
• Ecosystem Infra 2018 Q1 OKRs (TODO: better organized objectives)
  

[Harald Alvestrand]

Speaking for WebRTC: Blockers for "Exclusively":

- Need to test some aspects (like garbage collection interaction) that need private APIs for the testing

- Need to test legacy APIs that aren't in the spec and won't ever get there, but still should continue to work until we retire them

- Need to test according to our implementation choices, including the choice to leave out certain features for now.

- Need to test stuff that isn't required by the spec (such as VP9 support)

Blockers for "Mainly".

- None that seem important to me!

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYeAicRGf_QZ9zFw9-DWZP_9LbC_BZhZuy7GSmhg0B%3DvFA%40mail.gmail.com.

[Philip Jägenstedt]

Thanks for your replies, Harald! Comments inline.

On Mon, Dec 18, 2017 at 8:37 PM Harald Alvestrand <h...@google.com> wrote:

Speaking for WebRTC: Blockers for "Exclusively":

- Need to test some aspects (like garbage collection interaction) that need private APIs for the testing

- Need to test legacy APIs that aren't in the spec and won't ever get there, but still should continue to work until we retire them

These make sense. In particular a case like the old getStats seems not worth trying to share tests for.

- Need to test according to our implementation choices, including the choice to leave out certain features for now.

Do you mean the problem of tests for feature X that also depend on feature Y? Where it actually make sense to test feature X without feature Y, I think this fixes for this should be welcome in the tests. At least two ways:

• Put shared setup code in a script and split into two test files.
• If Y depends on X, at the end of tests for X, add more test/async_test/promise_test for Y, so that they failing those doesn't affect feature X's tests.

That's a bit of work and means that one can tell from the structure of the tests which order things were implemented in practice, but I think that's OK.

- Need to test stuff that isn't required by the spec (such as VP9 support)

Is this left entirely untested by the shared tests? Is there a way to detect support for VP9, so that some tests could be run only if it's supported? That would match spec language like "if VP9 is supported, then [...] must [...]"

Blockers for "Mainly".

- None that seem important to me!

Yay!

 

On Mon, Dec 18, 2017 at 2:24 PM, Philip Jägenstedt <foo...@chromium.org> wrote:

Hi all,

Good progress is being made towards web-platform-tests being the default way of testing, well summarized by James Graham in This year in web-platform-tests on Mozilla's dev-platform list.

Then there's also Next year in web-platform-tests, and I'd like to ask the same question to blink-dev as James asked of dev-platform:

What are your blockers to using web-platform-tests more, or exclusively?

Relevant OKRs (public):

• Ecosystem Infra 2018 OKRs
  
• Ecosystem Infra 2018 Q1 OKRs (TODO: better organized objectives)
  

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.