Dealing with prove-it-then-spec-it situations in web-platform-tests

[Philip Jägenstedt]

Hi all,

There is a "when automerging of tests goes awry" thread on public-test-infra that could be of interest to anyone who's had to deal with a situation where implementation and shipping precedes changing the spec + wpt in order to prove something web compatible before committing to it.

Please comment there (or here) if you have ideas for how to make that workflow awesome in the new 2-way sync world.

Thanks!

[Anne van Kesteren]

It seems to me that your second option doesn't actually address the
problem, which is that an existing test is changed based on a change
to a feature that is not agreed on yet. If you change an existing
test, renaming it or placing it elsewhere will break the workflow of
others that expect it to remain where it is and expect it to remain
constant.


--
https://annevankesteren.nl/

[Boris Zbarsky]

On 3/7/17 3:57 AM, Anne van Kesteren wrote:
> It seems to me that your second option doesn't actually address the
> problem, which is that an existing test is changed based on a change
> to a feature that is not agreed on yet.

My understanding of the second option was that one creates a _new_ test
with the -tentative name or in the tentative location, instead of
modifying the existing test. Which seems reasonable.

-Boris

[Philip Jägenstedt]

Right, that is what I mean. I am also assuming that the tentative plan has some level of buy-in, and not that anyone who hopes for a certain outcome should go ahead and change their implementation and put the tests in web-platform-tests.

[Ojan Vafai]

On Tue, Mar 7, 2017 at 6:38 AM Philip Jägenstedt <foo...@chromium.org> wrote:

On Tue, Mar 7, 2017 at 10:25 PM Boris Zbarsky <bzba...@mit.edu> wrote:

On 3/7/17 3:57 AM, Anne van Kesteren wrote:
> It seems to me that your second option doesn't actually address the
> problem, which is that an existing test is changed based on a change
> to a feature that is not agreed on yet.

My understanding of the second option was that one creates a _new_ test
with the -tentative name or in the tentative location, instead of
modifying the existing test.  Which seems reasonable.

Makes sense to me. I prefer filename to location except in the cases where we're writing tests as we write the spec for new APIs, in which case the directory is more convenient. Could we support both? I guess just directory would be fine too if you can nest a tentative directory in an existing test suite.

 

Right, that is what I mean. I am also assuming that the tentative plan has some level of buy-in, and not that anyone who hopes for a certain outcome should go ahead and change their implementation and put the tests in web-platform-tests.

Is buy-in important? At some level, having tests for every vendor's behavior is better than the vendor changing behavior and not sharing the test, right? It's even OK IMO if we have two tentative tests that disagree with each other. Once the behavior is proven in the wild and we get cross-vendor agreement, we update the test appropriately.

We should try to make sure that any tooling that measures conformance with the test suite properly splits out tentative tests to avoid putting pressure on vendors to pass the tentative tests before they've been proven in the wild and agreed upon.

 

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYfHLeKejeQ%3DvQziartuq-aWdzKqS7v0fnH0FwgrHZSaFQ%40mail.gmail.com.

[Mark Dittmer]

I also like the second option. Re: file name vs. director(y|ies): does one or the other make it easier to filter out "tentative" tests or "tentative tests from [insert platform vendor]"? I think we should track metrics surrounding these tests. If there are too many of them (particularly too many "old" ones, for some notion of old), this counts against WP health, IMHO.

//Mark

On Tue, Mar 7, 2017 at 9:16 PM, Ojan Vafai <oj...@chromium.org> wrote:

On Tue, Mar 7, 2017 at 6:38 AM Philip Jägenstedt <foo...@chromium.org> wrote:

On Tue, Mar 7, 2017 at 10:25 PM Boris Zbarsky <bzba...@mit.edu> wrote:

On 3/7/17 3:57 AM, Anne van Kesteren wrote:
> It seems to me that your second option doesn't actually address the
> problem, which is that an existing test is changed based on a change
> to a feature that is not agreed on yet.

My understanding of the second option was that one creates a _new_ test
with the -tentative name or in the tentative location, instead of
modifying the existing test.  Which seems reasonable.

Makes sense to me. I prefer filename to location except in the cases where we're writing tests as we write the spec for new APIs, in which case the directory is more convenient. Could we support both? I guess just directory would be fine too if you can nest a tentative directory in an existing test suite.

 

Right, that is what I mean. I am also assuming that the tentative plan has some level of buy-in, and not that anyone who hopes for a certain outcome should go ahead and change their implementation and put the tests in web-platform-tests.

Is buy-in important? At some level, having tests for every vendor's behavior is better than the vendor changing behavior and not sharing the test, right? It's even OK IMO if we have two tentative tests that disagree with each other. Once the behavior is proven in the wild and we get cross-vendor agreement, we update the test appropriately.

We should try to make sure that any tooling that measures conformance with the test suite properly splits out tentative tests to avoid putting pressure on vendors to pass the tentative tests before they've been proven in the wild and agreed upon.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYfHLeKejeQ%3DvQziartuq-aWdzKqS7v0fnH0FwgrHZSaFQ%40mail.gmail.com.

--

You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CANMdWTsKbJYX5dQBDx6t90r818SbTuOAJjGGF2RjNNsdCtn7yw%40mail.gmail.com.