Hey folks,In last week's API owners meeting, we had a short discussion around experiments that aren't appropriate for the Origin Trial framework, but instead would benefit from being rolled out to a percentage of users. I think we agreed on something like the following:
- Percentage experiments that only reach developer-focused releases (those limited to Chrome's Canary and Dev channel, for instance) do not require API owners' involvement. These releases are explicitly unstable, have a limited reach, and can't support strong security or privacy guarantees due to the rapid churn of changes. The value of %-experiments in these environments pretty clearly outweighs the minimal risk of baking in a behavior based on exposure.
- Broadly-accessible releases (Chrome's Beta and Stable channels, for instance) reach many more developers and users, and we'd like API owners to evaluate and approve percentage experiments via an explicit Intent to Experiment. This both ensures that we're evaluating the effect of an experiment on the ecosystem, and gives developers an opportunity to understand what changes might be coming their way.
- Percentage experiments that reach a certain level of the population (say, ~5% of stable) are difficult to distinguish from shipping with regard to their impact on developers' expectations. While it's possible that such experiments might really be experimental (Trust Tokens is a recent example), the risks are similar enough to shipping that we'd see this kind of experiment as a rare exception.
Assuming that matches others' recollection of the conversation, I'd propose adding the above requirements to https://www.chromium.org/blink/launching-features, alongside the discussion of origin trials in step 5 the of "For New Feature Incubations" section. Perhaps along the lines of:
"""If you want to gather data on the usability of your feature that an Origin Trial can help collect, or data about your feature's impact that a percentage-based experiment could support, then proceed to the "Public Experiment" stage in ChromeStatus and fill out the required fields detailing what you hope to learn. This will generate an Intent to Experiment mail that you should send to blink-dev. After receiving at least one LGTM from API Owners, you can proceed with your origin trial or percentage experiement. Collect data and respond to any issues. From here, you may wish to return to Dev Trials, proceed to Prepare to Ship, or park the feature.
Please note that these experiments are not exempt from requiring cross-functional approvals from the Chrome launch review process."""This would also require corresponding changes to the Intent to Experiment template, and to the ChromeStatus workflow as well. It also might be reasonable to create a separate page describing the experimental options in more detail, as well as the approval requirements for exceptions.WDYT?-mike
--
You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-d...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/CAKXHy%3DeiMG3TLA1Kuhr51MDwHFHQgiAmV7vZAy1i9df8v%3DC0tQ%40mail.gmail.com.
On Mon, Dec 14, 2020 at 9:56 AM Mike West <mk...@chromium.org> wrote:Hey folks,In last week's API owners meeting, we had a short discussion around experiments that aren't appropriate for the Origin Trial framework, but instead would benefit from being rolled out to a percentage of users. I think we agreed on something like the following:
- Percentage experiments that only reach developer-focused releases (those limited to Chrome's Canary and Dev channel, for instance) do not require API owners' involvement. These releases are explicitly unstable, have a limited reach, and can't support strong security or privacy guarantees due to the rapid churn of changes. The value of %-experiments in these environments pretty clearly outweighs the minimal risk of baking in a behavior based on exposure.
- Broadly-accessible releases (Chrome's Beta and Stable channels, for instance) reach many more developers and users, and we'd like API owners to evaluate and approve percentage experiments via an explicit Intent to Experiment. This both ensures that we're evaluating the effect of an experiment on the ecosystem, and gives developers an opportunity to understand what changes might be coming their way.
- Percentage experiments that reach a certain level of the population (say, ~5% of stable) are difficult to distinguish from shipping with regard to their impact on developers' expectations. While it's possible that such experiments might really be experimental (Trust Tokens is a recent example), the risks are similar enough to shipping that we'd see this kind of experiment as a rare exception.
The Chrome launch process differentiates between 1% and >1% stable. Should we use the same here instead of picking 5%?
----Assuming that matches others' recollection of the conversation, I'd propose adding the above requirements to https://www.chromium.org/blink/launching-features, alongside the discussion of origin trials in step 5 the of "For New Feature Incubations" section. Perhaps along the lines of:
"""If you want to gather data on the usability of your feature that an Origin Trial can help collect, or data about your feature's impact that a percentage-based experiment could support, then proceed to the "Public Experiment" stage in ChromeStatus and fill out the required fields detailing what you hope to learn. This will generate an Intent to Experiment mail that you should send to blink-dev. After receiving at least one LGTM from API Owners, you can proceed with your origin trial or percentage experiement. Collect data and respond to any issues. From here, you may wish to return to Dev Trials, proceed to Prepare to Ship, or park the feature.
Please note that these experiments are not exempt from requiring cross-functional approvals from the Chrome launch review process."""This would also require corresponding changes to the Intent to Experiment template, and to the ChromeStatus workflow as well. It also might be reasonable to create a separate page describing the experimental options in more detail, as well as the approval requirements for exceptions.WDYT?-mike
You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-d...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/CAKXHy%3DeiMG3TLA1Kuhr51MDwHFHQgiAmV7vZAy1i9df8v%3DC0tQ%40mail.gmail.com.
You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-d...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/CALjhuifW8n-wuY9XhXcCh886Nmy4L-R%3DkQK13Nv%2BqNBLCCFXRQ%40mail.gmail.com.
On Mon, Dec 14, 2020 at 10:17 AM Jochen Eisinger <joc...@chromium.org> wrote:On Mon, Dec 14, 2020 at 9:56 AM Mike West <mk...@chromium.org> wrote:Hey folks,In last week's API owners meeting, we had a short discussion around experiments that aren't appropriate for the Origin Trial framework, but instead would benefit from being rolled out to a percentage of users. I think we agreed on something like the following:
- Percentage experiments that only reach developer-focused releases (those limited to Chrome's Canary and Dev channel, for instance) do not require API owners' involvement. These releases are explicitly unstable, have a limited reach, and can't support strong security or privacy guarantees due to the rapid churn of changes. The value of %-experiments in these environments pretty clearly outweighs the minimal risk of baking in a behavior based on exposure.
- Broadly-accessible releases (Chrome's Beta and Stable channels, for instance) reach many more developers and users, and we'd like API owners to evaluate and approve percentage experiments via an explicit Intent to Experiment. This both ensures that we're evaluating the effect of an experiment on the ecosystem, and gives developers an opportunity to understand what changes might be coming their way.
--
- Percentage experiments that reach a certain level of the population (say, ~5% of stable) are difficult to distinguish from shipping with regard to their impact on developers' expectations. While it's possible that such experiments might really be experimental (Trust Tokens is a recent example), the risks are similar enough to shipping that we'd see this kind of experiment as a rare exception.
The Chrome launch process differentiates between 1% and >1% stable. Should we use the same here instead of picking 5%?I mentioned 5% here because it was the bar we set for Trust Tokens. Aligning with 1% vs >1% sounds perfectly reasonable to me.-mike----Assuming that matches others' recollection of the conversation, I'd propose adding the above requirements to https://www.chromium.org/blink/launching-features, alongside the discussion of origin trials in step 5 the of "For New Feature Incubations" section. Perhaps along the lines of:
"""If you want to gather data on the usability of your feature that an Origin Trial can help collect, or data about your feature's impact that a percentage-based experiment could support, then proceed to the "Public Experiment" stage in ChromeStatus and fill out the required fields detailing what you hope to learn. This will generate an Intent to Experiment mail that you should send to blink-dev. After receiving at least one LGTM from API Owners, you can proceed with your origin trial or percentage experiement. Collect data and respond to any issues. From here, you may wish to return to Dev Trials, proceed to Prepare to Ship, or park the feature.
Please note that these experiments are not exempt from requiring cross-functional approvals from the Chrome launch review process."""This would also require corresponding changes to the Intent to Experiment template, and to the ChromeStatus workflow as well. It also might be reasonable to create a separate page describing the experimental options in more detail, as well as the approval requirements for exceptions.WDYT?-mike
You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-d...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/CAKXHy%3DeiMG3TLA1Kuhr51MDwHFHQgiAmV7vZAy1i9df8v%3DC0tQ%40mail.gmail.com.
You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-d...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/CALjhuifW8n-wuY9XhXcCh886Nmy4L-R%3DkQK13Nv%2BqNBLCCFXRQ%40mail.gmail.com.
You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-d...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/CAKXHy%3DdV4D9ZFSE%2Bi8RoMD5pficK3Cv-vjG4Cct2RUHaUhZ72w%40mail.gmail.com.
On Mon, Dec 14, 2020 at 1:34 AM Mike West <mk...@chromium.org> wrote:On Mon, Dec 14, 2020 at 10:17 AM Jochen Eisinger <joc...@chromium.org> wrote:On Mon, Dec 14, 2020 at 9:56 AM Mike West <mk...@chromium.org> wrote:Hey folks,In last week's API owners meeting, we had a short discussion around experiments that aren't appropriate for the Origin Trial framework, but instead would benefit from being rolled out to a percentage of users. I think we agreed on something like the following:
- Percentage experiments that only reach developer-focused releases (those limited to Chrome's Canary and Dev channel, for instance) do not require API owners' involvement. These releases are explicitly unstable, have a limited reach, and can't support strong security or privacy guarantees due to the rapid churn of changes. The value of %-experiments in these environments pretty clearly outweighs the minimal risk of baking in a behavior based on exposure.
I'm ok with this as long as the percentage is, say, 50% or less. Anything more might confuse the issue.
- Broadly-accessible releases (Chrome's Beta and Stable channels, for instance) reach many more developers and users, and we'd like API owners to evaluate and approve percentage experiments via an explicit Intent to Experiment. This both ensures that we're evaluating the effect of an experiment on the ecosystem, and gives developers an opportunity to understand what changes might be coming their way.
I think we should also explicitly say that we don't think experimenting in this way is typical or recommended for features.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/CA%2BN6QZvbc7xfpF%2Bh77GGODYQfjOcih%2BmxDVTiJawez_C8YC7PA%40mail.gmail.com.
On Mon, Dec 14, 2020 at 1:34 AM Mike West <mk...@chromium.org> wrote:On Mon, Dec 14, 2020 at 10:17 AM Jochen Eisinger <joc...@chromium.org> wrote:On Mon, Dec 14, 2020 at 9:56 AM Mike West <mk...@chromium.org> wrote:Hey folks,In last week's API owners meeting, we had a short discussion around experiments that aren't appropriate for the Origin Trial framework, but instead would benefit from being rolled out to a percentage of users. I think we agreed on something like the following:
- Percentage experiments that only reach developer-focused releases (those limited to Chrome's Canary and Dev channel, for instance) do not require API owners' involvement. These releases are explicitly unstable, have a limited reach, and can't support strong security or privacy guarantees due to the rapid churn of changes. The value of %-experiments in these environments pretty clearly outweighs the minimal risk of baking in a behavior based on exposure.
I'm ok with this as long as the percentage is, say, 50% or less. Anything more might confuse the issue.
- Broadly-accessible releases (Chrome's Beta and Stable channels, for instance) reach many more developers and users, and we'd like API owners to evaluate and approve percentage experiments via an explicit Intent to Experiment. This both ensures that we're evaluating the effect of an experiment on the ecosystem, and gives developers an opportunity to understand what changes might be coming their way.
I think we should also explicitly say that we don't think experimenting in this way is typical or recommended for features.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/CA%2BN6QZu4Jse8WOQ-jgXroKE%3DFW4utD63ZJzq%3DfQDLgDkVwyD2g%40mail.gmail.com.
I think we agree, but it might be worthwhile to clarify where experiments are a legitimate tool and where they are not.IMO, they are a legitimate tool for compatibility and performance related experiments. and can be used when:a) We want to test the performance implications of (potentially web-exposed) behavior changesb) We want to get a better understanding of the compatibility implications of new features or removals(b) is something we often refer to as a "Finch rollout", as is a practice we encouraged, following an approved intent.Did you have something else than (b) in mind for experimenting with features? Or were you just saying that it's not something people should do without an intent? (Or, am I missing your intent entirely?)
On Wed, Dec 16, 2020 at 2:38 PM Yoav Weiss <yoav...@google.com> wrote:I think we agree, but it might be worthwhile to clarify where experiments are a legitimate tool and where they are not.IMO, they are a legitimate tool for compatibility and performance related experiments. and can be used when:a) We want to test the performance implications of (potentially web-exposed) behavior changesb) We want to get a better understanding of the compatibility implications of new features or removals(b) is something we often refer to as a "Finch rollout", as is a practice we encouraged, following an approved intent.Did you have something else than (b) in mind for experimenting with features? Or were you just saying that it's not something people should do without an intent? (Or, am I missing your intent entirely?)I am just saying that we should say it's an unusual situation to experiment via Finch *before* an approved I2S.
I think that if we go down this route, we should introduce a new kind of i2e, or modify the i2e to state that if you run some kind of 3p OT w/o OT usage limits (because you use Finch for that), you'll need 3 lgtms or similar.I don't think we can require an intent to ship, this would send the wrong signal to the developer community. I'd imagine us approving an i2s for the next Potassium API while it's still under development, for example, would damage the trust in this process and the potassium project.
Since nobody has objected, I think we all agree that dev and
canary (<=50%) can be experimented on without explicit api
owner approval? Just checking.
/Daniel
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/CA%2BN6QZu8zUnr9wHcifsa4-qOqMGZ-2VR9RLVPbPg-i%2BxMEYp-g%40mail.gmail.com.
Since nobody has objected, I think we all agree that dev and canary (<=50%) can be experimented on without explicit api owner approval? Just checking.