--
You received this message because you are subscribed to the Google Groups "experimentation-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation-dev+unsub...@chromium.org.
To post to this group, send email to experimentation-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/experimentation-dev/7feff4a4-5043-4471-9da1-194369301222%40chromium.org.
I agree that it makes a lot of sense to revise this criteria as we have now learned more, and definitely agree that more important than the specific constants is a clearly expressed policy and a system that does not apply any judgement or discretion.Alex's suggestion of a 14 day threshold makes sense to me. As Rick says, the automatic disabling is the most powerful lever we have to prevent burn in anyway, and I don't see almost any increased risk of a few additional days over the target threshold.Chris Harrelson has also confirmed in blink-dev that the 0.03% deprecation threshold is explicitly out of date. This further supports updating it.Specifically, as a starting point I'd suggest capping 14-day average usage at 0.5% of page loads.
Half a percent feels about right to me as preventing the feature from having too significant positive impact on Chrome users such that other browsers feel forced to fast follow on an unproven design, but also large enough to allow developers to experiment and collect statistically significant data which is often vital, such as in the case of Navigation Preload.
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation-dev+unsubscribe...@chromium.org.
To post to this group, send email to experimentation-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/experimentation-dev/7feff4a4-5043-4471-9da1-194369301222%40chromium.org.
--
You received this message because you are subscribed to the Google Groups "experimentation-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation-dev+unsub...@chromium.org.
To post to this group, send email to experimentation-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/experimentation-dev/c99dd2ed-bbd6-49b4-a982-789d99e31a74%40chromium.org.
Something I'd point out here is that the "burn in" of an API is not necessarily only related to its global usage.Let's say Facebook is responsible for X% of global page views (let's assume X is higher than any threshold we're talking about today). If Facebook launches an origin trial to 90% of its users, but nobody else uses the feature I'd argue that the burn in is minimal. First, since 10% of fb users currently manage without the feature, it can't become something so critical that we depend on it. Second, we would launch such an experiment with active involvement with chrome -- we're likely to be able to work through any issues with the experiment amicably. On Rick's openness point, we would never want to push browsers to standardize a substandard API just to get a few extra months of user benefit from it. We've been bitten way too many times by bad APIs to go down this route :-)On the other hand, let's say that 90 sites that are 1/100th the size of Facebook launch the feature to 100% of their users. I'd argue that this feature is at risk of being burned in. Now there are 90 sites that aren't testing their code without the existence of this origin trial.IMHO the key to avoiding burn in is forcing a percentage of users to not be in the experiment. I'm not sure of the best technical way to do that. One way would be to ignore the opt-in on N% of page views, but this gets tricky because it can really mess with A/B tests (most of our A/B tests are based on a hash of the user ID, so it's hard to A/B test something which has its own opt-out mechanism).Perhaps a simple solution here is to mandate that any site which wishes to do an origin trial that would pass 0.1% of pageviews engage in a deeper relationship where they would detail their plan to avoid burn-in and commit to maintaining the openness of the platform.
-b
On Friday, February 10, 2017 at 10:17:51 AM UTC-8, Alex Komoroske wrote:Hey folks,At BlinkOn7 there was a break-out discussion about Origin Trials and the criteria for the global "fuse" blowing. There are have also been a number of recent discussions about performance-related features that are primarily useful to big, mature, savvy sites, but whose use could easily blow the fuse accidentally. We're stuck in a position where we don't know if the design of the API will actually have the right performance characteristics in the wild, but the only way to get that feedback (experimentation in the wild) is closed off.In some ways, the criteria are working as intended (preventing a large customer from accidentally "burning in" the API). But in other ways, it seems like it's being too restrictive (not allowing any big customers, even when they want to be responsible and do a short, time-limited trial).Is there a way to modify the "fuse" criteria, so that even large customers can do experiments-- as long as they're short-term time limited? Today the criteria is effectively "the fuse blows if at any moment there is more than the deprecation threshold of uses." But what if it was something like "the fuse blows if the median daily usage across the past 14 days is greater than the deprecation threshold?" That way, large customers could do experiments for whatever number of users they needed, but if they ran the experiment for more than 7 days, the fuse would blow.Thoughts?--Alex
--
You received this message because you are subscribed to the Google Groups "experimentation-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation-dev+unsub...@chromium.org.
To post to this group, send email to experimentation-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/experimentation-dev/145d5e13-2a6f-4f05-b9cc-586244ff9c2c%40chromium.org.
In practice the risk is roughly proportional to how widely used a service is and how badly users depend on it--that's the strongest predictor of how loud the blow-back will be if a site breaks: how many angry bugs will be filled on crbug, how much loud complaining there will be, etc.
-b
--
You received this message because you are subscribed to the Google Groups "experimentation-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation-dev+unsub...@chromium.org.
To post to this group, send email to experimentation-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/experimentation-dev/CABgOVaJTWyWu3GQOJGrcPw49ovzhMfrgcUVo9xpshCyvqtEu_Q%40mail.gmail.com.
On Tue, Feb 14, 2017 at 12:42 PM, Ben Maurer <ben.m...@gmail.com> wrote:On Tue, Feb 14, 2017 at 12:31 PM, Alex Komoroske <komo...@chromium.org> wrote:In practice the risk is roughly proportional to how widely used a service is and how badly users depend on it--that's the strongest predictor of how loud the blow-back will be if a site breaks: how many angry bugs will be filled on crbug, how much loud complaining there will be, etc.My point here is that a breakage takes two steps:1) A site must take on a dependency to the feature in question2) The feature must be removed"don't let origin trials go beyond 0.5% of page views" is targeted at step 2. It says "reduce the damage caused by removing the feature".I'm suggesting we target step 1 for the larger sites. If a large site is only ever able to roll out an origin trial to 90% of chrome users they'll have to have a plan in place for the remaining 10%. No site is going to deploy a feature that will cause 10% of their users to file crbugs the day they launch it. Regardless of how big that 90% of users who have the feature is you know that the site has a vested interest in keeping their site usable if the trial is turned off.Ah, that's a great point.We've designed OT to date specifically for the general case that we can enforce easily, but I wonder if there are other approaches we can take to enforce a 90/10 for the use cases of large sites.
-b
--
You received this message because you are subscribed to the Google Groups "experimentation-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation-dev+unsubscribe...@chromium.org.
To post to this group, send email to experimentation-dev@chromium.org.
-b
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation-dev+unsub...@chromium.org.
To post to this group, send email to experimentation-dev@chromium.org.
-b
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation-dev+unsubscribe...@chromium.org.
To post to this group, send email to experimentation-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/experimentation-dev/CABgOVaJTWyWu3GQOJGrcPw49ovzhMfrgcUVo9xpshCyvqtEu_Q%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "experimentation-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation-dev+unsub...@chromium.org.
To post to this group, send email to experimentation-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/experimentation-dev/07617bfa-31a3-4f91-b943-6039e6c99d87%40chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/experimentation-dev/07617bfa-31a3-4f91-b943-6039e6c99d87%40chromium.org.
--
You received this message because you are subscribed to the Google Groups "experimentation-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation-dev+unsub...@chromium.org.
To post to this group, send email to experimentation-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/experimentation-dev/CAK_TSXL0ZvOR_Qnm1Rb5OXDhmk%2Bq-r67qdqn_OefkJER5GE49g%40mail.gmail.com.
-b
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation-dev+unsub...@chromium.org.
To post to this group, send email to experimentation-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/experimentation-dev/CABgOVaJTWyWu3GQOJGrcPw49ovzhMfrgcUVo9xpshCyvqtEu_Q%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "experimentation-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation-dev+unsub...@chromium.org.
To post to this group, send email to experimentation-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/experimentation-dev/07617bfa-31a3-4f91-b943-6039e6c99d87%40chromium.org.
--
You received this message because you are subscribed to the Google Groups "experimentation-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation-dev+unsub...@chromium.org.
To post to this group, send email to experimentation-dev@chromium.org.
-b
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation-dev+unsubscribe...@chromium.org.
To post to this group, send email to experimentation-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/experimentation-dev/CABgOVaJTWyWu3GQOJGrcPw49ovzhMfrgcUVo9xpshCyvqtEu_Q%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "experimentation-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation-dev+unsubscribe...@chromium.org.
To post to this group, send email to experimentation-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/experimentation-dev/07617bfa-31a3-4f91-b943-6039e6c99d87%40chromium.org.
--
You received this message because you are subscribed to the Google Groups "experimentation-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation-dev+unsubscribe...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/experimentation-dev/CAK_TSXL0ZvOR_Qnm1Rb5OXDhmk%2Bq-r67qdqn_OefkJER5GE49g%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "experimentation-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation-dev+unsub...@chromium.org.
To post to this group, send email to experimentation-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/experimentation-dev/be946de7-0da8-47cd-8952-ccd679b825bc%40chromium.org.