Origin trial timelines with Chrome 4 week releases

18 views
Skip to first unread message

Jason Chase

unread,
Jun 15, 2021, 2:55:33 PM6/15/21
to blink-api-owners-discuss, Chris Wilson, Johnny Stenback, Jason Robbins, Panos Astithas, experimentation-dev
Hi API Owners,

With Chrome moving to 4 week releases, that will impact timelines for origin trials.

From an I2E perspective, I think the primary impact is considering the "length" of trial:
  • The typical/recommended length for a trial is 3 milestones, currently ~18 weeks.
  • A cap of 6 milestones, total, for extended trials (without evidence of changes to avoid burn-in).
I suggest we keep the length/duration guidelines the same, in terms of elapsed time, and adjust number of milestones to match.
  • Typical length: 4 milestones, or ~16 weeks (a little shorter than currently).
  • Cap on extended trials: 9 milestones, or ~36 weeks.
We see that trials are often extended because developers aren't ready to participate and thus we don't collect enough feedback. Staying with 3 cycles would shorten the duration (in days), which would only make that problem worse.

I don't recall that the recommended 3 cycles was chosen based on deep analysis or hard/fast rules. Most likely it was a reasonable round number, given that releases were usually every 6 weeks (e.g. we don't worry that some releases are longer than others).

Questions:
  • Are there other timelines in the I2E process that we need to consider?
  • Thoughts on the suggestions above?

Thanks,
Jason

Chris Harrelson

unread,
Jun 15, 2021, 3:59:05 PM6/15/21
to Daniel Bratell, Jason Chase, blink-api-owners-discuss, Chris Wilson, Johnny Stenback, Jason Robbins, Panos Astithas, experimentation-dev
+1. This change to adjust to have a consistent total time rather than release count makes sense.



On Tue, Jun 15, 2021 at 12:53 PM Daniel Bratell <brat...@gmail.com> wrote:

I think it makes sense to increase the number of milestones, and possibly shorten the number of weeks.

In the origin trial the team will need to collect feedback, do updates, collect more feedbacks, and so on, and with shorter cycles, more such feature updates will be possible in fewer release cycles. In an ideal world,

So your suggestion seems good to me.

/Daniel

--
You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-d...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/bb2d8100-cd8b-4c18-bc33-823580e75553n%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-d...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/9b8db03a-d7fe-3c88-4592-85ca23fca05f%40gmail.com.

Johnny Stenback

unread,
Jun 15, 2021, 4:26:23 PM6/15/21
to Chris Harrelson, Daniel Bratell, Jason Chase, blink-api-owners-discuss, Chris Wilson, Jason Robbins, Panos Astithas, experimentation-dev
Likewise, +1 from me as well, tying trials to amount of time rather than # of releases (conceptually) makes much more sense to me than the other way around.

Rick Byers

unread,
Jun 15, 2021, 6:05:14 PM6/15/21
to Johnny Stenback, Chris Harrelson, Daniel Bratell, Jason Chase, blink-api-owners-discuss, Chris Wilson, Jason Robbins, Panos Astithas, experimentation-dev
+1 to just staying consistent with time (absent some argument for why it was a bad choice). Thanks for driving this Jason!

Mike West

unread,
Jul 15, 2021, 4:07:49 AM7/15/21
to Yoav Weiss, Rick Byers, Johnny Stenback, Chris Harrelson, Daniel Bratell, Jason Chase, blink-api-owners-discuss, Chris Wilson, Jason Robbins, Panos Astithas, experimentation-dev
Thanks for the suggestion, Jason!

I've documented this change at https://sites.google.com/a/chromium.org/dev/blink/origin-trials/running-an-origin-trial#TOC-How-long-do-Origin-Trials-typically-last-. Is there other documentation we should adjust as well?

-mike


On Wed, Jun 16, 2021 at 8:26 AM 'Yoav Weiss' via blink-api-owners-discuss <blink-api-ow...@chromium.org> wrote:
+1 from me as well!

--
You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-d...@chromium.org.
Reply all
Reply to author
Forward
0 new messages