Origin trial timelines with Chrome 4 week releases

21 views
Skip to first unread message

Jason Chase

unread,
Jun 15, 2021, 2:55:32 PM6/15/21
to blink-api-owners-discuss, Chris Wilson, Johnny Stenback, Jason Robbins, Panos Astithas, experimentation-dev
Hi API Owners,

With Chrome moving to 4 week releases, that will impact timelines for origin trials.

From an I2E perspective, I think the primary impact is considering the "length" of trial:
  • The typical/recommended length for a trial is 3 milestones, currently ~18 weeks.
  • A cap of 6 milestones, total, for extended trials (without evidence of changes to avoid burn-in).
I suggest we keep the length/duration guidelines the same, in terms of elapsed time, and adjust number of milestones to match.
  • Typical length: 4 milestones, or ~16 weeks (a little shorter than currently).
  • Cap on extended trials: 9 milestones, or ~36 weeks.
We see that trials are often extended because developers aren't ready to participate and thus we don't collect enough feedback. Staying with 3 cycles would shorten the duration (in days), which would only make that problem worse.

I don't recall that the recommended 3 cycles was chosen based on deep analysis or hard/fast rules. Most likely it was a reasonable round number, given that releases were usually every 6 weeks (e.g. we don't worry that some releases are longer than others).

Questions:
  • Are there other timelines in the I2E process that we need to consider?
  • Thoughts on the suggestions above?

Thanks,
Jason

Daniel Bratell

unread,
Jun 15, 2021, 3:53:42 PM6/15/21
to Jason Chase, blink-api-owners-discuss, Chris Wilson, Johnny Stenback, Jason Robbins, Panos Astithas, experimentation-dev

I think it makes sense to increase the number of milestones, and possibly shorten the number of weeks.

In the origin trial the team will need to collect feedback, do updates, collect more feedbacks, and so on, and with shorter cycles, more such feature updates will be possible in fewer release cycles. In an ideal world,

So your suggestion seems good to me.

/Daniel

--
You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-d...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/bb2d8100-cd8b-4c18-bc33-823580e75553n%40chromium.org.

Chris Harrelson

unread,
Jun 15, 2021, 3:59:05 PM6/15/21
to Daniel Bratell, Jason Chase, blink-api-owners-discuss, Chris Wilson, Johnny Stenback, Jason Robbins, Panos Astithas, experimentation-dev
+1. This change to adjust to have a consistent total time rather than release count makes sense.



Johnny Stenback

unread,
Jun 15, 2021, 4:26:24 PM6/15/21
to Chris Harrelson, Daniel Bratell, Jason Chase, blink-api-owners-discuss, Chris Wilson, Jason Robbins, Panos Astithas, experimentation-dev
Likewise, +1 from me as well, tying trials to amount of time rather than # of releases (conceptually) makes much more sense to me than the other way around.

Rick Byers

unread,
Jun 15, 2021, 6:05:14 PM6/15/21
to Johnny Stenback, Chris Harrelson, Daniel Bratell, Jason Chase, blink-api-owners-discuss, Chris Wilson, Jason Robbins, Panos Astithas, experimentation-dev
+1 to just staying consistent with time (absent some argument for why it was a bad choice). Thanks for driving this Jason!

Yoav Weiss

unread,
Jun 16, 2021, 2:26:41 AM6/16/21
to Rick Byers, Johnny Stenback, Chris Harrelson, Daniel Bratell, Jason Chase, blink-api-owners-discuss, Chris Wilson, Jason Robbins, Panos Astithas, experimentation-dev

Mike West

unread,
Jul 15, 2021, 4:07:49 AM7/15/21
to Yoav Weiss, Rick Byers, Johnny Stenback, Chris Harrelson, Daniel Bratell, Jason Chase, blink-api-owners-discuss, Chris Wilson, Jason Robbins, Panos Astithas, experimentation-dev
Reply all
Reply to author
Forward
0 new messages