Intent to Implement: Experimental Framework

[Jason Chase]

Contact emails

cha...@chromium.org, owe...@chromium.org

The experimentation-dev group exists for general discussion.

Spec (in progress)

Explainer: https://github.com/jpchase/ExperimentalFramework/blob/master/explainer.md

A formal spec will be developed in the future, as the design stabilizes.

Summary

We’re ready to start prototyping the Experimental Framework, according to the design doc shared with blink-dev.  This design is a work in progress, but in short our goal is to explore possible approaches for exposing experimental APIs in the browser, but only to specific/registered origins, for limited durations and subject to global usage caps.

Developer registration will be required to limit experiment access, but implementing the registration system is independent of the Chromium intent process.  We haven’t started work on the registration system, other than some initial design.  Currently, we expect to require developers sign up similarly to the way the Chrome Web Store works for extensions and app developers today. Additionally, developers will need to agree to provide feedback as the experiment progresses.

In addition, dk...@chromium.org is working on a proposal for updating the Blink process to handle experiments.  The new process should not allow auto-renewing of experiments without explicit discussion.  We anticipate that proposal will be shared with blink-dev in the near future.

Motivation

As explained in Doing Science On The Web, a better approach is needed for experimentation with new APIs. The ultimate goal is to enable experimentation and iteration on pre-standardization APIs, in a way that gathers meaningful feedback, but without burn-in of the APIs.

Compatibility Risk

From a web-facing API perspective, there is low risk, as very little of the framework is exposed to web developers. Specifically, the only impact is the addition of a <meta> tag variant to provide a key to the browser which will enable features for specific origins. We intend to iterate on this as we gather feedback from other browser vendors and web developers.

A specific goal of the Framework is to ensure long-term compatibility risk for new features is reduced. This will be accomplished through UMA and RAPPOR monitoring of feature use and rate limiting of feature use to ensure that experimental features do not reach usage levels which would prevent them from being withdrawn from the platform.

Ongoing technical constraints

None.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Windows, Mac, Linux, ChromeOS, Android: Yes

Android WebView: Unsure. First, it’s not yet clear that experimentation is needed within a web view.  Second, some of the anticipated infrastructure may not be compatible.

OWP launch tracking bug

https://code.google.com/p/chromium/issues/detail?id=538593

Link to entry on the feature dashboard

https://www.chromestatus.com/features/6331324939894784

Requesting approval to ship?

No

[Jochen Eisinger]

Glad to see this moving forward!

Can this framework also be used for features that interact with the network, i.e. rely on network headers? How would you feature detect in a css stylesheet?

How does this relate to navigator.connect()?

How will this framework interact with finch?

best

-jochen

[PhistucK]

On Mon, Oct 12, 2015 at 10:01 AM, Jochen Eisinger <joc...@chromium.org> wrote:

How would you feature detect in a css stylesheet?

@supports (property: value)

{

}

For selectors, media queries and @ rules, yes, it would probably be more difficult, yes (perhaps @supports-media or @supports-at should be introduced ;)).

☆PhistucK

[Jason Chase]

On Monday, October 12, 2015 at 3:02:03 AM UTC-4, Jochen Eisinger wrote:

Glad to see this moving forward!

Can this framework also be used for features that interact with the network, i.e. rely on network headers? How would you feature detect in a css stylesheet?

The current design and prototype work is focused on experimenting with JavaScript APIs.  Longer term, the goal is for the Experimental Framework to support features that interact with rendering, css, etc.  That is, if a feature needs experimentation, ideally that should be possible with the framework.  There's been some initial consideration of how that could work with rendering and css, but nothing concrete yet.  I haven't yet considered features that interact with the network.  As you suggest, feature detection is one challenge, so that web developers can provide fallback functionality. Another challenge is providing/validating API keys (or equivalent) such that a rendering/css/network feature can be enabled early enough in the pipeline.

How does this relate to navigator.connect()?

The Experimental Framework is independent of navigator.connect().  Initially, it seemed like navigator.connect() would be a good mechanism to expose experimental APIs.  However, that probably only works for JavaScript APIs. Also, the spec/use cases for navigator.connect() were still changing.  It made sense to simplify the approach to exclude integration with/use of navigator.connect().

How will this framework interact with finch?

 

Still to be determined in the long term.  For the initial prototype, there will be no interaction between Finch and the framework for running experiments.  It's an either/or decision for an API author who wishes to run an experiment.  Likely, the Finch infrastructure will be used to allow for the Experimental Framework to throttle/end experiments that exceed usage thresholds.

best

-jochen

[Jochen Eisinger]

Thanks for the responses!

If you don't use finch, will you implement a separate pipeline for the gathering of stats and for the evaluation of the experiments?

Thanks
Jochen

[Elliott Sprehn]

Is there a spec for how the meta tag works? You probably want to require that the meta comes before any <script> is inserted by the parser, otherwise you'd see the JS API magically "appear". You could still observe it in an iframe even then, but that's less likely.

[Jason Chase]

For prototyping, no interaction with Finch means that the Experimental Framework will not attempt to limit access to experiments based on field trials or % of users.  For gathering and evaluating stats, the goal is reuse existing infrastructure, such as UMA.  We still need to flesh out the requirements for experiment stats, so the design is still pending.

Thanks,

Jason

[Jason Chase]

We don't have a spec for how the meta works (just a placeholder). I think we need to get further with the prototyping to figure out the behaviour.

We have done some investigation into implementing an [ExperimentEnabled] IDL attribute, to support feature detection, similar to [RuntimeEnabled].  Initial results are promising, but more work is needed.  The design will be updated once there are more concrete ideas to address the questions below.

Thanks,

Jason

On Wednesday, October 21, 2015 at 9:55:50 AM UTC-4, Philip Jägenstedt wrote:

We discussed this very question in the design doc, starting with my question "Making it part of the markup does not sound tricky," but it wasn't really resolved, so I'm still curious. Making it all work with RuntimeEnabledFeatures would be sweet, but that's simply not compatible with doing it in markup.

[Kentaro Hara]

Imagine that we have [ExperimentEnabled] on window.xxx.

The current design is:

- when the experiment is enabled, window.xxx exists and works as a valid web API.

- when the experiment is disabled, window.xxx does not exist.

The latter requires to hide the DOM attribute/method from the window object, which adds a lot of complexity to the binding layer (I'm reviewing this CL).

Alternately, wouldn't the following behavior be acceptable?

- when the experiment is enabled, window.xxx exists and works as a valid web API.

- when the experiment is disabled, window.xxx exists but returns undefined with a kind error message.

--

Kentaro Hara, Tokyo, Japan

[Rick Byers]

Is the plan that the experimental APIs will always have unpredictable names?  If so, the risk to exposing the properties when the experiment is disabled seems pretty low to me.  I agree that it would be unfortunate to have to add this much complexity to the bindings.  

Feature detection would be a little awkward I assume (instead of "'feature_x1591' in window" you'd probably want to test against undefined explicitly: "window.feature_x1591 !== undefined", right?).  Perhaps the properties for experimental APIs should always be unenumerable to further reduce the risk of impacting sites that haven't explicitly enabled them?

I assume interface objects could be handled in a similar way without too much trouble, right?  I.e. 'window.MyClass_b87a4' would have the value undefined when the experiment is disabled, yes 'new MyClass_b87a4()' would work when it's enabled.

That said, there are other reasons to want the ability to add/remove bindings at runtime, so if there is a good way to do it properly, perhaps it's worth the effort?

[Ian Clelland]

On Tue, Dec 15, 2015 at 9:22 PM, Kentaro Hara <har...@chromium.org> wrote:

Imagine that we have [ExperimentEnabled] on window.xxx.

The current design is:

- when the experiment is enabled, window.xxx exists and works as a valid web API.

- when the experiment is disabled, window.xxx does not exist.

The latter requires to hide the DOM attribute/method from the window object, which adds a lot of complexity to the binding layer (I'm reviewing this CL).

Alternately, wouldn't the following behavior be acceptable?

- when the experiment is enabled, window.xxx exists and works as a valid web API.

- when the experiment is disabled, window.xxx exists but returns undefined with a kind error message.

That is certainly easier to implement, and fulfils *most* of the conditions that we'd need so that developers can do feature detection. window.xxx returning undefined is indistinguishable from not existing in most cases, but there are still ways to detect that it's there. In particular, I'd really like for us to be able to remove the property from it's parent's propertly list, so that:

  - window.hasOwnProperty("xxx") returns false

  - ("xxx" in window) returns false

  - xxx doesn't show up in a loop of "for (property in window)"

etc.

I really do like the idea of logging to the console on first use, some message that the API is experimental, and not available unless the origin enables it. Kind of the inverse of a deprecation message, but to encourage developers to sign up for the experiment, rather than discourage them from using the API.

[Ian Clelland]

On Tue, Dec 15, 2015 at 9:53 PM, Rick Byers <rby...@chromium.org> wrote:

Is the plan that the experimental APIs will always have unpredictable names?  If so, the risk to exposing the properties when the experiment is disabled seems pretty low to me.  I agree that it would be unfortunate to have to add this much complexity to the bindings.  

No -- we threw that idea around for a while -- it definitely does get the message across that you're going to have to update your site's code every few weeks, but in the end I think we felt like it was more important that the API we expose "feel" as much like the proposed final API as possible.

Using the actual proposed property names also reduces developer friction. It makes it more likely that developers who want to try out code that they copy, from a blog post, for instance, have a positive experience with the API, and don't get stuck and frustrated because they missed one place where they needed to customize the property name.

 

Feature detection would be a little awkward I assume (instead of "'feature_x1591' in window" you'd probably want to test against undefined explicitly: "window.feature_x1591 !== undefined", right?).  Perhaps the properties for experimental APIs should always be unenumerable to further reduce the risk of impacting sites that haven't explicitly enabled them?

I assume interface objects could be handled in a similar way without too much trouble, right?  I.e. 'window.MyClass_b87a4' would have the value undefined when the experiment is disabled, yes 'new MyClass_b87a4()' would work when it's enabled.

That said, there are other reasons to want the ability to add/remove bindings at runtime, so if there is a good way to do it properly, perhaps it's worth the effort?

Definitely (a couple of us were just talking about about that bug last week in this context, actually). And I'm sure that once we have the capability, we'll find all kinds of new uses for it. It may not be necessary for an MVP of this framework, but I'd really like to see us aim for that kind of implementation in the long run.

[Kentaro Hara]

On Wed, Dec 16, 2015 at 12:59 PM, Ian Clelland <icle...@chromium.org> wrote:

On Tue, Dec 15, 2015 at 9:53 PM, Rick Byers <rby...@chromium.org> wrote:

Is the plan that the experimental APIs will always have unpredictable names?  If so, the risk to exposing the properties when the experiment is disabled seems pretty low to me.  I agree that it would be unfortunate to have to add this much complexity to the bindings.  

No -- we threw that idea around for a while -- it definitely does get the message across that you're going to have to update your site's code every few weeks, but in the end I think we felt like it was more important that the API we expose "feel" as much like the proposed final API as possible.

Using the actual proposed property names also reduces developer friction. It makes it more likely that developers who want to try out code that they copy, from a blog post, for instance, have a positive experience with the API, and don't get stuck and frustrated because they missed one place where they needed to customize the property name.

 

Feature detection would be a little awkward I assume (instead of "'feature_x1591' in window" you'd probably want to test against undefined explicitly: "window.feature_x1591 !== undefined", right?).  Perhaps the properties for experimental APIs should always be unenumerable to further reduce the risk of impacting sites that haven't explicitly enabled them?

I assume interface objects could be handled in a similar way without too much trouble, right?  I.e. 'window.MyClass_b87a4' would have the value undefined when the experiment is disabled, yes 'new MyClass_b87a4()' would work when it's enabled.

That said, there are other reasons to want the ability to add/remove bindings at runtime, so if there is a good way to do it properly, perhaps it's worth the effort?

Definitely (a couple of us were just talking about about that bug last week in this context, actually). And I'm sure that once we have the capability, we'll find all kinds of new uses for it. It may not be necessary for an MVP of this framework, but I'd really like to see us aim for that kind of implementation in the long run.

Hmm. I guess it will confuse developers if we have the ability to add/remove DOM bindings at runtime.

Imagine the following scenario:

(1) Element.xxx is added.

(2) e = document.createElement("div")

(3) e.xxx exists and works as a valid web API.

(4) Element.xxx is removed.

(5) e2 = document.createElement("div")

(6) e2.xxx does not exist. e.xxx exists but does not work. This would be confusing.

To prevent the confusion, whether the DOM binding is enabled or disabled should be determined when the JS context is initialized and it should not change until the context terminates. What do you think?

[Rick Byers]

On Wed, Dec 16, 2015 at 3:24 AM, Harald Alvestrand <h...@google.com> wrote:

Isn't the confusion we get from add/remove bindings exactly the same confusion as libraries / users can cause to themselves by installing JS shims on the DOM objects?

If so, the users' feet may already be so full of holes that adding another footgun doesn't increase the pain by any noticeable margin.

On Wed, Dec 16, 2015 at 5:11 AM, Kentaro Hara <har...@chromium.org> wrote:

On Wed, Dec 16, 2015 at 12:59 PM, Ian Clelland <icle...@chromium.org> wrote:

On Tue, Dec 15, 2015 at 9:53 PM, Rick Byers <rby...@chromium.org> wrote:

Is the plan that the experimental APIs will always have unpredictable names?  If so, the risk to exposing the properties when the experiment is disabled seems pretty low to me.  I agree that it would be unfortunate to have to add this much complexity to the bindings.  

No -- we threw that idea around for a while -- it definitely does get the message across that you're going to have to update your site's code every few weeks, but in the end I think we felt like it was more important that the API we expose "feel" as much like the proposed final API as possible.

Using the actual proposed property names also reduces developer friction. It makes it more likely that developers who want to try out code that they copy, from a blog post, for instance, have a positive experience with the API, and don't get stuck and frustrated because they missed one place where they needed to customize the property name.

Ok, I see the upsides there.  I assume people are OK with the risk that, when it comes time to actually ship an API that was previously experimental, we might (in the worst case) need to rename the API in order to be compatible with code left over on the web designed for the experimental version, right?

Feature detection would be a little awkward I assume (instead of "'feature_x1591' in window" you'd probably want to test against undefined explicitly: "window.feature_x1591 !== undefined", right?).  Perhaps the properties for experimental APIs should always be unenumerable to further reduce the risk of impacting sites that haven't explicitly enabled them?

I assume interface objects could be handled in a similar way without too much trouble, right?  I.e. 'window.MyClass_b87a4' would have the value undefined when the experiment is disabled, yes 'new MyClass_b87a4()' would work when it's enabled.

That said, there are other reasons to want the ability to add/remove bindings at runtime, so if there is a good way to do it properly, perhaps it's worth the effort?

Definitely (a couple of us were just talking about about that bug last week in this context, actually). And I'm sure that once we have the capability, we'll find all kinds of new uses for it. It may not be necessary for an MVP of this framework, but I'd really like to see us aim for that kind of implementation in the long run.

Hmm. I guess it will confuse developers if we have the ability to add/remove DOM bindings at runtime.

Imagine the following scenario:

(1) Element.xxx is added.

(2) e = document.createElement("div")

(3) e.xxx exists and works as a valid web API.

(4) Element.xxx is removed.

(5) e2 = document.createElement("div")

(6) e2.xxx does not exist. e.xxx exists but does not work. This would be confusing.

To prevent the confusion, whether the DOM binding is enabled or disabled should be determined when the JS context is initialized and it should not change until the context terminates. What do you think?

Do we delay initializing the JS context until after we have received/parsed at least the HTML head?  Even if we do, is it important to support the scenario where the <meta> tag is added by JS at runtime?

Perhaps we can simplify this / remove footguns by saying that we'll support dynamic addition of APIs only, not dynamic removal?

If it helps, I believe IE/Edge have found a facility like this to be useful in their engine.  I've heard a bit about how they can dynamically enable the touch event APIs (satisfying popular feature detection like "'ontouchstart' in window") as needed on a site-by-site basis.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Ian Clelland]

On Wed, Dec 16, 2015 at 9:19 AM, Rick Byers <rby...@chromium.org> wrote:

On Wed, Dec 16, 2015 at 3:24 AM, Harald Alvestrand <h...@google.com> wrote:

Isn't the confusion we get from add/remove bindings exactly the same confusion as libraries / users can cause to themselves by installing JS shims on the DOM objects?

If so, the users' feet may already be so full of holes that adding another footgun doesn't increase the pain by any noticeable margin.

On Wed, Dec 16, 2015 at 5:11 AM, Kentaro Hara <har...@chromium.org> wrote:

On Wed, Dec 16, 2015 at 12:59 PM, Ian Clelland <icle...@chromium.org> wrote:

On Tue, Dec 15, 2015 at 9:53 PM, Rick Byers <rby...@chromium.org> wrote:

Is the plan that the experimental APIs will always have unpredictable names?  If so, the risk to exposing the properties when the experiment is disabled seems pretty low to me.  I agree that it would be unfortunate to have to add this much complexity to the bindings.  

No -- we threw that idea around for a while -- it definitely does get the message across that you're going to have to update your site's code every few weeks, but in the end I think we felt like it was more important that the API we expose "feel" as much like the proposed final API as possible.

Using the actual proposed property names also reduces developer friction. It makes it more likely that developers who want to try out code that they copy, from a blog post, for instance, have a positive experience with the API, and don't get stuck and frustrated because they missed one place where they needed to customize the property name.

Ok, I see the upsides there.  I assume people are OK with the risk that, when it comes time to actually ship an API that was previously experimental, we might (in the worst case) need to rename the API in order to be compatible with code left over on the web designed for the experimental version, right?

Feature detection would be a little awkward I assume (instead of "'feature_x1591' in window" you'd probably want to test against undefined explicitly: "window.feature_x1591 !== undefined", right?).  Perhaps the properties for experimental APIs should always be unenumerable to further reduce the risk of impacting sites that haven't explicitly enabled them?

I assume interface objects could be handled in a similar way without too much trouble, right?  I.e. 'window.MyClass_b87a4' would have the value undefined when the experiment is disabled, yes 'new MyClass_b87a4()' would work when it's enabled.

That said, there are other reasons to want the ability to add/remove bindings at runtime, so if there is a good way to do it properly, perhaps it's worth the effort?

Definitely (a couple of us were just talking about about that bug last week in this context, actually). And I'm sure that once we have the capability, we'll find all kinds of new uses for it. It may not be necessary for an MVP of this framework, but I'd really like to see us aim for that kind of implementation in the long run.

Hmm. I guess it will confuse developers if we have the ability to add/remove DOM bindings at runtime.

Imagine the following scenario:

(1) Element.xxx is added.

(2) e = document.createElement("div")

(3) e.xxx exists and works as a valid web API.

(4) Element.xxx is removed.

(5) e2 = document.createElement("div")

(6) e2.xxx does not exist. e.xxx exists but does not work. This would be confusing.

Yes, that would be confusing, and we should avoid that situation. Once we've allowed execution to proceed to the point where e.xxx has been accessed, and it has been observed to have a value other than 'undefined', then we have committed to that API, and we shouldn't arbitrarily remove it.

To prevent the confusion, whether the DOM binding is enabled or disabled should be determined when the JS context is initialized and it should not change until the context terminates. What do you think?

Do we delay initializing the JS context until after we have received/parsed at least the HTML head?  Even if we do, is it important to support the scenario where the <meta> tag is added by JS at runtime?

That's a nice-to-have, but I don't think it's necessary for an MVP. And I don't think we want to support *removing* the <meta> tags at runtime - or at least we can declare that as having potentially undefined results.

 

Perhaps we can simplify this / remove footguns by saying that we'll support dynamic addition of APIs only, not dynamic removal?

That's all that the experimental framework is concerned with, as far as I know. The only case that I have for removal of APIs is in something like a deprecation scenario -- I'm imagining a case where JS in a page accesses window.requestAnimationFrame, and so we dynamically remove webkitRequestAnimationFrame from window immediately, before the page has a chance to even see it.

(This isn't part of the EF at all, just something else we could conceivably do with this mechanism)

[Rick Byers]

Even if we could, it wouldn't work for navigation cases requiring a shared renderer (eg. window.open).

On Wed, Dec 16, 2015 at 3:40 PM, Philip Jägenstedt <phi...@opera.com> wrote:

Is it entirely hopeless to build the experimental framework on top of RuntimeEnabledFeatures by pre-scanning for the <meta> tag somewhere in the networking code before the renderer process is even started, or at least while there's time to kill and start another renderer instead?

Are any <meta http-equiv="x"> currently handled outside of Blink?

[Marijn Kruisselbrink]

If having the API keys in the <meta> tag turns out to be a problem, it might also be an option to instead pass the API keys as response headers? I think that was also originally considered, but I don't quite remember what the downsides of that were. Hopefully response headers are available early enough to hide APIs?

On Wed, Dec 16, 2015 at 1:16 PM, Philip Jägenstedt <phi...@opera.com> wrote:

Wouldn't it be OK to simply require that the <meta> be in the first resource that you fetch when entering a URL and pressing enter? (The question is moot if it isn't actually possible, of course.)

[Vincent Scheib]

Response headers don't let me try a cute example and host on github. I think it would add friction to enthusiasts trying out an API.

[Ian Clelland]

API Keys in response headers are still something that I want to try -- I think it'll be the best method for enabling experimental APIs in Service Worker.

It's definitely not sufficient, though -- many developers don't have any control over the server platform, so we need an in-html solution for them.

[Jeremy Roman]

Can't we still end up hitting our max number of renderer processes (IIUC, such a limit exists, and is especially small on Android)? What do we do then?

On Wed, Dec 16, 2015 at 4:51 PM, Philip Jägenstedt <phi...@opera.com> wrote:

Right, HTTP headers only doesn't seem sufficient, but if HTTP headers will part of it, then using <meta http-equiv> comes quite naturally. So, are any <meta http-equiv> headers recognized before the renderer is started?

[Elliott Sprehn]

How does process limit relate to this? This should just be toggling the visibility of APIs in the v8::Context. I do think we need to make this like charset where it must be at the start of the <head> before any script runs in that context.

[Ian Clelland]

That makes sense, as far as creating an execution environment that developers can reason about. We could disable adding any new APIs via the experimental framework as soon as a script or link tag is encountered, for instance.

That may mean that it's no longer possible to introduce a JavaScript API to add keys, say, from a service worker, but we still have HTTP headers for that if we need them.

[Jeremy Roman]

On Wed, Dec 16, 2015 at 7:47 PM, Elliott Sprehn <esp...@chromium.org> wrote:

How does process limit relate to this? This should just be toggling the visibility of APIs in the v8::Context. I do think we need to make this like charset where it must be at the start of the <head> before any script runs in that context.

I was responding not to the original proposal, but to philipj's comment that this should just set the (process-global) RuntimeEnabledFeatures flags, by detecting the enabled experiments before the renderer starts. I don't think that approach would work, if the new navigation must (due to the limit) be hosted in an existing renderer process, which hosts other pages with other experiment <meta> tags.

[PhistucK]

What about experimental HTML or network features?

For example, a new attribute - <html make-page-awesome="on">?

I guess for those, you must use HTTP headers...

☆PhistucK

You received this message because you are subscribed to the Google Groups "experimentation-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to experimentation...@chromium.org.
To post to this group, send email to experimen...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/experimentation-dev/CAK_TSXJyH6ZOfXac81AqJCihh_bMZsG69e%3DhNFHrfvXjaNS4zg%40mail.gmail.com.

[Kentaro Hara]

(I chatted with dhnishi@ offline.)

Just to sum up the discussion:

Imagine that we introduce a new interface X with [ExperimentEnabled]. In order to determine whether X should be exposed to DOMWindow or not (i.e., whether DOMWindow.X should exist or not), the list of experimentally-enabled interfaces must be available at the point where DOMWindow is initialized. This means that the list of experimentally-enabled interfaces must be available at the point where JS's context is initialized. We've discussed many ways to realize the behavior (e.g., use a <meta> tag, use HTTP's response header etc) but haven't yet reached agreement. We need more discussion.

On the other hand, this Intent-to-Implement LGTM assuming that it's implemented under a runtime flag. At the moment, we can just unconditionally expose DOMWindow.X when the runtime flag is enabled. Once we have reached agreement on the above issue, we can update the implementation.