Intent to Remove: Appcache in non-secure contexts.

1,216 views
Skip to first unread message

Mike West

unread,
Feb 2, 2018, 10:14:30 AM2/2/18
to blink-dev

Primary eng (and PM) emails

mk...@chromium.org


Link to “Intent to Deprecate” thread

https://groups.google.com/a/chromium.org/d/msg/blink-dev/UKF8cK0EwMI/NLhsIrs-AQAJ


We deprecated this in early 2016, and failed to follow through with the removal. I'd like to try again, in support of Mozilla's stab at doing the same.


Summary

I'd like to make several changes:


1. Mark `window.applicationCache`, and the `ApplicationCache` and `ApplicationCacheErrorEvent` interfaces as `[SecureContext]`.

2. Prevent non-secure contexts from declaring manifests via a `manifest` attribute.

3. Prevent HTTP documents from using manifests they've registered.

4. Add an enterprise policy to remove these restrictions, with some reasonable advertised lifetime.


#3 is a little tricky, as we don't actually have the context we need in the browser-side appcache handlers to do a full secure context check (e.g. walk the frame tree). Given that the eventual goal is to get rid of appcache everywhere, this seems like a reasonable tradeoff between the letter and spirit of "[SecureContext]".


Motivation

HTML spells out a number of security concerns around appcache: https://html.spec.whatwg.org/multipage/offline.html#security-concerns-with-offline-applications-caches. In particular, the FALLBACK mechanism (and Chrome's proprietary CHROMIUM-INTERCEPT mechanism to an even greater degree) puts sites delivered over non-secure connections at real risk of persistent MitM attack. evn@ spelled this out more clearly in https://readable-email.org/list/whatwg/topic/appcache-content-type-security-considerations.


TLS allows us to ensure that the server we think we're talking to is actually the one asserting a manifest for itself, and that we're not being tricked by the Delicious But Evil Coffee Shop(tm) that we happened to be sitting in yesterday.


Interoperability and Compatibility Risk

Edge: No signals

Firefox: Shipping the restriction in Firefox 62, https://groups.google.com/d/msg/mozilla.dev.platform/qLTTpdzcDkw/WKJeq-4HAQAJ.

Safari: No signals.


I didn't see response to Firefox's announcement from Edge or Safari folks. I poked relevant people on Twitter; let's see what they say.


Alternative implementation suggestion for web developers

Developers who wish to continue using appcache can migrate their sites to TLS. While doing so, they could also upgrade to service workers, which are the right choice for the medium-term.


Usage information from UseCounter

`window.applicationCache` is accessed from a non-secure context in ~0.15% of page views. Digging through the 445 hits in HTTP Archive[1], the vast majority of this usage appears to be feature-detection, along the lines of Modernizr's `'applicationCache' in window`, which would make the `[SecureContext]` assignment fairly safe.


Appcache manifest selection is triggered from a non-secure context in ~0.08% of page views. I spent some time this morning digging through the 312 hits in HTTP Archive[2], with interesting results (raw data available at https://docs.google.com/spreadsheets/d/1mGUvSKyF_DOhkx9PK_NawRCVMgA416R0jJTZH1YOgU4/edit?usp=sharing):

The usage is certainly non-trivial, but I'd posit that the security risks posed by non-secure usage of the feature should push us towards lowering the bar. Moreover, the prevalence of nested appcache contexts suggests to me that the actual breakage experienced by users would be lower than the raw numbers would suggest.


I've added these metrics to UKM, but I don't have any data yet. As 66 rolls through the channels, we can verify the data I've gathered here to see if hotspots crop up that we weren't expecting.


[1]:

```

SELECT

url, JSON_EXTRACT(payload, '$._blinkFeatureFirstUsed.Features.ApplicationCacheAPIInsecureOrigin') AS feature FROM [httparchive:har.2018_01_15_chrome_pages] HAVING feature IS NOT NULL ORDER BY url

```


[2]:

```

SELECT

url, JSON_EXTRACT(payload, '$._blinkFeatureFirstUsed.Features.ApplicationCacheManifestSelectInsecureOrigin') AS feature FROM [httparchive:har.2018_01_15_chrome_pages] HAVING feature IS NOT NULL ORDER BY url

```


OWP launch tracking bug

https://crbug.com/588931


Entry on the feature dashboard

https://www.chromestatus.com/feature/5714236168732672


Thanks! WDYT?


-mike

Yoav Weiss

unread,
Feb 3, 2018, 8:56:12 AM2/3/18
to Mike West, blink-dev
I'm super excited about this work! :)

Killing AppCache (entirely) will make our preloadScanner faster and will significantly help sending out requests earlier.

One interesting question: once this removal is in place, should we (and could we) make non-secure-context page-loads faster, by not forcing them to wait for the document element to be available?

On the one hand - faster is better. On the other, that could create the wrong incentives when trying to move people towards secure contexts.

On Fri, Feb 2, 2018 at 4:14 PM Mike West <mk...@chromium.org> wrote:

Primary eng (and PM) emails

mk...@chromium.org


Link to “Intent to Deprecate” thread

https://groups.google.com/a/chromium.org/d/msg/blink-dev/UKF8cK0EwMI/NLhsIrs-AQAJ


We deprecated this in early 2016, and failed to follow through with the removal. I'd like to try again, in support of Mozilla's stab at doing the same.


Summary

I'd like to make several changes:


1. Mark `window.applicationCache`, and the `ApplicationCache` and `ApplicationCacheErrorEvent` interfaces as `[SecureContext]`.

2. Prevent non-secure contexts from declaring manifests via a `manifest` attribute.

3. Prevent HTTP documents from using manifests they've registered.

4. Add an enterprise policy to remove these restrictions, with some reasonable advertised lifetime.


#3 is a little tricky, as we don't actually have the context we need in the browser-side appcache handlers to do a full secure context check (e.g. walk the frame tree). Given that the eventual goal is to get rid of appcache everywhere, this seems like a reasonable tradeoff between the letter and spirit of "[SecureContext]".


I'm not sure I understand the implications of that. Does that mean that for already registered manifests, they'd continue to work in non-secure contexts?


Motivation

HTML spells out a number of security concerns around appcache: https://html.spec.whatwg.org/multipage/offline.html#security-concerns-with-offline-applications-caches. In particular, the FALLBACK mechanism (and Chrome's proprietary CHROMIUM-INTERCEPT mechanism to an even greater degree) puts sites delivered over non-secure connections at real risk of persistent MitM attack. evn@ spelled this out more clearly in https://readable-email.org/list/whatwg/topic/appcache-content-type-security-considerations.


TLS allows us to ensure that the server we think we're talking to is actually the one asserting a manifest for itself, and that we're not being tricked by the Delicious But Evil Coffee Shop(tm) that we happened to be sitting in yesterday.


Interoperability and Compatibility Risk

Edge: No signals

Firefox: Shipping the restriction in Firefox 62, https://groups.google.com/d/msg/mozilla.dev.platform/qLTTpdzcDkw/WKJeq-4HAQAJ.

Safari: No signals.


I didn't see response to Firefox's announcement from Edge or Safari folks. I poked relevant people on Twitter; let's see what they say.


The response from WebKit seems encouraging.
  

Alternative implementation suggestion for web developers

Developers who wish to continue using appcache can migrate their sites to TLS. While doing so, they could also upgrade to service workers, which are the right choice for the medium-term.


Usage information from UseCounter

`window.applicationCache` is accessed from a non-secure context in ~0.15% of page views. Digging through the 445 hits in HTTP Archive[1], the vast majority of this usage appears to be feature-detection, along the lines of Modernizr's `'applicationCache' in window`, which would make the `[SecureContext]` assignment fairly safe.


Is it possible to change the use counters (or add others) so that we can distinguish between real usage and feature detection?
I don't think data on that should block this intent, but wondering about killing the API in secure contexts as well. Maybe the high usage there is also a result of feature detection shenanigans.
 



Appcache manifest selection is triggered from a non-secure context in ~0.08% of page views. I spent some time this morning digging through the 312 hits in HTTP Archive[2], with interesting results (raw data available at https://docs.google.com/spreadsheets/d/1mGUvSKyF_DOhkx9PK_NawRCVMgA416R0jJTZH1YOgU4/edit?usp=sharing):

The usage is certainly non-trivial, but I'd posit that the security risks posed by non-secure usage of the feature should push us towards lowering the bar. Moreover, the prevalence of nested appcache contexts suggests to me that the actual breakage experienced by users would be lower than the raw numbers would suggest.


What would breakage look like in your view? I'm assuming that first view will work fine, but some resources/frames may not work on consecutive views if the user is offline. Given that the top-level document doesn't have AppCache backed offline experience, does that even matter?

 


I've added these metrics to UKM, but I don't have any data yet. As 66 rolls through the channels, we can verify the data I've gathered here to see if hotspots crop up that we weren't expecting.


[1]:

```

SELECT

url, JSON_EXTRACT(payload, '$._blinkFeatureFirstUsed.Features.ApplicationCacheAPIInsecureOrigin') AS feature FROM [httparchive:har.2018_01_15_chrome_pages] HAVING feature IS NOT NULL ORDER BY url

```


[2]:

```

SELECT

url, JSON_EXTRACT(payload, '$._blinkFeatureFirstUsed.Features.ApplicationCacheManifestSelectInsecureOrigin') AS feature FROM [httparchive:har.2018_01_15_chrome_pages] HAVING feature IS NOT NULL ORDER BY url

```


OWP launch tracking bug

https://crbug.com/588931


Entry on the feature dashboard

https://www.chromestatus.com/feature/5714236168732672


Thanks! WDYT?


-mike

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKXHy%3De8i6irMQiPXasTobZND%2B1ob3Fr22bxt%2BtYdRU_m_QRmA%40mail.gmail.com.

PhistucK

unread,
Feb 3, 2018, 10:04:24 AM2/3/18
to Mike West, blink-dev
What? CHROMIUM-INTERCEPT?
I never heard of it... I wonder how many more unknown proprietary demons Chrome/Chromium has...
Can this be removed from any context (not only insecure)?

Those surprises keep popping. :(


☆PhistucK

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

Mike West

unread,
Feb 5, 2018, 3:05:56 AM2/5/18
to Yoav Weiss, blink-dev
On Sat, Feb 3, 2018 at 2:55 PM, Yoav Weiss <yo...@yoav.ws> wrote:
I'm super excited about this work! :)

Killing AppCache (entirely) will make our preloadScanner faster and will significantly help sending out requests earlier.

One interesting question: once this removal is in place, should we (and could we) make non-secure-context page-loads faster, by not forcing them to wait for the document element to be available?

On the one hand - faster is better. On the other, that could create the wrong incentives when trying to move people towards secure contexts.

I suspect that work won't actually be worth doing until we get rid of the whole thing, given that more page views are secure than not, but I'll defer that question to someone who knows things about perf. :)

On Fri, Feb 2, 2018 at 4:14 PM Mike West <mk...@chromium.org> wrote:

Primary eng (and PM) emails

mk...@chromium.org


Link to “Intent to Deprecate” thread

https://groups.google.com/a/chromium.org/d/msg/blink-dev/UKF8cK0EwMI/NLhsIrs-AQAJ


We deprecated this in early 2016, and failed to follow through with the removal. I'd like to try again, in support of Mozilla's stab at doing the same.


Summary

I'd like to make several changes:


1. Mark `window.applicationCache`, and the `ApplicationCache` and `ApplicationCacheErrorEvent` interfaces as `[SecureContext]`.

2. Prevent non-secure contexts from declaring manifests via a `manifest` attribute.

3. Prevent HTTP documents from using manifests they've registered.

4. Add an enterprise policy to remove these restrictions, with some reasonable advertised lifetime.


#3 is a little tricky, as we don't actually have the context we need in the browser-side appcache handlers to do a full secure context check (e.g. walk the frame tree). Given that the eventual goal is to get rid of appcache everywhere, this seems like a reasonable tradeoff between the letter and spirit of "[SecureContext]".


I'm not sure I understand the implications of that. Does that mean that for already registered manifests, they'd continue to work in non-secure contexts?

It means that an appcache registered in a secure top-level context might be used to serve requests for that origin, even if it's embedded in a non-secure document. I don't think that would cause any actual breakage, but it's conceptually incorrect behavior.

Motivation

HTML spells out a number of security concerns around appcache: https://html.spec.whatwg.org/multipage/offline.html#security-concerns-with-offline-applications-caches. In particular, the FALLBACK mechanism (and Chrome's proprietary CHROMIUM-INTERCEPT mechanism to an even greater degree) puts sites delivered over non-secure connections at real risk of persistent MitM attack. evn@ spelled this out more clearly in https://readable-email.org/list/whatwg/topic/appcache-content-type-security-considerations.


TLS allows us to ensure that the server we think we're talking to is actually the one asserting a manifest for itself, and that we're not being tricked by the Delicious But Evil Coffee Shop(tm) that we happened to be sitting in yesterday.


Interoperability and Compatibility Risk

Edge: No signals

Firefox: Shipping the restriction in Firefox 62, https://groups.google.com/d/msg/mozilla.dev.platform/qLTTpdzcDkw/WKJeq-4HAQAJ.

Safari: No signals.


I didn't see response to Firefox's announcement from Edge or Safari folks. I poked relevant people on Twitter; let's see what they say.


The response from WebKit seems encouraging.

It does!

Alternative implementation suggestion for web developers

Developers who wish to continue using appcache can migrate their sites to TLS. While doing so, they could also upgrade to service workers, which are the right choice for the medium-term.


Usage information from UseCounter

`window.applicationCache` is accessed from a non-secure context in ~0.15% of page views. Digging through the 445 hits in HTTP Archive[1], the vast majority of this usage appears to be feature-detection, along the lines of Modernizr's `'applicationCache' in window`, which would make the `[SecureContext]` assignment fairly safe.


Is it possible to change the use counters (or add others) so that we can distinguish between real usage and feature detection?
I don't think data on that should block this intent, but wondering about killing the API in secure contexts as well. Maybe the high usage there is also a result of feature detection shenanigans.

We could add counters to `ApplicationCache`'s methods, and look for event handlers, I suppose.

Appcache manifest selection is triggered from a non-secure context in ~0.08% of page views. I spent some time this morning digging through the 312 hits in HTTP Archive[2], with interesting results (raw data available at https://docs.google.com/spreadsheets/d/1mGUvSKyF_DOhkx9PK_NawRCVMgA416R0jJTZH1YOgU4/edit?usp=sharing):

The usage is certainly non-trivial, but I'd posit that the security risks posed by non-secure usage of the feature should push us towards lowering the bar. Moreover, the prevalence of nested appcache contexts suggests to me that the actual breakage experienced by users would be lower than the raw numbers would suggest.


What would breakage look like in your view? I'm assuming that first view will work fine, but some resources/frames may not work on consecutive views if the user is offline. Given that the top-level document doesn't have AppCache backed offline experience, does that even matter?

Correct. Since the top-level document wouldn't be available, it's hard to see how this would be a problem in practice.

I've added these metrics to UKM, but I don't have any data yet. As 66 rolls through the channels, we can verify the data I've gathered here to see if hotspots crop up that we weren't expecting.


[1]:

```

SELECT

url, JSON_EXTRACT(payload, '$._blinkFeatureFirstUsed.Features.ApplicationCacheAPIInsecureOrigin') AS feature FROM [httparchive:har.2018_01_15_chrome_pages] HAVING feature IS NOT NULL ORDER BY url

```


[2]:

```

SELECT

url, JSON_EXTRACT(payload, '$._blinkFeatureFirstUsed.Features.ApplicationCacheManifestSelectInsecureOrigin') AS feature FROM [httparchive:har.2018_01_15_chrome_pages] HAVING feature IS NOT NULL ORDER BY url

```


OWP launch tracking bug

https://crbug.com/588931


Entry on the feature dashboard

https://www.chromestatus.com/feature/5714236168732672


Thanks! WDYT?



-mike

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

Mike West

unread,
Feb 5, 2018, 3:12:53 AM2/5/18
to PhistucK, blink-dev
On Sat, Feb 3, 2018 at 4:03 PM, PhistucK <phis...@gmail.com> wrote:
What? CHROMIUM-INTERCEPT?
I never heard of it... I wonder how many more unknown proprietary demons Chrome/Chromium has...
Can this be removed from any context (not only insecure)?

Those surprises keep popping. :(

It would indeed be better if we had fewer unspecified bits in Blink. Removing CHROMIUM-INTERCEPT is certainly on my to-do list, but I expect that to just get wrapped up in a more general removal of appcache as Service Workers drive down usage over time.

-mike

Rick Byers

unread,
Feb 5, 2018, 10:00:53 AM2/5/18
to Mike West, PhistucK, blink-dev
The HTTP Archive analysis seems promising, and I'm hopeful that the enterprise opt-out will be sufficient to address the sorts of complaints we heard last time.

While usage is high, my intuition is that the page views impacted (in terms of actually be broken) will actually be very low since very few of these resources will actually be accessed offline.  Is there any easy way to quantify that?  Eg. how often do http documents use an appcache manifest which doesn't get re-validated during the page's lifetime (i.e. the user is offline for the duration of the page view)?

What sort of deprecation period are you proposing?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKXHy%3Dc-7J%2Bx%3DbYfYj%3D915zEK9P8RWqGXsV%3DqLmkSatz%2Bjdf8g%40mail.gmail.com.

Mike West

unread,
Feb 8, 2018, 4:50:18 AM2/8/18
to Rick Byers, blink-dev
On Mon, Feb 5, 2018 at 4:00 PM, Rick Byers <rby...@chromium.org> wrote:
The HTTP Archive analysis seems promising, and I'm hopeful that the enterprise opt-out will be sufficient to address the sorts of complaints we heard last time.

While usage is high, my intuition is that the page views impacted (in terms of actually be broken) will actually be very low since very few of these resources will actually be accessed offline.  Is there any easy way to quantify that?  Eg. how often do http documents use an appcache manifest which doesn't get re-validated during the page's lifetime (i.e. the user is offline for the duration of the page view)?

For the nested cases, I agree with Yoav's earlier suggestion that there's zero user-visible impact.

For top-level cases, we should be able to add some counters to quantify the totally-offline status in theory, but I recall piping a "true" view of the user's online status to the points at which we're making a decision about whether to use the manifest being a sticking-point with the earlier deprecation effort. I doubt it's gotten any easier over time.

It sounds like what you're asking for is different, though: you'd like to count the number of pages that were originally served from appcache, and that didn't successfully update the manifest? That should be possible to UseCounter. I'll poke at that.
 
What sort of deprecation period are you proposing?

We've been sending a deprecation message since 2016. So. Um. :)

I'd suggest that we land the new behavior behind a flag with a two-milestone explicit deadline. If I can get the code landed in the next week, that would be M68. If not, I'd still change the message to something explicit, but target M69.

WDYT?

Ojan Vafai

unread,
Feb 8, 2018, 12:34:14 PM2/8/18
to Mike West, Rick Byers, blink-dev
Lgtm

 


☆PhistucK

--
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKXHy%3Dc4X1gZzUEG6NMLYst%2B%3Daz_G%3DO1LD7bE_2M4W1ueF3BJw%40mail.gmail.com.

Yoav Weiss

unread,
Feb 9, 2018, 5:34:12 AM2/9/18
to Ojan Vafai, Mike West, Rick Byers, blink-dev
LGTM2

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CANMdWTuhszdd9ZTX5MAf6-yyc7G1yQFLGFp3O6gxBFzzs6pojw%40mail.gmail.com.

Rick Byers

unread,
Feb 9, 2018, 10:34:53 AM2/9/18
to Yoav Weiss, Ojan Vafai, Mike West, blink-dev
LGTM3

On Fri, Feb 9, 2018 at 5:33 AM, Yoav Weiss <yo...@yoav.ws> wrote:
LGTM2

On Thu, Feb 8, 2018 at 5:34 PM Ojan Vafai <oj...@chromium.org> wrote:
Lgtm

On Thu, Feb 8, 2018, 1:50 AM Mike West <mk...@chromium.org> wrote:
On Mon, Feb 5, 2018 at 4:00 PM, Rick Byers <rby...@chromium.org> wrote:
The HTTP Archive analysis seems promising, and I'm hopeful that the enterprise opt-out will be sufficient to address the sorts of complaints we heard last time.

While usage is high, my intuition is that the page views impacted (in terms of actually be broken) will actually be very low since very few of these resources will actually be accessed offline.  Is there any easy way to quantify that?  Eg. how often do http documents use an appcache manifest which doesn't get re-validated during the page's lifetime (i.e. the user is offline for the duration of the page view)?

For the nested cases, I agree with Yoav's earlier suggestion that there's zero user-visible impact.

For top-level cases, we should be able to add some counters to quantify the totally-offline status in theory, but I recall piping a "true" view of the user's online status to the points at which we're making a decision about whether to use the manifest being a sticking-point with the earlier deprecation effort. I doubt it's gotten any easier over time.

It sounds like what you're asking for is different, though: you'd like to count the number of pages that were originally served from appcache, and that didn't successfully update the manifest? That should be possible to UseCounter. I'll poke at that.

I don't know what's best.  Basically: if there's something we can relatively easily measure which we believe is a tighter bound on the level of risk of broken user experiences, then it would be valuable to do so.  Eg. maybe even just measuring usage of appache over http in the top document context (I agree embedded contexts are much less likely to matter to the user in offline contexts, especially given your evidence of ads and weather widgets using it).  I trust your judgement on what is the best tradeoff of being both a meaningful measure of the risk and practical to implement.

What sort of deprecation period are you proposing?

We've been sending a deprecation message since 2016. So. Um. :)

Hah hah, I forgot (this predates our "no hopeful deprecation" messages policy).  Hopefully those messages did get moved into the lower (filtered-by-default) severity bucket?  But regardless, any warning without a concrete milestone is not really a real deprecation warning.

I'd suggest that we land the new behavior behind a flag with a two-milestone explicit deadline. If I can get the code landed in the next week, that would be M68. If not, I'd still change the message to something explicit, but target M69.

Sounds perfect to me, thanks.
 

WDYT?

 


☆PhistucK

--
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKXHy%3Dc4X1gZzUEG6NMLYst%2B%3Daz_G%3DO1LD7bE_2M4W1ueF3BJw%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.
Reply all
Reply to author
Forward
0 new messages