Intent to Deprecate: AppCache

[Victor Costan‏]

Primary eng (and PM) emails

pwn...@chromium.org and natt...@chromium.org

Summary

We would like to immediately add a DevTools warning for pages that have a manifest attribute on the html element, or that access the ApplicationCache API. We hope to get the usage to drop significantly by migrating a large internal customer, and intend to follow up with an Intent to Remove afterwards.

Removing AppCache from insecure origins was previously approved and has landed in M70. This intent addresses deprecating AppCache in secure origins.

Motivation

AppCache is documented as deprecated and under removal in MDN and in the WHATWG standard, and marked as obsolete in W3C’s HTML 5.1.

The modern replacement for AppCache, service workers, is a very different type of API -- the former is declarative while the latter is imperative. As a consequence, switching a Web application from AppCache to service workers has a non-trivial technical cost. For this reason, we consider that the benefit of helping developers avoid using AppCache in new applications is worth the DevTools noise caused by the proposed deprecation warning.

Furthermore, Firefox and Safari have already deprecated AppCache (see below). We think we need to follow suit so developers don’t incorrectly infer that we support keeping the feature in the Web platform.

The upcoming removal intent hits most of the bullet points in the removal guidelines:

• Code health - Our AppCache implementation has been on life support for the past couple of years, in anticipation of the Web’s migration to service workers. Maintenance has been limited to fixing critical bugs, and we’ve accumulated a fair amount of minor bugs. Furthermore, the AppCache implementation currently has tendrils in //net and the network service, and removing it would reduce complexity throughout our network stack.

• Interoperability - Two other major browsers have already deprecated the feature.

• Known bad practice - The AppCache API is difficult to use correctly. Lighthouse considers it a bad practice.

Interoperability and Compatibility Risk

Edge: Supported, deprecation TBD

Firefox: Supported, already deprecated

Safari: Supported, already deprecated

Alternative implementation suggestion for web developers

Modern applications should use service workers, which are supported across all major browsers.

Usage information from UseCounter

The ApplicationCache API usage looks as follows:

On secure origins - ApplicationCacheAPISecureOrigin: 1.57%

On insecure origins - ApplicationCacheAPIInsecureOrigin: 1.65%

The (non-empty) manifest attribute usage is:

On secure origins - ApplicationCacheManifestSelectSecureOrigin: 0.33%

On insecure origins - ApplicationCacheManifestSelectInsecureOrigin: 0.02%

Insecure usage will go to zero in M70, per the removal mentioned above.

Entry on the feature dashboard

https://www.chromestatus.com/feature/6192449487634432

Requesting approval to remove too?

No.

We are currently working to transition large internal customers off of AppCache. We expect that the usage will drop significantly after the transition, and would like to come back with a removal proposal at that time. While we don’t have a timeline, the Chrome Storage team is committed to removing our AppCache implementation.

We acknowledge that deprecation without a strict removal date is discouraged. In this case, we believe that not posting a deprecation warning would significantly hurt Web developers who may commit new applications to AppCache. The deep architectural differences between AppCache and its replacement (service workers) lead to non-trivial switching costs.

[Yoav Weiss‏]

Thanks for working on deprecating AppCache and hopefully remove it! :)

On Fri, Aug 24, 2018 at 3:54 AM Victor Costan <pwn...@chromium.org> wrote:

Primary eng (and PM) emails

pwn...@chromium.org and natt...@chromium.org

Summary

We would like to immediately add a DevTools warning for pages that have a manifest attribute on the html element, or that access the ApplicationCache API. We hope to get the usage to drop significantly by migrating a large internal customer, and intend to follow up with an Intent to Remove afterwards.

Removing AppCache from insecure origins was previously approved and has landed in M70. This intent addresses deprecating AppCache in secure origins.

Motivation

AppCache is documented as deprecated and under removal in MDN and in the WHATWG standard, and marked as obsolete in W3C’s HTML 5.1.

The modern replacement for AppCache, service workers, is a very different type of API -- the former is declarative while the latter is imperative. As a consequence, switching a Web application from AppCache to service workers has a non-trivial technical cost. For this reason, we consider that the benefit of helping developers avoid using AppCache in new applications is worth the DevTools noise caused by the proposed deprecation warning.

Furthermore, Firefox and Safari have already deprecated AppCache (see below). We think we need to follow suit so developers don’t incorrectly infer that we support keeping the feature in the Web platform.

The upcoming removal intent hits most of the bullet points in the removal guidelines:

• Code health - Our AppCache implementation has been on life support for the past couple of years, in anticipation of the Web’s migration to service workers. Maintenance has been limited to fixing critical bugs, and we’ve accumulated a fair amount of minor bugs. Furthermore, the AppCache implementation currently has tendrils in //net and the network service, and removing it would reduce complexity throughout our network stack.

• Interoperability - Two other major browsers have already deprecated the feature.

• Known bad practice - The AppCache API is difficult to use correctly. Lighthouse considers it a bad practice.

On top of all of the above, AppCache support is also responsible for delaying preloadScanned requests, effectively slowing down the platform. Removing AppCache support will make the entire web faster.

 

Interoperability and Compatibility Risk

Edge: Supported, deprecation TBD

Firefox: Supported, already deprecated

Safari: Supported, already deprecated

Alternative implementation suggestion for web developers

Modern applications should use service workers, which are supported across all major browsers.

Usage information from UseCounter

The ApplicationCache API usage looks as follows:

On secure origins - ApplicationCacheAPISecureOrigin: 1.57%

On insecure origins - ApplicationCacheAPIInsecureOrigin: 1.65%

The (non-empty) manifest attribute usage is:

On secure origins - ApplicationCacheManifestSelectSecureOrigin: 0.33%

On insecure origins - ApplicationCacheManifestSelectInsecureOrigin: 0.02%

Can you speculate as to why the API's usage is significantly higher than the manifest?

Is the eventual intent to remove both?

 

Insecure usage will go to zero in M70, per the removal mentioned above.

Entry on the feature dashboard

https://www.chromestatus.com/feature/6192449487634432

Requesting approval to remove too?

No.

We are currently working to transition large internal customers off of AppCache. We expect that the usage will drop significantly after the transition, and would like to come back with a removal proposal at that time. While we don’t have a timeline, the Chrome Storage team is committed to removing our AppCache implementation.

I understand that there are no strict timelines for this, but can you timebox it? Do you expect that to happen within a few months? quarters? years?

We acknowledge that deprecation without a strict removal date is discouraged. In this case, we believe that not posting a deprecation warning would significantly hurt Web developers who may commit new applications to AppCache. The deep architectural differences between AppCache and its replacement (service workers) lead to non-trivial switching costs. 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAP_mGKrHWvavNK%3DiLWzHWAMoATVXRUh_6onM8ATEZQgwqrG6oA%40mail.gmail.com.

[PhistucK‏]

Yoav, I would imagine the API suffers from the window enumeration problem and so the metrics are inflated.

For an alternative for web developers - is there a library that polyfills application cache using service workers?

It would basically register a service worker, read the manifest during its installation and register fetch events and others accordingly (and might even polyfill the actual API).

If such a library does not exist, perhaps it -

1. Should exist. :)

2. Can replace (without feature detecting the existence of application cache) the implementation, which should remove such cases from the application cache usage metrics (the API, at least. for what it is worth) because feature detection can generally inflate metrics as well in some cases.

3. Prove that service workers are a strict subset of the functionality served by application cache (no functionality loss whatsoever).

4. Can even be shipped built into the browser during the short-or-long deprecation period (allowing the technical and architectural debts to be reduced to nothing).

5. Can benefit developers that are looking for declarative measures in the long run.

☆PhistucK

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACj%3DBEh1kYkQDND--XzAs_-7SDLdk5jMZvhsJyTb7eqcP5QV%3DA%40mail.gmail.com.

[Jake Archibald‏]

There's workbox (https://developers.google.com/web/tools/workbox/) which helps generate a service worker for you.

As for an Appcache polyfill, we're currently lacking good support for https://w3c.github.io/ServiceWorker/#fetch-event-resultingclientid, which we'd need to associate a page to a given cache for its lifespan. Once we've got that, building a polyfill is possible.

[Lucas Garron‏]

I would also like to see a polyfill that works as much as possible like a safe drop-in replacement before we force everyone off AppCache.

I have tried several times to move projects from AppCache to service workers by following whatever recipes looked most-recommended at the time, but I've always ended up with something that was a flakier user experience – especially when the connection isn't great.

Maybe Workbox does this well, but it looks like a time investment to set up and I can't find a simple "migrate from AppCache to Workbox with minimal pain" article that gives me confidence.

»Lucas

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/20fc41a8-fff4-43d0-8290-eb664cdf6232%40chromium.org.

[Chris Harrelson‏]

On Fri, Aug 24, 2018 at 9:45 AM 'Jake Archibald' via blink-dev <blin...@chromium.org> wrote:

There's workbox (https://developers.google.com/web/tools/workbox/) which helps generate a service worker for you.

As for an Appcache polyfill, we're currently lacking good support for https://w3c.github.io/ServiceWorker/#fetch-event-resultingclientid, which we'd need to associate a page to a given cache for its lifespan. Once we've got that, building a polyfill is possible.

Interesting. Do you think it's likely that such a polyfill would be useful enough to drive down usage to acceptable deprecation levels?

This seems to be tracked under crbug.com/778497. Matt, is this feasible to implement soon?

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/20fc41a8-fff4-43d0-8290-eb664cdf6232%40chromium.org.

[Chris Harrelson‏]

On Fri, Aug 24, 2018 at 9:57 AM Lucas Garron <lga...@chromium.org> wrote:

I would also like to see a polyfill that works as much as possible like a safe drop-in replacement before we force everyone off AppCache.

I have tried several times to move projects from AppCache to service workers by following whatever recipes looked most-recommended at the time, but I've always ended up with something that was a flakier user experience – especially when the connection isn't great.

Hi Lucas,

Could you expand on why it's more flaky? Is there a missing feature? Or is it just harder to use?

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CANCvrdRaupMehM89mkRyUvomXFbsL%2BcqNyS_yb2XtA%2B4PdN7Lg%40mail.gmail.com.

[paulc...@gmail.com‏]

Thanks for working on this!   When I saw the intent to deprecate, I did some analysis in the HTTP Archive to see where AppCache is still being used in the wild.   Out of over 1.2 million sites, I found 1,126 sites using AppCache, and 301 of them using it in an insecure context.  More details here - https://discuss.httparchive.org/t/how-many-sites-are-still-using-appcache/1443 

[Matt Falkenhagen‏]

Yes, the implementation is underway. I'm just trying to fix some things in the HTML spec about redirects and add WPT tests: https://github.com/whatwg/html/pull/3891

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Levi Allen Goff‏]

Application Cache uses Apache which must go through the proper channels before being authorized for use in this manner. The protocol use is done through HTML and opened through a web browser (Chromium - Chrome, Gecko, Safari bundle)

In this case it is through Chrome which is not accepted. Chrome releases to 70 on October 23, 2018.

Levi #2211 puts('G')

The G is possessive in place of an apostrophe S. 13 means family.

 

--

[Rick Byers‏]

Chris, Alex, Yoav and I discussed this in the API owners meeting today.

It's still the policy of the API owners that we don't do "hopeful deprecations" that lack a credible removal timeframe. We've been burned too many times by these causing more pain/distrust than benefit. There are probably things we should be doing to discourage new uses of AppCache - eg. ensuring they generate a warning in lighthouse. But deprecation warnings are not the correct tool for that (they're ineffective and seriously erode trust in our deprecation process), and better communicating best practices is not something you need to seek API owner approval for.

That said, we're optimistic that AppCache is on a path to a credible removal plan. Having per-site analysis (UKM/HttpArchive) and doing outreach to drive down both the number of top site and the total overall usage seems valuable, and a drop-in polyfill as discussed here seems likely to help in such discussions. I believe, for example, that mobile GMail (what you get when you go to gmail.com on Android or iOS) still relies on AppCache for good load performance, so seeing the performance metrics results from some sort of trial with them (eg. to just drop in the polyfill) would be good concrete data to contribute to this discussion.

If you can get the number of top 10k sites which appear to depend on AppCache for functionality/performance down to a manageable number (eg. <50), then another potential tool to help enable this transition is to do a "reverse origin trial" (as is planned for the ShadowDOM v0 deprecation). Hopefully the polyfill is a superior choice for most developers, but if there are cases where a site owner would prefer an HTTP opt-in to having to ship more JS, then an OT can give you a way to remove by default while still allowing for a 1-year grace period or something.

Rick

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAMq%3Dbeh02OKD4iQMi_w4rTCQWTfEtMY5RPXNRE8g_buNDFBtbQ%40mail.gmail.com.

[Philip Jägenstedt‏]

Hi Victor,

We (me, Alex, Chris, Daniel) discussed this briefly again today. Have you looked into adding UKM metrics for this?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFUtAY9CweUXUOnBA17Yvbzam7C_Bbeq%2B19gNp6D8-b6qpLREw%40mail.gmail.com.

[Ben Kelly‏]

On Tue, Sep 11, 2018 at 12:12 PM Philip Jägenstedt <foo...@chromium.org> wrote:

We (me, Alex, Chris, Daniel) discussed this briefly again today. Have you looked into adding UKM metrics for this?

I'm going to take this for Victor.  See https://crbug.com/883441.

Thanks.

Ben

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYdRVcGO%3D-dk1sMgpKXV4-7W7jEZjphQ12FFegFuqpW3Bw%40mail.gmail.com.

[Victor Costan‏]

Thank you very much, Ben!

I'm very grateful that you've stepped up to own this request!

   Victor

[cos...@google.com‏]

I’d like to update this intent thread with new information. In summary, AppCache has become a significant burden to Chrome’s security and stability, a polyfill has been developed, and Firefox has signaled an intent to remove AppCache. In response, we’d like to deprecate AppCache immediately (in M79), and target removal in M82. Full template below.

Contact emails

pwn...@chromium.org

Design docs/spec

Specification: https://html.spec.whatwg.org/multipage/offline.html#offline

Summary

AppCache was standardized in the Offline Web applications section of the HTML specification. The standard is formally referred to as application caches.

AppCache support was removed from insecure origins in M70. This intent addresses AppCache usage in secure origins.

Motivation

First and foremost, AppCache is a deprecated standard with serious architectural concerns. Second, Chrome's AppCache implementation is a security and stability liability.

AppCache is documented as deprecated and under removal in MDN and in the HTML standard. The AppCache API is difficult to use correctly. Lighthouse considers AppCache usage to be a bad practice. AppCache is incompatible with CORS, making it unfriendly for usage with CDNs.

From a security perspective, AppCache has recently become the 2nd most troublesome client-side storage API in the Web Platform. In 2018-2019, AppCache has required 14 merges to Beta, and 3 merges to Stable. In 2019, AppCache has also received 14 critical fixes, addressing Web Platform security issues and/or crashes. Details at https://docs.google.com/document/d/1sQoB_6bySEn3wbGtTokKqf6jp-fwaYDOd_rkYOf35-s/. Overall, AppCache was changed in over 400 Chromium CLs in 2018-2019. It has imposed a tax on all of Chrome’s significant architectural efforts: Mojofication, Onion Souping, and the Network Service.

Interoperability and Compatibility Risk

Edge: Supported, neutral to removal

Edge on Chromium intends to follow Chrome's deprecation and removal schedule.

Firefox: Deprecated, positive to removal

Intent to remove in two stages. Firefox 72 will expose the API but not populate or use caches. Firefox 74 will remove the API.

Safari: Deprecated, positive to removal

Deprecated in 2018/01, logs a deprecation warning in DevTools. No timeline indicated for removal.

Alternative implementation suggestion for web developers

New Web applications should be built around Service Workers. Existing applications that use AppCache should migrate to Service Workers.

Migrating from AppCache to Service Worker requires fundamental architectural changes. To facilitate the migration, the Chrome team has published a polyfill that implements AppCache on top Service Worker at https://github.com/GoogleChromeLabs/sw-appcache-behavior

Applications with a modern Service Worker-based architecture can use the Workbox library published at https://developers.google.com/web/tools/workbox/

Usage information from UseCounter

The ApplicationCache API usage is tracked by ApplicationCacheAPISecureOrigin: 2.70%

The (non-empty) manifest attribute usage is tracked by ApplicationCacheManifestSelectSecureOrigin: 0.28%

AppCache has very high usage for a removal. However, when the user is online, the lack of AppCache will at most result in degraded performance. Site functionality will only be compromised for offline users.

Is this feature fully tested by web-platform-tests?

No

AppCache has some WPT coverage at html/browsers/offline/

https://wpt.fyi/results/?q=html%2Fbrowsers%2Foffline%2F&run_id=318400011&run_id=333870007&run_id=312700004&run_id=331930007

Link to entry on the Chrome Platform Status

https://www.chromestatus.com/feature/6192449487634432

Requesting approval to remove too?

Yes. Asking for approval to deprecate immediately (in M79), with warning messages targeting removal in M82.

[Mike West‏]

LGTM1.

There's certainly compat risk here (though it interestingly cuts in both directions given Firefox's removal), but I agree with your assessment of the feature's security risk. Given that balance of direct risk to users, and architectural challenges in the feature's maintenance, I fully support deprecation and removal.

(Out of curiosity, what do you consider the _most_ troublesome client-side storage API? :) )

-mike

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/3878c222-dee2-420f-a572-13f5bbcecb94%40chromium.org.

[Chris Harrelson‏]

LGTM2

As with v0 Web Components APIs, it may be difficult to get the use of the API down below safe thresholds. In addition to outreach to migrate sites, I recommend you find whatever data you can that demonstrates enough sites using the API *won't* be broken by removal, and will just fall back to fewer cache hits.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKXHy%3Df18-whyP6bD4%2BG2kYoujZ%3D9g_c4fsOEzgp-aY%2B69S-1A%40mail.gmail.com.

[Yoav Weiss‏]

LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw_yVa5evqCU%2B8gd3CTLmbhruX%2BiCYsvp_9kwy%3DqKOEGdg%40mail.gmail.com.

[d.ste...@namirial.com‏]

Hi,

I'm following this issue because we actively use Application Cache in some of our products.

We need to know what happens to pages using it when it will be removed in order to chose the best strategy replacing it... I was waiting to see it in action on Chrome Canary 82, but I updated it to 82 a couple of days ago and I can see Application Cache is still working.

[Ben Kelly‏]

Canary is just the tip of trunk and will eventually become M82 stable.  The code to disable appcache will be added some time between now and when M82 is branched.  So I expect some time in the next 6 weeks.  There is some possibility it could be added later and merged to M82 while beta.

The point about testing is well taken, though.

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/153cc94b-8b25-4c80-8b52-76f1025954bc%40chromium.org.

[Adrienne Walker‏]

Following up on this discussion, I wanted to link to a blog post that describes the timeline and details of the AppCache deprecation and removal: https://web.dev/appcache-removal/

Regarding the previous question of how to test AppCache removal ahead of time, please see https://web.dev/appcache-removal/#testing-prior-to-removal.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK7rkMiuCXwwJwWkJQcnuUy_3Lo7KOw6KXGeEivdFGKiPfiGQw%40mail.gmail.com.

[vinoth ravi‏]

Hi,  we are using app cache in our application. we realized that our application is not working in offline due to chrome version upgrade since app cache has been removed. For temporary support, we saw that there is an option to generate the trail period token to be placed in index.html and app manifest file. we have done the same. but now, we are getting error as Application Cache Error event: Failed to commit new cache to storage. Could you please advise on to resolve the issue.