Intent to Deprecate and Remove: Persistent Quota

[Chris Fredrickson]

Contact emails

cfre...@chromium.org, m...@chromium.org

Explainer

https://docs.google.com/document/d/1Cm2Hv_-RMqQPqruGL2rUyiTRgqMyHMJbNyNvhLH6KWU/edit

Specification

None

Summary

We plan to deprecate and remove support of the `window.PERSISTENT` quota type in `webkitRequestFileSystem`. `webkitRequestFileSystem` will still accept a `type` parameter and use of the PERSISTENT and TEMPORARY types will create filesystems with separate roots, but the PERSISTENT type will no longer grant access to a persistent filesystem.

We will also modify `navigator.webkitPersistentStorage` to be an alias for `navigator.webkitTemporaryStorage`. 

Finally, we will also begin ignoring the `storageType` parameter (either `window.PERSISTENT` or `window.TEMPORARY`) in methods of the `webkitStorageInfo` API (which is deprecated). 

Blink component

Blink>Storage

Motivation

Support for the PERSISTENT quota type contributes some amount of complexity to the quota system, but `webkitRequestFileSystem` is the only consumer, with relatively low usage. Additionally, support for persistent quota will hinder work on storage partitioning, so it would be helpful to simplify the quota system prior to partitioning storage.

Initial public proposal

None

TAG review

None

TAG review status

Not applicable

Risks

Because we plan to just change the “lifetime” of the persistent filesystem (rather than, e.g., merging it with the temporary filesystem), risks are fairly minimal. Usage of the PERSISTENT type with `webkitRequestFileSystem` is low, about 0.4% of pageloads.

Regarding making `navigator.webkitPersistentStorage` an alias for `navigator.webkitTemporaryStorage`, we believe this presents the least risk, since this way the quota for the `window.PERSISTENT` filesystem will still be accurately reported to sites that query it.

Regarding ignoring `storageType`: since persistent quota will no longer exist, our choices are to ignore the storage type completely, or throw an exception if kPersistent is used. The former presents the least risk of breakage.

Interoperability and Compatibility

None. No other browser supports the APIs affected here.

Gecko: No signal

WebKit: No signal

Web developers: No signals

Is this feature fully tested by web-platform-tests?

No

Flag name

None

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5176235376246784

This intent message was generated by Chrome Platform Status.

[Domenic Denicola]

From: Chris Fredrickson <cfre...@chromium.org>

> Motivation
> Support for the PERSISTENT quota type contributes some amount of complexity to the quota system, but `webkitRequestFileSystem` is the only consumer, with relatively low usage. Additionally, support for persistent quota will hinder work on storage partitioning, so it would be helpful to simplify the quota system prior to partitioning storage.

Just curious, is this type of "persistent quota storage" unrelated to navigator.storage.persist() and navigator.storage.persisted()? I would have thought those APIs gave persistent quota storage...

[Victor Costan]

navigator.storage.persist() and navigator.storage.persisted() deal with durable storage. (Term chosen intentionally to avoid a conflict with the PERSISTENT type in the to-be-deprecated File System API.)

Quoting from this very well-written document, NOTE: there are two meanings for "persistent" - the old PERSISTENT namespace used in Chrome, and the "persistent" permission defined in the Storage API. To disambiguate, the latter is often referred to as "durable" in the Chrome codebase. More details below.

I hope this helps!

    Victor

[Mike West]

Hey folks!

I'm quite supportive of deprecating the entire `webkitRequestFileSystem` API, and this seems like a reasonable first step. I appreciate the data you've gathered thus far through HTTP Archive, and your analysis matches my suppositions: incognito detection drove much of the usage, and since that abuse doesn't even work anymore, the API serves little purpose.

That said, 0.4% is a lot of percent. Is there a migration story for folks who were using it in good faith? You pointed to one or two sites in your analysis that seemed to be doing so: would we just wipe user's previously-persistent data, or would it be transparently switched to temporary storage?

Also, I note that third-party usage of PERSISTENT is quite low, hovering around 0.0003% (https://www.chromestatus.com/metrics/feature/timeline/popularity/3879). I wonder if there's an opportunity to simply break it in those contexts? Have you done any analysis on that subset of pages?

Also also, I'm also somewhat dismayed to note that TEMPORARY usage is at 10% of page views (https://www.chromestatus.com/metrics/feature/timeline/popularity/2996). That's high! Can I safely assume that that's overwhelmingly incognito detection as well? If so, is there a path to removal?

From a process perspective, I'm comfortable both with this not requiring signals from other vendors (as they've clearly expressed their opinion that the spec is dead), and the TAG's input is likewise unnecessary IMO.

-mike

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c415cd1e-fbba-4cb6-9c83-62e7d608e04an%40chromium.org.

[Marijn Kruisselbrink]

On Thu, Jul 22, 2021 at 11:52 PM Mike West <mk...@chromium.org> wrote:

Hey folks!

I'm quite supportive of deprecating the entire `webkitRequestFileSystem` API, and this seems like a reasonable first step. I appreciate the data you've gathered thus far through HTTP Archive, and your analysis matches my suppositions: incognito detection drove much of the usage, and since that abuse doesn't even work anymore, the API serves little purpose.

That said, 0.4% is a lot of percent. Is there a migration story for folks who were using it in good faith? You pointed to one or two sites in your analysis that seemed to be doing so: would we just wipe user's previously-persistent data, or would it be transparently switched to temporary storage?

It perhaps wasn't entirely clear from what we wrote (and reading the explainer/design doc again, I agree it is sort of unclear), but what we're proposing won't cause any API call that previously worked to no longer do anything. We will still (for now) support `webkitRequestFileSystem`, both with PERSISTENT and TEMPORARY options, and both those will still give you separate file systems. I.e. sites using either of both of these will keep working. What will change is the quota treatment of the PERSISTENT file system. I.e. we will start counting usage of the persistent file system against the same quota as all other storage APIs. Concretely for websites that means that `navigator.webkitPersistentStorage` will alias `navigator.webkitTemporaryStorage`, and similarly the methods on `window.webkitStorageInfo` will start ignoring the "type" parameter (as long as it is one of TEMPORARY or PERSISTENT).

With this, the expectation is that the number of websites that would actually be broken by this should be much much less than 0.4%. It's hard to put an exact number on it, since we're not currently proposing to make any API not work anymore, rather we're slightly changing the behavior of some of the legacy methods. A hypothetical website that somehow relies on temporary and persistent quota being separate could be broken by this, although I'm not sure how such reliance would have to work to actually cause issues.

We did consider more "aggressive" breakage of PERSISTENT, for example by making it an alias for TEMPORARY when passed to `webkitRequestFileSystem`. But as you point out, that would then require some kind of migration story, and would increase the risk of breakage of otherwise well-intentioned websites, without having significant extra benefits. I.e. merely changing how PERSISTENT is treated for quota purposes gets us all the same benefits for reducing code complexity and eliminating confusing UI.

Also, I note that third-party usage of PERSISTENT is quite low, hovering around 0.0003% (https://www.chromestatus.com/metrics/feature/timeline/popularity/3879). I wonder if there's an opportunity to simply break it in those contexts? Have you done any analysis on that subset of pages?

Good question, I don't think we've looked into that yet. While good for the web (in that we'd be getting rid of some portion of a chrome-only API) doing so wouldn't help us much with implementation complexity etc, so the benefits are fairly minimal.

Also also, I'm also somewhat dismayed to note that TEMPORARY usage is at 10% of page views (https://www.chromestatus.com/metrics/feature/timeline/popularity/2996). That's high! Can I safely assume that that's overwhelmingly incognito detection as well? If so, is there a path to removal?

A large part of that probably is incognito detection, yes. However there are a number of big websites/partners that are still relying on webkitRequestFileSystem as well. Long term we definitely would like to deprecate  and remove the entire `webkitRequestFileSystem` API (and the various legacy quota APIs also involved in this intent), but realistically this is not something we expect to be able to make much headway on in the near future. On the deprecating storage APIs front we're first focussing on WebSQL, as we feel removing that has a much bigger impact on security and maintainability of the web. Getting rid of `webkitRequestFileSystem` will be nice to get to a more cross-browser interoperable web, but the vast majority of the implementation is shared with other features, so the additional maintenance burden cause by keeping it around a bit longer is fairly minimal.

 

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKXHy%3Dd_r83Lf4NDokigV%2BKCOs98b_EeJ9NcWASj1eVOD_4Mqg%40mail.gmail.com.

[Yoav Weiss]

Since the 0.4% usage numbers are suspected to be a very loose upper-bound, I wonder what's the best strategy here.

Is there a way to use-count cases where storage quota would run out as temporary but not as persistent? 

If not, are you planning to roll this out with a kill-switch while monitoring filed issues for breakage?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BOSsVa1KT2sc4eSJNvG9hc0LSHNc1YNJhgi7DwnqSHjZ7tvCQ%40mail.gmail.com.

[Marijn Kruisselbrink]

On Mon, Jul 26, 2021 at 2:26 AM Yoav Weiss <yoav...@chromium.org> wrote:

Since the 0.4% usage numbers are suspected to be a very loose upper-bound, I wonder what's the best strategy here.

Is there a way to use-count cases where storage quota would run out as temporary but not as persistent? 

Good question. I think today we actually generally allow websites to store much more data in the temporary quota bucket than in the persistent quota bucket. The latter is capped at 10GB, while the former in our histograms is more than 10GB 97% of the time on desktop, and even on mobile is larger than that 80% of the time (and actual usage doesn't tend to get anywhere remotely near these amounts in the vast majority of cases). So even if websites that are using the persistent file system are on average using a lot more data than other websites, in the majority of cases that should keep working just the same way.

 

If not, are you planning to roll this out with a kill-switch while monitoring filed issues for breakage?

The described approach was intentionally picked to minimize the risk of breakage, as such I'm fairly confident that just rolling out the change directly and relying on dev & beta coverage for issues should be safe enough. Having to support persistent quota for longer will make some other ongoing projects/refactoring in the area more challenging, so the sooner we could get rid of the code the better. But if you feel the risk might be too high we certainly could roll this out with a kill-switch, we'd just prefer not to.  

[Marijn Kruisselbrink]

To elaborate a bit more on the potential for breakage, I believe the only way a website could be truly broken from this change is if it somehow relies on temporary and persistent quota being separate buckets. Not sure what kind of logic they would have to employ to actually be broken by that no longer being the case though. As such my expectation is that it is extremely unlikely that any site will be broken by this. Certainly none of the sites in httparchive I looked at.

[Yoav Weiss]

Thanks for clarifying! This seems like a low risk indeed.

LGTM1 to deprecate and remove (while keeping track of related issue, just in case)

[Chris Harrelson]

LGTM2

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfVjqW2hFpzhTzwj05P7AmvOKN%3DPFJq3pz5DwDSoqjOucw%40mail.gmail.com.

[Alex Russell]

Do we have stats on potential storage pressure evictions that would be changed as a result of this change (as that appears to be the only place behavior differs)? And is there any UI that we display (e.g. in cache/storage clearing) that will be affected?

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c415cd1e-fbba-4cb6-9c83-62e7d608e04an%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKXHy%3Dd_r83Lf4NDokigV%2BKCOs98b_EeJ9NcWASj1eVOD_4Mqg%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BOSsVa1KT2sc4eSJNvG9hc0LSHNc1YNJhgi7DwnqSHjZ7tvCQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Marijn Kruisselbrink]

On Thu, Aug 5, 2021 at 2:41 PM Alex Russell <sligh...@chromium.org> wrote:

Do we have stats on potential storage pressure evictions that would be changed as a result of this change (as that appears to be the only place behavior differs)? And is there any UI that we display (e.g. in cache/storage clearing) that will be affected?

Good questions. No, I don't think we have stats that would tell us potential changes to storage pressure evictions as a result of this change. Data stored in the persistent file system would indeed go from never being evicted today to being evicted with all other data an origin stores. Having said that, in more than 99.9% of the cases where we evict data, that data has been last accessed years ago.

As far as affected UI, the current UI isn't very good in displaying this legacy persistent storage. I don't expect any of the storage management UI to change, if anything getting rid of persistent quota will mean less chance for bugs where we might inadvertently not count or include persistent storage when displaying information about a site. The main thing that will change UI wise is that there will no longer be a "foo.com wants to permanently store data on your device" permission prompt that accompanied usage of persistent quota.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c415cd1e-fbba-4cb6-9c83-62e7d608e04an%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKXHy%3Dd_r83Lf4NDokigV%2BKCOs98b_EeJ9NcWASj1eVOD_4Mqg%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BOSsVa1KT2sc4eSJNvG9hc0LSHNc1YNJhgi7DwnqSHjZ7tvCQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Chris Fredrickson]

Relating to the safety of this change, we've become aware of a large website that currently assumes that temporary quota and persistent quota are counted separately - so we'll modify the plan to gate these changes behind a feature before we remove the functionality.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c415cd1e-fbba-4cb6-9c83-62e7d608e04an%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKXHy%3Dd_r83Lf4NDokigV%2BKCOs98b_EeJ9NcWASj1eVOD_4Mqg%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BOSsVa1KT2sc4eSJNvG9hc0LSHNc1YNJhgi7DwnqSHjZ7tvCQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Alex Russell]

Thanks for letting us know. Please ping back here when you want to revive this intent.

[Chris Fredrickson]

We'd still like to consider this intent active - with the amendment that we'll implement the changes behind a flag, work with the site we know of so they can verify their fixes (by flipping that flag on local clients during their testing), then we'll enable the flag by default and let that change propagate through the channels.

[Alex Russell]

API OWNERs met and discussed; LGTM3.

[Colm O Flynn]

Quick question in relation to this.

Is there any debug logging one can enable if one thinks that the browser is re-claiming files deleted from the temporary file system?

i.e. Where the browser would log that is deleting file x because storage limit has been hit?

Thanks in advance

[Joe Medley]

Hi,

When are you planning to ship?

Joe Medley | Technical Writer, Chrome DevRel | jme...@google.com | 816-678-7195

If an API's not documented it doesn't exist.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/80b73287-a86b-4f15-93dd-1b2740726a3fn%40chromium.org.

[Chris Fredrickson]

Hi,

Apologies for the delayed responses.

Colm: there's no DevTools logging (e.g.) for eviction events, but you can observe storage eviction events on chrome://tracing, under the "browsing_data" category.

Joe: we're waiting for an impacted partner before we ship this deprecation.

Chris

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Ayu Ishii]

Hi everyone,

Thank you for the approvals! I apologize for the delay here.

We've confirmed that our impacted partner is now migrated off and we are ready to proceed.
We are targeting deprecation for M105.

Thanks,
Ayu

[Ayu Ishii]

Hi everyone, just updating the thread to inform that this removal milestone has been changed to M106.