Intent to Deprecate and Remove: Persistent Quota

185 views
Skip to first unread message

Chris Fredrickson

unread,
Jul 22, 2021, 5:01:27 PMJul 22
to blink-dev

Contact emails

cfre...@chromium.org, m...@chromium.org


Explainer

https://docs.google.com/document/d/1Cm2Hv_-RMqQPqruGL2rUyiTRgqMyHMJbNyNvhLH6KWU/edit


Specification

None


Summary

We plan to deprecate and remove support of the `window.PERSISTENT` quota type in `webkitRequestFileSystem`. `webkitRequestFileSystem` will still accept a `type` parameter and use of the PERSISTENT and TEMPORARY types will create filesystems with separate roots, but the PERSISTENT type will no longer grant access to a persistent filesystem.

We will also modify `navigator.webkitPersistentStorage` to be an alias for `navigator.webkitTemporaryStorage`. 

Finally, we will also begin ignoring the `storageType` parameter (either `window.PERSISTENT` or `window.TEMPORARY`) in methods of the `webkitStorageInfo` API (which is deprecated). 


Blink component

Blink>Storage


Motivation

Support for the PERSISTENT quota type contributes some amount of complexity to the quota system, but `webkitRequestFileSystem` is the only consumer, with relatively low usage. Additionally, support for persistent quota will hinder work on storage partitioning, so it would be helpful to simplify the quota system prior to partitioning storage.


Initial public proposal

None


TAG review

None


TAG review status

Not applicable


Risks

Because we plan to just change the “lifetime” of the persistent filesystem (rather than, e.g., merging it with the temporary filesystem), risks are fairly minimal. Usage of the PERSISTENT type with `webkitRequestFileSystem` is low, about 0.4% of pageloads.

Regarding making `navigator.webkitPersistentStorage` an alias for `navigator.webkitTemporaryStorage`, we believe this presents the least risk, since this way the quota for the `window.PERSISTENT` filesystem will still be accurately reported to sites that query it.

Regarding ignoring `storageType`: since persistent quota will no longer exist, our choices are to ignore the storage type completely, or throw an exception if kPersistent is used. The former presents the least risk of breakage.


Interoperability and Compatibility

None. No other browser supports the APIs affected here.

Gecko: No signal

WebKit: No signal

Web developers: No signals


Is this feature fully tested by web-platform-tests?

No


Flag name

None


Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5176235376246784


This intent message was generated by Chrome Platform Status.


Domenic Denicola

unread,
Jul 22, 2021, 6:11:54 PMJul 22
to Chris Fredrickson, blink-dev
From: Chris Fredrickson <cfre...@chromium.org>

> Motivation
> Support for the PERSISTENT quota type contributes some amount of complexity to the quota system, but `webkitRequestFileSystem` is the only consumer, with relatively low usage. Additionally, support for persistent quota will hinder work on storage partitioning, so it would be helpful to simplify the quota system prior to partitioning storage.

Just curious, is this type of "persistent quota storage" unrelated to navigator.storage.persist() and navigator.storage.persisted()? I would have thought those APIs gave persistent quota storage...

Victor Costan

unread,
Jul 22, 2021, 8:10:20 PMJul 22
to Domenic Denicola, Chris Fredrickson, blink-dev
navigator.storage.persist() and navigator.storage.persisted() deal with durable storage. (Term chosen intentionally to avoid a conflict with the PERSISTENT type in the to-be-deprecated File System API.)

Quoting from this very well-written documentNOTE: there are two meanings for "persistent" - the old PERSISTENT namespace used in Chrome, and the "persistent" permission defined in the Storage API. To disambiguate, the latter is often referred to as "durable" in the Chrome codebase. More details below.

I hope this helps!
    Victor

Mike West

unread,
Jul 23, 2021, 2:52:06 AMJul 23
to Chris Fredrickson, blink-dev
Hey folks!

I'm quite supportive of deprecating the entire `webkitRequestFileSystem` API, and this seems like a reasonable first step. I appreciate the data you've gathered thus far through HTTP Archive, and your analysis matches my suppositions: incognito detection drove much of the usage, and since that abuse doesn't even work anymore, the API serves little purpose.

That said, 0.4% is a lot of percent. Is there a migration story for folks who were using it in good faith? You pointed to one or two sites in your analysis that seemed to be doing so: would we just wipe user's previously-persistent data, or would it be transparently switched to temporary storage?

Also, I note that third-party usage of PERSISTENT is quite low, hovering around 0.0003% (https://www.chromestatus.com/metrics/feature/timeline/popularity/3879). I wonder if there's an opportunity to simply break it in those contexts? Have you done any analysis on that subset of pages?

Also also, I'm also somewhat dismayed to note that TEMPORARY usage is at 10% of page views (https://www.chromestatus.com/metrics/feature/timeline/popularity/2996). That's high! Can I safely assume that that's overwhelmingly incognito detection as well? If so, is there a path to removal?

From a process perspective, I'm comfortable both with this not requiring signals from other vendors (as they've clearly expressed their opinion that the spec is dead), and the TAG's input is likewise unnecessary IMO.

-mike


--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c415cd1e-fbba-4cb6-9c83-62e7d608e04an%40chromium.org.

Marijn Kruisselbrink

unread,
Jul 23, 2021, 12:21:59 PMJul 23
to Mike West, Chris Fredrickson, blink-dev
On Thu, Jul 22, 2021 at 11:52 PM Mike West <mk...@chromium.org> wrote:
Hey folks!

I'm quite supportive of deprecating the entire `webkitRequestFileSystem` API, and this seems like a reasonable first step. I appreciate the data you've gathered thus far through HTTP Archive, and your analysis matches my suppositions: incognito detection drove much of the usage, and since that abuse doesn't even work anymore, the API serves little purpose.

That said, 0.4% is a lot of percent. Is there a migration story for folks who were using it in good faith? You pointed to one or two sites in your analysis that seemed to be doing so: would we just wipe user's previously-persistent data, or would it be transparently switched to temporary storage?

It perhaps wasn't entirely clear from what we wrote (and reading the explainer/design doc again, I agree it is sort of unclear), but what we're proposing won't cause any API call that previously worked to no longer do anything. We will still (for now) support `webkitRequestFileSystem`, both with PERSISTENT and TEMPORARY options, and both those will still give you separate file systems. I.e. sites using either of both of these will keep working. What will change is the quota treatment of the PERSISTENT file system. I.e. we will start counting usage of the persistent file system against the same quota as all other storage APIs. Concretely for websites that means that `navigator.webkitPersistentStorage` will alias `navigator.webkitTemporaryStorage`, and similarly the methods on `window.webkitStorageInfo` will start ignoring the "type" parameter (as long as it is one of TEMPORARY or PERSISTENT).

With this, the expectation is that the number of websites that would actually be broken by this should be much much less than 0.4%. It's hard to put an exact number on it, since we're not currently proposing to make any API not work anymore, rather we're slightly changing the behavior of some of the legacy methods. A hypothetical website that somehow relies on temporary and persistent quota being separate could be broken by this, although I'm not sure how such reliance would have to work to actually cause issues.

We did consider more "aggressive" breakage of PERSISTENT, for example by making it an alias for TEMPORARY when passed to `webkitRequestFileSystem`. But as you point out, that would then require some kind of migration story, and would increase the risk of breakage of otherwise well-intentioned websites, without having significant extra benefits. I.e. merely changing how PERSISTENT is treated for quota purposes gets us all the same benefits for reducing code complexity and eliminating confusing UI.

Also, I note that third-party usage of PERSISTENT is quite low, hovering around 0.0003% (https://www.chromestatus.com/metrics/feature/timeline/popularity/3879). I wonder if there's an opportunity to simply break it in those contexts? Have you done any analysis on that subset of pages?

Good question, I don't think we've looked into that yet. While good for the web (in that we'd be getting rid of some portion of a chrome-only API) doing so wouldn't help us much with implementation complexity etc, so the benefits are fairly minimal.


Also also, I'm also somewhat dismayed to note that TEMPORARY usage is at 10% of page views (https://www.chromestatus.com/metrics/feature/timeline/popularity/2996). That's high! Can I safely assume that that's overwhelmingly incognito detection as well? If so, is there a path to removal?

A large part of that probably is incognito detection, yes. However there are a number of big websites/partners that are still relying on webkitRequestFileSystem as well. Long term we definitely would like to deprecate  and remove the entire `webkitRequestFileSystem` API (and the various legacy quota APIs also involved in this intent), but realistically this is not something we expect to be able to make much headway on in the near future. On the deprecating storage APIs front we're first focussing on WebSQL, as we feel removing that has a much bigger impact on security and maintainability of the web. Getting rid of `webkitRequestFileSystem` will be nice to get to a more cross-browser interoperable web, but the vast majority of the implementation is shared with other features, so the additional maintenance burden cause by keeping it around a bit longer is fairly minimal.
 
 

Yoav Weiss

unread,
Jul 26, 2021, 5:27:02 AMJul 26
to Marijn Kruisselbrink, Mike West, Chris Fredrickson, blink-dev
Since the 0.4% usage numbers are suspected to be a very loose upper-bound, I wonder what's the best strategy here.
Is there a way to use-count cases where storage quota would run out as temporary but not as persistent? 
If not, are you planning to roll this out with a kill-switch while monitoring filed issues for breakage?

Marijn Kruisselbrink

unread,
Jul 26, 2021, 4:25:46 PMJul 26
to Yoav Weiss, Mike West, Chris Fredrickson, blink-dev
On Mon, Jul 26, 2021 at 2:26 AM Yoav Weiss <yoav...@chromium.org> wrote:
Since the 0.4% usage numbers are suspected to be a very loose upper-bound, I wonder what's the best strategy here.
Is there a way to use-count cases where storage quota would run out as temporary but not as persistent? 
Good question. I think today we actually generally allow websites to store much more data in the temporary quota bucket than in the persistent quota bucket. The latter is capped at 10GB, while the former in our histograms is more than 10GB 97% of the time on desktop, and even on mobile is larger than that 80% of the time (and actual usage doesn't tend to get anywhere remotely near these amounts in the vast majority of cases). So even if websites that are using the persistent file system are on average using a lot more data than other websites, in the majority of cases that should keep working just the same way.
 
If not, are you planning to roll this out with a kill-switch while monitoring filed issues for breakage?
The described approach was intentionally picked to minimize the risk of breakage, as such I'm fairly confident that just rolling out the change directly and relying on dev & beta coverage for issues should be safe enough. Having to support persistent quota for longer will make some other ongoing projects/refactoring in the area more challenging, so the sooner we could get rid of the code the better. But if you feel the risk might be too high we certainly could roll this out with a kill-switch, we'd just prefer not to.  

Marijn Kruisselbrink

unread,
Jul 28, 2021, 12:41:19 PMJul 28
to Yoav Weiss, Mike West, Chris Fredrickson, blink-dev
To elaborate a bit more on the potential for breakage, I believe the only way a website could be truly broken from this change is if it somehow relies on temporary and persistent quota being separate buckets. Not sure what kind of logic they would have to employ to actually be broken by that no longer being the case though. As such my expectation is that it is extremely unlikely that any site will be broken by this. Certainly none of the sites in httparchive I looked at.

Yoav Weiss

unread,
Jul 28, 2021, 12:48:10 PMJul 28
to Marijn Kruisselbrink, Mike West, Chris Fredrickson, blink-dev
Thanks for clarifying! This seems like a low risk indeed.

LGTM1 to deprecate and remove (while keeping track of related issue, just in case)

Chris Harrelson

unread,
Jul 29, 2021, 7:13:57 PMJul 29
to Yoav Weiss, Marijn Kruisselbrink, Mike West, Chris Fredrickson, blink-dev

Alex Russell

unread,
Aug 5, 2021, 5:41:41 PMAug 5
to blink-dev, Chris Harrelson, Marijn Kruisselbrink, Mike West, Chris Fredrickson, blink-dev, Yoav Weiss
Do we have stats on potential storage pressure evictions that would be changed as a result of this change (as that appears to be the only place behavior differs)? And is there any UI that we display (e.g. in cache/storage clearing) that will be affected?

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

Marijn Kruisselbrink

unread,
Aug 5, 2021, 6:18:30 PMAug 5
to Alex Russell, blink-dev, Chris Harrelson, Mike West, Chris Fredrickson, Yoav Weiss
On Thu, Aug 5, 2021 at 2:41 PM Alex Russell <sligh...@chromium.org> wrote:
Do we have stats on potential storage pressure evictions that would be changed as a result of this change (as that appears to be the only place behavior differs)? And is there any UI that we display (e.g. in cache/storage clearing) that will be affected?
Good questions. No, I don't think we have stats that would tell us potential changes to storage pressure evictions as a result of this change. Data stored in the persistent file system would indeed go from never being evicted today to being evicted with all other data an origin stores. Having said that, in more than 99.9% of the cases where we evict data, that data has been last accessed years ago.

As far as affected UI, the current UI isn't very good in displaying this legacy persistent storage. I don't expect any of the storage management UI to change, if anything getting rid of persistent quota will mean less chance for bugs where we might inadvertently not count or include persistent storage when displaying information about a site. The main thing that will change UI wise is that there will no longer be a "foo.com wants to permanently store data on your device" permission prompt that accompanied usage of persistent quota.


To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

Chris Fredrickson

unread,
Aug 11, 2021, 5:25:20 PMAug 11
to blink-dev, Marijn Kruisselbrink, blink-dev, Chris Harrelson, Mike West, Chris Fredrickson, Yoav Weiss, Alex Russell
Relating to the safety of this change, we've become aware of a large website that currently assumes that temporary quota and persistent quota are counted separately - so we'll modify the plan to gate these changes behind a feature before we remove the functionality.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

Alex Russell

unread,
Aug 12, 2021, 3:34:32 PMAug 12
to blink-dev, Chris Fredrickson, Marijn Kruisselbrink, blink-dev, Chris Harrelson, Mike West, Yoav Weiss, Alex Russell
Thanks for letting us know. Please ping back here when you want to revive this intent.

Chris Fredrickson

unread,
Aug 17, 2021, 1:48:10 PMAug 17
to blink-dev, Alex Russell, Chris Fredrickson, Marijn Kruisselbrink, blink-dev, Chris Harrelson, Mike West, Yoav Weiss
We'd still like to consider this intent active - with the amendment that we'll implement the changes behind a flag, work with the site we know of so they can verify their fixes (by flipping that flag on local clients during their testing), then we'll enable the flag by default and let that change propagate through the channels.

Alex Russell

unread,
Aug 19, 2021, 3:31:49 PMAug 19
to blink-dev, Chris Fredrickson, Alex Russell, Marijn Kruisselbrink, blink-dev, Chris Harrelson, Mike West, Yoav Weiss
API OWNERs met and discussed; LGTM3.
Reply all
Reply to author
Forward
0 new messages