Web-Facing Change PSA: Predictable reported storage quota

[Chromestatus]

Contact emails

ree...@chromium.org

Specification

None

Summary

Report a predictable storage quota from StorageManager's estimate API for sites that do not have unlimited storage permissions. It is possible to detect a user's browsing mode via the reported storage quota because the storage space made available is significantly smaller in incognito mode than in regular mode. This is a mitigation that prevents detection of a user's browsing mode via the storage API by reporting an artificial quota, equal to usage + 10 Gib, in all browsing modes for sites with limited storage permissions. Sites with unlimited storage permissions will be unaffected. Enforced quota will also be unaffected.

Blink component

Blink>Storage>Quota

TAG review

None

TAG review status

Not applicable

Risks

Interoperability and Compatibility

None

Gecko: No signal

WebKit: No signal

Web developers: No signals

Other signals:

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

None

Debuggability

None

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, ChromeOS, Android, and Android WebView)?

No

Is this feature fully tested by web-platform-tests?

No

Flag name on about://flags

predictable-reported-quota

Finch feature name

StaticStorageQuota

Requires code in //chrome?

False

Tracking bug

https://issues.chromium.org/issues/369865059

Estimated milestones

Shipping on desktop	133
Shipping on Android	133
Shipping on WebView	133

Anticipated spec changes

Open questions about a feature may be a source of future web compat or interop issues. Please list open issues (e.g. links to known github issues in the project for the feature specification) whose resolution may introduce web compat/interop risk (e.g., changing to naming or structure of the API in a non-backward-compatible way).

None

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/4977371751645184?gate=4955779474653184

This intent message was generated by Chrome Platform Status.

[Daniel Herr]

So the API will essentially be a useless lie to most websites. That's not ideal in my opinion. It would be better to instead throw an error or not expose the method so feature detection could work properly.

[Mike Taylor]

On 12/6/24 5:48 AM, Chromestatus wrote:

Contact emails

ree...@chromium.org

Specification

None

Summary

Report a predictable storage quota from StorageManager's estimate API for sites that do not have unlimited storage permissions. It is possible to detect a user's browsing mode via the reported storage quota because the storage space made available is significantly smaller in incognito mode than in regular mode. This is a mitigation that prevents detection of a user's browsing mode via the storage API by reporting an artificial quota, equal to usage + 10 Gib, in all browsing modes for sites with limited storage permissions. Sites with unlimited storage permissions will be unaffected. Enforced quota will also be unaffected.

A small explainer (or more details) would be useful here, it's not immediately obvious what changes you're proposing to make. Are we making this change only to incognito mode, or to regular mode as well? Do we need to update a spec somewhere, or is this already allowed (pointer to the relevant spec would be useful)?

Blink component

Blink>Storage>Quota

TAG review

None

TAG review status

Not applicable

Risks

Interoperability and Compatibility

Could you flesh out interop and compat risks here please, i.e. What do other browsers do? What do we expect to break (or not) as a result? You mention Incognito mode detection (I'm making an educated guess that "user's browsing mode" refers to) - have you looked at different use cases and how they might be impacted? Do we have test coverage?

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/675211ae.050a0220.55f02.00d8.GAE%40google.com.

[Reema A]

Thanks for the feedback. Wrote out some more details and answers to questions that have been asked below:

Problem:
It is trivially easy to detect if a user is in incognito mode through the Storage Manager’s estimate API because the amount of storage made available in incognito mode is significantly smaller than in regular mode. We have found some libraries that seem to be taking advantage of this fact and using navigator.storage.estimate() to detect if a user is in incognito mode.

Goals:
Mitigate detecting incognito mode through navigator.storage.estimate() and navigator.storageBuckets.estimate()
Reduce fingerprinting value of the estimate() API
Allow estimate() to still be functional for sites with unlimited storage permissions
Leave quota enforcement unaffected
Minimize potential site breakages

Non-goals:
Mitigating all possible methods of incognito mode detection
Mitigating detecting incognito mode through quota exhausted errors

Relevant spec:
The storage spec defines quota as follows: “The storage quota of a storage shelf is an implementation-defined conservative estimate of the total amount of bytes it can hold. This amount should be less than the total storage space on the device. It must not be a function of the available storage space on the device.”

Current state:
The value returned by estimate() is already just an estimate and in some cases the actual amount of space available to use may be different.

Proposed change:
Return an artificial quota equal to usage + 10 GiB in the Storage Manager and Storage Bucket APIs estimate() method in both incognito mode and regular mode. However, continue to return the old value returned if the site has unlimited storage permission. Additionally, enforced quota will be unaffected.

Details:
navigator.storage.estimate().quota returns usage + 10 GiB. For storage buckets, StorageBucket.estimate().quota will return either the requested quota set when opening the bucket or usage + 10 GiB if the default quota is being used.

FAQ:
Q: What about sites that rely on the quota value returned?
As mentioned in the spec, the quota is only an estimate and sites should already be handling QuotaExceededErrors.

Q: Why not just return some error indicator?
A: This is more likely to unexpectedly break sites.

Q: Why return the same value in incognito and non-incognito?
A: To ensure that they’re indistinguishable.

Q: Why 10 GiB?
This number was proposed because it is likely to be sufficiently high enough that sites are unlikely to change their behavior based on the quota estimate being too low for their use case. It is also similar to the Firefox implementation.

Q: Why not a random value?
This could result in a unique ID that could be used for fingerprinting.

Q: Why (usage + 10 GiB) and not just 10 GiB?
A: To ensure that usage is always less than the quota estimate to avoid a counterintuitive behavior that might break a site.

Q: What do other browsers do?
Firefox: In best-effort mode, Firefox returns the minimum of 10GiB or 10% of the total disk size. If the origin has been granted persistent storage, then it returns the min of 8 TiB or 50% of the total disk size. [source 1, source 2]
Safari: The docs say “Note that the quota is an upper limit of how much can be stored — there is no guarantee that a site can store that much, so error handling for QuotaExceededError is necessary. Also, to reduce fingerprinting risk introduced by exposing usage and quota, quota might change based on factors like existing usage and site visit frequency.”

Q: Have you looked at different use cases and how they might be impacted?
We have manually looked at how sites seem to be using navigator.storage.estimate() and most of the cases we’ve seen seem to be using it for incognito detection.

Q: Do we have test coverage?
Yes, we have unit tests, browser tests, and web platform tests. CLs

Q: What if sites break?
Developers can disable this change via chrome://flags/#predictable-reported-quota to validate if this is the cause of the breakage. We can also flip the flag off via Finch if needed.

Notes:
This is based on an investigation and solution proposed by t...@chromium.org.

Thanks!

Reema

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Mike Taylor]

Thanks Reema - these are helpful answers. And it seems you're most of the way to an I2S here - I think "PSA" was probably the incorrect category.

> We have manually looked at how sites seem to be using navigator.storage.estimate() and most of the cases we’ve seen seem to be using it for incognito detection.

I'd like to better understand the risk to sites who are not using this for incognito detection. Could you do a random sampling of say, 10 non-FP usages of quota estimation and see if they are in fact handling QuotaExceededErrors?

One more question: what are the actual Quota limits for storage on mobile (including low-end devices), and WebView? Are any of them lower than 10GiB?

[Mike Taylor]

We discussed this in our OWNERs meeting, and agreed this should be an I2S that requires 3LGTMs to ship. But, we can just use this thread - no need to send more mail. Some other folks have other questions, but I'll let them send them independently.

[Yoav Weiss (@Shopify)]

On Wed, Dec 18, 2024 at 5:32 PM Mike Taylor <mike...@chromium.org> wrote:

We discussed this in our OWNERs meeting, and agreed this should be an I2S that requires 3LGTMs to ship. But, we can just use this thread - no need to send more mail. Some other folks have other questions, but I'll let them send them independently.

On 12/13/24 12:57 PM, Mike Taylor wrote:

Thanks Reema - these are helpful answers. And it seems you're most of the way to an I2S here - I think "PSA" was probably the incorrect category.

> We have manually looked at how sites seem to be using navigator.storage.estimate() and most of the cases we’ve seen seem to be using it for incognito detection.

I'd like to better understand the risk to sites who are not using this for incognito detection. Could you do a random sampling of say, 10 non-FP usages of quota estimation and see if they are in fact handling QuotaExceededErrors?

One more question: what are the actual Quota limits for storage on mobile (including low-end devices), and WebView? Are any of them lower than 10GiB?

On 12/12/24 1:41 PM, Reema A wrote:

Thanks for the feedback. Wrote out some more details and answers to questions that have been asked below:

Problem:
It is trivially easy to detect if a user is in incognito mode through the Storage Manager’s estimate API because the amount of storage made available in incognito mode is significantly smaller than in regular mode. We have found some libraries that seem to be taking advantage of this fact and using navigator.storage.estimate() to detect if a user is in incognito mode.

Probably a silly question, but why is the storage made available in incognito inherently smaller than regular mode?

Couldn't we increase the incognito quotas, while still keeping them ephemeral?  

Goals:
Mitigate detecting incognito mode through navigator.storage.estimate() and navigator.storageBuckets.estimate()
Reduce fingerprinting value of the estimate() API
Allow estimate() to still be functional for sites with unlimited storage permissions
Leave quota enforcement unaffected
Minimize potential site breakages

Non-goals:
Mitigating all possible methods of incognito mode detection
Mitigating detecting incognito mode through quota exhausted errors

Relevant spec:
The storage spec defines quota as follows: “The storage quota of a storage shelf is an implementation-defined conservative estimate of the total amount of bytes it can hold. This amount should be less than the total storage space on the device. It must not be a function of the available storage space on the device.”

Current state:
The value returned by estimate() is already just an estimate and in some cases the actual amount of space available to use may be different.

Proposed change:
Return an artificial quota equal to usage + 10 GiB in the Storage Manager and Storage Bucket APIs estimate() method in both incognito mode and regular mode. However, continue to return the old value returned if the site has unlimited storage permission. Additionally, enforced quota will be unaffected.

Would that cause issues for sites where `quota`-`usage` < 10GB ?
Would developers run a risk of thinking they are safe to save more data when in fact they are out of quota? (I guess I'm not familiar with how developers use `estimate()` today and how confident they are that the estimate is accurate) 

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/675211ae.050a0220.55f02.00d8.GAE%40google.com.

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/866e1227-d7b2-4a7c-bb9e-026cdfc376f7%40chromium.org.

[Reema A]

> I'd like to better understand the risk to sites who are not using this for incognito detection. Could you do a random sampling of say, 10 non-FP usages of quota estimation and see if they are in fact handling QuotaExceededErrors?

Still working on sampling some sites to provide this analysis, will get back to you ASAP. So far the only one I’ve found that doesn’t have minified JS and have been trivially able to Ctrl+F for relevant strings does seem to be handling the error.

> One more question: what are the actual Quota limits for storage on mobile (including low-end devices), and WebView? Are any of them lower than 10GiB?

There is some data on this available internally here. I’ll see if I can follow up with more specifics.

> Probably a silly question, but why is the storage made available in incognito inherently smaller than regular mode? Couldn't we increase the incognito quotas, while still keeping them ephemeral?  

Incognito uses in-memory storage only to avoid data leaks to persistent storage. In theory this could be changed and it would fix the underlying problem but doing so would be a much, much larger effort (for example, here’s a prior proposal). I don’t think there’s a way to increase incognito quota to be significantly closer to non-incognito quota while still using in-memory storage.

> Would that cause issues for sites where `quota`-`usage` < 10GB ? Would developers run a risk of thinking they are safe to save more data when in fact they are out of quota? (I guess I'm not familiar with how developers use `estimate()` today and how confident they are that the estimate is accurate) 

Yes, it’s possible, but as mentioned in my reply above sites should already be handling QuotaExceededErrors since the estimate can already be quite different than the actual quota (more data about this to come as per Mike’s request).

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/675211ae.050a0220.55f02.00d8.GAE%40google.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Reema A]

Sorry for the delayed response!

> One more question: what are the actual Quota limits for storage on mobile (including low-end devices), and WebView? Are any of them lower than 10GiB?

Quota is 60% of total disk space per origin. For UMA-enabled Chrome installations on Android, ~3% of weekly active clients have less than 16.67GB of total storage and thus less than 10GB of quota.
For WebView, we can't get per-device quota numbers; we can only get data about the number of UMA records, which means some devices will be counted multiple times. With that caveat, <4% of UMA records come from devices with less than 10GB of quota. The fraction of low quota devices may be similar to the fraction of records, but this cannot be verified.

> I'd like to better understand the risk to sites who are not using this for incognito detection. Could you do a random sampling of say, 10 non-FP usages of quota estimation and see if they are in fact handling QuotaExceededErrors?

I looked at a few examples.

Methodology:
- I searched in the Sources tab in Developer Tools for scripts using estimate() and checked if the same script had references to QuotaExceededErrors.
- I skipped scripts that seemed to just be logging the quota estimate to the console.
- I skipped scripts that seemed to be fingerprinting, which was the majority of examples.

Major caveat: It’s very hard to tell what these sites are doing due to code minification, obfuscation, the lack of context, etc.

I found 9 examples I looked at that were not obviously fingerprinting, although I couldn’t always tell what they were using the API for. Of these, I saw references to QuotaExceededErrors in 5 instances.

[Yoav Weiss (@Shopify)]

On Tue, Jan 21, 2025 at 8:08 PM Reema A <ree...@chromium.org> wrote:

Sorry for the delayed response!

> One more question: what are the actual Quota limits for storage on mobile (including low-end devices), and WebView? Are any of them lower than 10GiB?

Quota is 60% of total disk space per origin. For UMA-enabled Chrome installations on Android, ~3% of weekly active clients have less than 16.67GB of total storage and thus less than 10GB of quota.
For WebView, we can't get per-device quota numbers; we can only get data about the number of UMA records, which means some devices will be counted multiple times. With that caveat, <4% of UMA records come from devices with less than 10GB of quota. The fraction of low quota devices may be similar to the fraction of records, but this cannot be verified.

> I'd like to better understand the risk to sites who are not using this for incognito detection. Could you do a random sampling of say, 10 non-FP usages of quota estimation and see if they are in fact handling QuotaExceededErrors?

I looked at a few examples.

Methodology:
- I searched in the Sources tab in Developer Tools for scripts using estimate() and checked if the same script had references to QuotaExceededErrors.
- I skipped scripts that seemed to just be logging the quota estimate to the console.
- I skipped scripts that seemed to be fingerprinting, which was the majority of examples.

Major caveat: It’s very hard to tell what these sites are doing due to code minification, obfuscation, the lack of context, etc.

I found 9 examples I looked at that were not obviously fingerprinting, although I couldn’t always tell what they were using the API for. Of these, I saw references to QuotaExceededErrors in 5 instances.

Is it possible to test these instances out with the flag for this feature, to see if they are likely to break or not?

 

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/675211ae.050a0220.55f02.00d8.GAE%40google.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Reema A]

Sure, I enabled the flag in Chrome Beta and navigated around each of the examples. I didn't notice any issues. However, since as I mentioned I'm not sure how they're using the API, I can't ensure I'm using up quota.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/675211ae.050a0220.55f02.00d8.GAE%40google.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Reema A]

Any more info I can provide?

[Mike Taylor]

Did you happen to test the sites w/ the feature on and off, per Yoav's suggestion?

[Reema A]

Yes, I tested with the feature on and didn't see any issues as mentioned. I also tested with the feature off and didn't see any differences in behavior. But again, as I said, I'm not able to ensure I'm using up all of the quota.

[Reema A]

Any updates on this?

[Mike Taylor]

Hey Reema,

Very sorry that you've been on hold here.

Thank you for doing some manual investigation, but I'm still a bit worried about the compat implications. 4 out of 9 sites not handling QuotaExceededError could be bad luck, or could be a sign of trouble down the road.

I'm also worried about knee-capping the API, if it will be possible to return estimates that over-promise what's available (the case where a device or WebView instance has < 10GiB available, but the API tells the application that it has more). I don't have approval to share the disk size metrics for Clank and WebView, but it's high enough that we should re-think the approach, or try to run an experiment.

Maybe we can meet to discuss next steps, and come up with a plan together?

later,
Mike

[Reema A]

Thanks for the feedback! 

To address the concerns about devices with lower amounts of disk, an updated proposal is to report a lower quota value for such clients.

Options:

1. [Recommended] Base the value returned on disk space in both non-incognito and incognito. The formula would be `usage + min(10 GiB, ceil(disk_space))` in both modes. The ceiling is used to bucket the reported values into 1 GiB buckets to reduce the amount of information exposed about the device. 
   • Advantages
     • All the same advantages as the original change: mitigates the fingerprinting vector, makes detecting incognito mode less straightforward
       
   • Disadvantages
     • Reveals which disk space bucket incognito users fall into, even though disk space isn’t actually relevant to how much quota they have.
     • This avoids returning a quota value >10GiB for clients with less than 10GiB of total disk space. However, the reported quota value will be higher than the actual quota since per-site quota is a percentage of disk space.
2. Base the value returned on disk space in both non-incognito and incognito by using the non-incognito quota value for both. The formula would be `usage + min(10 GiB, ceil(disk_space * kTemporaryPoolSizeRatio * kPerStorageKeyTemporaryRatio))` in both modes. Again, the ceiling is used to reduce the amount of information exposed.
   
   • Advantages
     • Same as option #1
   • Disadvantages
     • Reveals which disk space bucket incognito users fall into, even though disk space isn’t actually relevant to how much quota they have.
       
     • More complicated code/logic to follow than option #1.

Alternatives considered:

1. Base the value returned on quota. In practice this will mean it’s based on disk for non-incognito and on RAM for incognito.
   
   • Rejected because incognito will still be easily differentiated from non-incognito. Reported quota will continue to be much lower for incognito mode than non-incognito mode.
2. Return a static number less than 10GiB for all clients.
   • Rejected because unless the number is very low, the percentage of users with disk size less than this will still be higher than we'd like.
3. Always base the quota estimate returned off of RAM (essentially pretend the user is always in incognito mode).
   • Reported quota value will be much smaller than actual quota for non-incognito mode.

Please let me know if the new proposal looks good to you, thanks!

-Reema

[Reema Al-Marzoog]

Any concerns about the new proposal? If not, can I get LGTMs? Thank you!

Any updates on this?

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/675211ae.050a0220.55f02.00d8.GAE%40google.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Chris Harrelson]

LGTM1 % question below.

If the API is returning an approximate value, is there an increased risk of sites trying to allocate up to the amount the browser is claiming they have, and then not being able to successfully use it? Or is that already a risk and this intent does not increase it?

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/87e83d3f-af2e-48f1-a92b-5f56070b2d83n%40chromium.org.

[Reema A]

That is already a possibility before this intent and developers are expected to treat the value as an estimate and handle QuotaExceededErrors. With this intent, there is an increased risk in incognito mode since the reported quota value is likely higher than the actual quota value since actual quota in incognito is based on RAM. For non incognito, it could also happen but for most clients in non-incognito mode the actual quota will be greater than the initial reported quota value so I think the risk is lower.

Any updates on this?

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/675211ae.050a0220.55f02.00d8.GAE%40google.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/866e1227-d7b2-4a7c-bb9e-026cdfc376f7%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Yoav Weiss (@Shopify)]

LGTM2

Any updates on this?

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/675211ae.050a0220.55f02.00d8.GAE%40google.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/866e1227-d7b2-4a7c-bb9e-026cdfc376f7%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Mike Taylor]

I think there's a non-zero risk here, but it's fairly small and Reema's newest proposal minimizes the risk in an acceptable way - and, importantly, we can killswitch it if we get it wrong.

LGTM3.