Intent to Ship: Storage Buckets API

[Evan Stade]

Contact emails

est...@chromium.org, ay...@chromium.org, ajayra...@chromium.org

Explainer

https://github.com/WICG/storage-buckets/blob/main/explainer.md

Specification

https://wicg.github.io/storage-buckets

Summary

Storage Buckets gives sites the ability to organize on-device data into separate "buckets", allowing user agents to evict the grouped data independently of that which is in other buckets, and enabling sites to ergonomically manage semantically related data. Each storage bucket can contain data associated with established storage APIs such as IndexedDB and CacheStorage.

Blink component

Blink>Storage>Buckets

TAG review

https://github.com/w3ctag/design-reviews/issues/562

TAG review status

Issues addressed

Chromium Trial Name

StorageBuckets

Link to origin trial feedback summary

https://docs.google.com/document/d/1C8LiQfjYj9wIt94Q4PeCVwiA57-uf37jSLLoBHYhUCM

Origin Trial documentation link

https://developer.chrome.com/blog/storage-buckets/

Risks

Interoperability and Compatibility

If no other browsers end up implementing this API, websites will only be able to use the default bucket that is supported today.

Gecko: Positive (https://mozilla.github.io/standards-positions/#storage-buckets)

WebKit: No signal (https://github.com/WebKit/standards-positions/issues/181)

Web developers: Positive

Other signals:

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

Doesn't change existing APIs.

Debuggability

Devtools

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

No

Is this feature fully tested by web-platform-tests?

No

Flag name on chrome://flags

storage-buckets

Finch feature name

StorageBuckets

Requires code in //chrome?

False

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1099413

Launch bug

https://launch.corp.google.com/launch/4227670

Measurement

https://chromestatus.com/metrics/feature/timeline/popularity/4378

Availability expectation

Feature is expected to be available on Firefox, but timeline unknown. Implementation in Safari would likely first require widespread use.

Adoption expectation

Expected to be applied **in new projects** soon after launch. Full utilization of the API would require data migration, which is not easy and may slow adoption in existing projects. Abstractions may be among the first adopters as the API is, itself, an abstraction layer that simplifies functionality such as namespacing and managing chunks of data.

Adoption plan

We intend to use the Buckets API to expose additional storage-related functionality to the web, for example, session-scoped data, performance controls, etc. This new functionality will help drive adoption.

Non-OSS dependencies

Does the feature depend on any code or APIs outside the Chromium open source repository and its open-source dependencies to function?

No

Estimated milestones

Shipping on desktop	121
OriginTrial desktop last	118
OriginTrial desktop first	115
DevTrial on desktop	110

Shipping on Android	121
OriginTrial Android last	118
OriginTrial Android first	115
DevTrial on Android	110

Shipping on WebView	121
OriginTrial webView last	118
OriginTrial webView first	115

Anticipated spec changes

Open questions about a feature may be a source of future web compat or interop issues. Please list open issues (e.g. links to known github issues in the project for the feature specification) whose resolution may introduce web compat/interop risk (e.g., changing to naming or structure of the API in a non-backward-compatible way).

Open issues/questions are add-only enhancements.

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5739224579964928

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/u/2/a/chromium.org/g/blink-dev/c/LZsMi8heTu0/m/bh0my7vpBwAJ Intent to Experiment: https://groups.google.com/a/chromium.org/g/blink-dev/c/LGfcc48qBHY/m/cXjQrqRIAAAJ?utm_medium=email&utm_source=footer

This intent message was generated by Chrome Platform Status.

[Yoav Weiss]

On Tuesday, November 14, 2023 at 9:38:55 PM UTC+1 Evan Stade wrote:

Contact emails

est...@chromium.org, ay...@chromium.org, ajayra...@chromium.org

Explainer

https://github.com/WICG/storage-buckets/blob/main/explainer.md

Specification

https://wicg.github.io/storage-buckets

Summary

Storage Buckets gives sites the ability to organize on-device data into separate "buckets", allowing user agents to evict the grouped data independently of that which is in other buckets, and enabling sites to ergonomically manage semantically related data. Each storage bucket can contain data associated with established storage APIs such as IndexedDB and CacheStorage.

Blink component

Blink>Storage>Buckets

TAG review

https://github.com/w3ctag/design-reviews/issues/562

TAG review status

Issues addressed

Chromium Trial Name

StorageBuckets

Link to origin trial feedback summary

https://docs.google.com/document/d/1C8LiQfjYj9wIt94Q4PeCVwiA57-uf37jSLLoBHYhUCM

Can you expand on the OT results? Are the "slow to adoption" comments relevant only to the OT itself or to the feature at large?

[Evan Stade]

Hi Yoav,

The reasoning about adoption also applies to the feature after launch, although less so than the OT. The farther the feature progresses (from OT, to launch, to support in other browsers), the less risk a site is taking in adopting. In the meantime, since a good polyfill is not really possible, the intermediate phase is painful/costly.

For existing apps, taking full advantage of the API requires schema redesign/migration, which we anticipate to be pretty involved for the kind of sophisticated apps this API appeals to. Less sophisticated apps that don't need to manage lots of data will not use the API at all as they don't worry about minimizing the impact of storage eviction (which is, for now, its primary appeal). New apps that are written at a time when Buckets are broadly available will not face these hurdles.

We hope that both sides of the risk/reward equation improve: adoption risks come down as other vendors implement, and we're able to keep adding useful functionality that justifies the investment to migrate.

-- Evan Stade

[Yoav Weiss]

Thanks!

As you mention that this feature won't be polyfillable, what would we advise web developers of new apps up until the point where the feature is ubiquitous? Would they have different quota systems on different browsers?

[Evan Stade]

It depends on the use case. Sometimes developers may be OK with missing functionality. For example, if some data such as downloaded media should be persisted whereas other data such as logs should be evictable, then not having buckets available probably just means that the site does not play as nicely during storage pressure related eviction rounds, which is a reasonable fallback. In other use cases put forth in the explainer, such as when a bucket holds all the data for a single user and is deleted on logout, a site might be able to reasonably degrade by only offering "sign out all accounts" for browsers that don't yet have buckets.

In cases where buckets functionality is mission critical, such as holding data that must not be accessible after a certain expiration date, a site may would need to do more work to support situations where Buckets aren't available. In these cases we would advise at least designing a schema that makes the eventual transition to buckets easier, e.g. by keeping all logs data that might eventually move to a logs bucket co-located in a single Bucket File System (aka OPFS) directory.

[Philip Jägenstedt]

Hi Evan,

A few questions inline.

On Tue, Nov 14, 2023 at 9:38 PM Evan Stade <est...@chromium.org> wrote:

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

No

Which platform will this not be supported on?

 

Is this feature fully tested by web-platform-tests?

No

Can this be fully tested in WPT? If it's not tested in WPT, how is it currently tested?

Best regards,

Philip

[Evan Stade]

[reply-all this time]

Hi Philip,

thanks for pointing out those two oversights. I have fixed the checkboxes on the chromestatus entry. It is in fact tested by WPT and supported on all Blink platforms.

[Philip Jägenstedt]

Hi Evan,

Is https://wpt.fyi/results/storage/buckets?label=experimental&label=master&aligned the full set of tests? Given the size of the spec, does that cover the whole feature well?

Best regards,

Philip

[Evan Stade]

Hi Philip,

Here is a comprehensive list of web tests that verify storageBuckets behavior. We do feel the size of this list aligns well with the spec, which is fairly concise. As you can see, some are spread out in directories for the other storage APIs that storageBuckets brokers.

That said, not all behavior is verifiable from the web. For example, it's not possible to reliably trigger an automatic eviction event from the web, so we can't really know if `persisted` is actually respected. Browser tests provide that kind of coverage.

Some of the WPT are internal but they can/should be moved to the external WPT directory. Thanks for the prompt.

external/wpt/IndexedDB/storage-buckets.https.any.js:
external/wpt/fs/script-tests/FileSystemBaseHandle-buckets.js:
external/wpt/service-workers/cache-storage/cache-storage-buckets.https.any.js:
external/wpt/storage/buckets/bucket-quota-indexeddb.tentative.https.any.js:
external/wpt/storage/buckets/bucket-storage-policy.tentative.https.any.js:
external/wpt/web-locks/storage-buckets.tentative.https.any.js:
http/tests/inspector-protocol/storage/cache-storage-request-cache-names.js:
http/tests/inspector-protocol/storage/cache-storage-storage-key-request-cache-names.js:
http/tests/inspector-protocol/storage/indexed-db-storage-key-clear.js:
http/tests/inspector-protocol/storage/indexed-db-storage-key-delete-db.js:
http/tests/inspector-protocol/storage/indexed-db-storage-key-delete-object-store-entries.js:
http/tests/inspector-protocol/storage/indexed-db-storage-key-get-metadata.js:
http/tests/inspector-protocol/storage/indexed-db-storage-key-request-data.js:
http/tests/inspector-protocol/storage/indexed-db-storage-key-request-database-names.js:
http/tests/inspector-protocol/storage/indexed-db-storage-key-request-database.js:
http/tests/inspector-protocol/storage/storage-bucket-storage-key-track-untrack.js:
http/tests/inspector-protocol/storage/storage-buckets-delete-storage-bucket.js:
wpt_internal/storage/buckets/bucket_names.tentative.https.any.js:
wpt_internal/storage/buckets/buckets_basic.tentative.https.any.js:
wpt_internal/storage/buckets/buckets_storage_policy.tentative.https.any.js:
wpt_internal/storage/buckets/detached-iframe.https.html:
wpt_internal/storage/buckets/idlharness-worker.https.any.js:
wpt_internal/storage/buckets/opaque-origin.https.window.js:
wpt_internal/storage/buckets/resources/opaque-origin-sandbox.html:
wpt_internal/storage/buckets/storage_bucket_object.tentative.https.any.js:

-- Evan Stade

[Philip Jägenstedt]

Thanks Evan for listing out the tests, they were more spread out than I thought. Great to see more of the tests being upstreamed too!

I wonder if we can do even better and test eviction however? Have you looked into defining a WebDriver endpoint for triggering eviction? It could make sense in https://web-platform-tests.org/writing-tests/testdriver.html#storage and there are plenty of examples in that documentation that might be close to the hook/trigger you need.

If you think this is an unreasonable request, please push back, my interest is only that this can be implemented interoperably. Tests are our best way of ensuring that, but it's not the only way, and it's always a tradeoff of effort vs. utility.

Best regards,

Philip

[Evan Stade]

Hi Philip,

After conferring with the team, we feel that there is an appropriate distribution of coverage provided by WPT and Chromium unit and browser tests. Leveraging testdriver for one-off hooks such as this would introduce substantial test-only code flows, diminishing the confidence that the tests were correctly verifying production behavior, while also imposing a larger maintenance burden going forward. The benefit, as we understand it, would be that the WPT could be shared by other vendors, but the test itself in this case would only be a few lines. The vast majority of LOC would be plumbing and internals logic which other vendors would need to provide for themselves.

Getting a bit off in the weeds relative to this i2s discussion, but I'll continue on this topic for the sake of debate:

We can observe that new testdriver APIs are seldom added. I only see ~4 changes originating from the Chromium project in the last 5 years. Putting all other reasoning aside, this fact alone would suggest that testdriver should be touched only in exceptional cases. I would not argue that testdriver is without utility. Unlike the one-off actions we'd need for these buckets WPT, more generic testdriver actions such as setting permission state can enable a wide range of tests. In those cases the calculus of cost vs reward is much more favorable. So I would assume that changes to testdriver should meet a high bar of broad utility, similar to how we treat core libraries in //base/.

Hope that addresses your concerns.

-- Evan Stade

[Philip Jägenstedt]

Hi Evan,

Thanks for looking into this, and for answering some of my questions off-list. To summarize, eviction is spec'd very loosely and a testdriver.js API could probably only be "evict everything" or "evict data in this bucket". Neither of those would test the interesting parts of how quota, expiry time and probably other signals are used to determine which buckets to evict. Bottom line, if we can't see a WebDriver endpoint that would poke at the right internals in all browsers, then it wouldn't be testing production code beyond what is already tested.

It looks like get or expire a bucket could be tested using bucket.setExpires(), but AFAICT this isn't tested.

So, LGTM1 to ship with added tests for bucket expiration.

Best regards,

Philip

P.S. I disagree that testdriver.js should only be used in exceptional cases. Rather often there's a spec concept which is invoked by the user or under implementation-defined conditions, and it's straightforward to poke this concept with a WebDriver endpoint. https://fedidcg.github.io/FedCM/#automation is a good recent example. API owners don't insist on this currently mainly because testdriver.js needs to be implemented for both Chrome and content_shell. With wptrunner in Chromium CI (thanks @Weizhong Xia!) I expect the need for double work will be removed, and I would like to raise the bar for test automation. (But even then there will be cases where it wouldn't be testing additional production code and isn't worthwhile.)

[Rick Byers]

LGTM2

+1 to raising the bar on test automation, but only in the cases where there aren't unreasonable burdens and there is clear significant test coverage value (i.e., not this specific case). Thanks for the analysis Evan and Philip!

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYfLUBkQCsJG4RYHSUqxLkN42NdMTRWAK817hofLU6CrPA%40mail.gmail.com.

[Mike Taylor]

LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFUtAY-7zXke53y9TQ44fBtZBUG%2BvM701WT1FXfTBfeVUjS5cg%40mail.gmail.com.

[Evan Stade]

Thanks all!

On Wed, Dec 13, 2023 at 5:10 AM Philip Jägenstedt <foo...@chromium.org> wrote:

Hi Evan,

Thanks for looking into this, and for answering some of my questions off-list. To summarize, eviction is spec'd very loosely and a testdriver.js API could probably only be "evict everything" or "evict data in this bucket". Neither of those would test the interesting parts of how quota, expiry time and probably other signals are used to determine which buckets to evict. Bottom line, if we can't see a WebDriver endpoint that would poke at the right internals in all browsers, then it wouldn't be testing production code beyond what is already tested.

It looks like get or expire a bucket could be tested using bucket.setExpires(), but AFAICT this isn't tested.

To write reliable tests for that, we'd need a way to manipulate the clock. Is there a way to do that from WPT? I can't seem to find it if so.

[Philip Jägenstedt]

On Fri, Dec 15, 2023 at 9:03 PM Evan Stade <est...@chromium.org> wrote:

Thanks all!

On Wed, Dec 13, 2023 at 5:10 AM Philip Jägenstedt <foo...@chromium.org> wrote:

Hi Evan,

Thanks for looking into this, and for answering some of my questions off-list. To summarize, eviction is spec'd very loosely and a testdriver.js API could probably only be "evict everything" or "evict data in this bucket". Neither of those would test the interesting parts of how quota, expiry time and probably other signals are used to determine which buckets to evict. Bottom line, if we can't see a WebDriver endpoint that would poke at the right internals in all browsers, then it wouldn't be testing production code beyond what is already tested.

It looks like get or expire a bucket could be tested using bucket.setExpires(), but AFAICT this isn't tested.

To write reliable tests for that, we'd need a way to manipulate the clock. Is there a way to do that from WPT? I can't seem to find it if so.

The clock can't be mocked using WPT, but I assumed from skimming the spec that it would work to just set the expiration time to a time in the past, most trivially `bucket.setExpires(0)`. If that's not the case, then LGTM to ship without any new tests, but I'd like to understand off-thread why it's not the case.