Does unlimitedStorage guarantee FileSystem data?

[Peter Coles]

I received a report from a user of my Chrome extension where they seem to have lost data that I have stored in the HTML5 FileSystem API (like 1000s of screenshots), but data from IndexedDB was persisting.

In the manifest (v2) it has the permissions: ["activeTab", "storage", "unlimitedStorage"]

It is my understanding that both APIs should be safe with "unlimitedStorage", but I wanted to double-check to see if there are choices the browser might make, especially when a computer is running out of space.

Secondary question—would there be any reason I should consider storing blobs (like screenshots that can be up to 12MB) in IndexedDB instead of the FileSystem?

[Jackie Han]

would there be any reason I should consider storing blobs (like screenshots that can be up to 12MB) in IndexedDB instead of the FileSystem?

In addition to IndexedDB, you can also use origin private file system (OPFS) 

On Tue, May 2, 2023 at 12:32 PM wOxxOm <wox...@gmail.com> wrote:

unlimitedStorage doesn't apply to any web storage used by extension, it's applied only to chrome.storage.local. Web storage that can grow infinitely (i.e. IndexedDB, FileSystem but not localStorage) uses the global quota manager so it's subject to eviction when the free space on the disk is low and extensions aren't excluded from eviction as well. See https://crbug.com/1209236 for more info.

> storing blobs (like screenshots that can be up to 12MB) in IndexedDB instead of the FileSystem?

+ Cross-browser compatibility

+ IndexedDB is arguably faster

+ The sandboxed FileSystem will be deprecated it seems, https://crbug.com/695508#c20

You can't use Blob/File directly in an img element on your page, but there are wrappers to emulate FS via IDB e.g. https://github.com/ebidel/idb.filesystem.js/

IndexedDB had several bugs in the past that caused wipeouts or corruption of data but that's an inevitable consequence of being actively developed.

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/7a1b9ccd-0347-45a6-838e-b8b43afb50f7n%40chromium.org.

[Peter Coles]

Thank you for the response! I’m breaking this into two parts:

(1) web storage eviction and extension origins—you shared https://crbug.com/1209236

• my understanding from that ticket is
  • “web storage” (e.g., window.localStorage, IndexedDB, FileSystem, OPFS) is not durable, even with "unlimitedStorage"
  • there was originally a consensus to make requesting persistent storage in Chrome make “web storage” durable
  • work on this started in June 2021 with details continuing in https://crbug.com/1225642 and by December 2021 it was suggested the prior idea was not a viable approach and following rdevlin.cronin@ ’s mention that a reasonable alternate approach is to auto-grant persistent storage to Extensions and Web apps (and that ticket appears to now be abandoned)
  • as it stands, “web storage” is still not durable and is subject to eviction—is there any plan to change this?
• relatedly Issue 680392: navigator.storage.persist() should grant persistent storage for extensions
  
  • Comment #5 on that one states: “Extension origins with ‘unlimitedStorage’ will not be evicted under storage pressure.”
  • There’s a lot of confusion in this thread, I was one of the people seeking a clear answer, and I later echo'ed comment #5—it appears I have to update that response as that is incorrect for “web storage”
  • Should this issue be clarified and merged into 1209236?
• what can I do in the meantime?
  • Should I use StorageManager.estimate() and warn users when usage gets too close to quota? Are there any heuristics I can use for when that matters?

(2) Storing large blobs in IndexedDB

• sounds like this is fine and should be performant (and I can even use that wrapper you linked)
• if I wanted to use another file system API it appears OPFS is the standard that is taking over for the HTML5 File System (but it is not as widely support as IndexedDB)

[Jackie Han]

About part-1, I recommend filing an issue at https://github.com/w3c/webextensions/issues, let browser vendors confirm the current state of this issue, as well as future plans.

[Jackie Han]

I filed an issue at https://github.com/w3c/webextensions/issues/388

[Simeon Vincent]

• if I wanted to use another file system API it appears OPFS is the standard that is taking over for the HTML5 File System (but it is not as widely support as IndexedDB)

Is current support an issue for you? If so, what browsers and version ranges do you target?

Simeon - @dotproto

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/CAAgdh1KXpuUD0Xq0zkzktdvk_2yYYviynuwPfsfpiLQkZUaOyw%40mail.gmail.com.

[Peter Coles]

RE issue: thanks Jackie.

RE support: Simeon, my extension currently lists Chrome 49 as the "minimum_chrome_version", so I’m seeing I would need to shed older versions or instead use IndexedDB if I were to replace the HTML5 FileSystem.

Regardless, I’m still most concerned about storage pressure eviction, so I’ve commented on https://crbug.com/1225642 and https://crbug.com/680392 and it seems like there’s maybe nothing I can do in the meantime other than try and warn users who are getting close to the StorageManager.estimate() quota limit?

[Jackie Han]

Here is a new article from Chrome about this topic: Storage and cookies in extensions. The article states that:

By default, extensions are subject to the normal quota restrictions on storage, which can be checked by calling navigator.storage.estimate(). Storage can also be evicted under heavy memory pressure, although this is rare. To avoid this:

• Request the "unlimitedStorage" permission, which affects both extension and web storage APIs and exempts extensions from both quota restrictions and eviction.

• Call navigator.storage.persist() for protection against eviction.

I don't know if it's right.

[Peter Coles]

Thanks for sharing this, Jackie! My best understanding from user bug reports I’ve received and from the crbugs I linked (#1225642, #680392) is that "unlimitedStorage" can still be subject to eviction too. IDK if anyone from the Chrome team has been able to look at this again?

[Oliver Dunk]

Hi all,

Glad you spotted the new article! I wrote that document recently which is hopefully a more up to date state of the world. Unfortunately it seems like there's a lot of discrepancy here between what has been discussed in issues and what is actually implemented in Chrome - fixing that is on my list but I wanted to prioritize some up to date documentation.

When an extension has the unlimitedStorage permission, we add it to a list of unlimited extensions: https://source.chromium.org/chromium/chromium/src/+/main:chrome/browser/extensions/extension_special_storage_policy.cc;l=242;drc=977dc02c431b4979e34c7792bc3d646f649dacb4

As a result, we return true from `IsStorageUnlimited` here: https://source.chromium.org/chromium/chromium/src/+/main:chrome/browser/extensions/extension_special_storage_policy.cc;l=155;drc=1872c7ab4d625c69ac97f0944e93d3deebab5242

This is treated as equivalent to having durable storage in the eviction manager: https://source.chromium.org/chromium/chromium/src/+/main:storage/browser/quota/quota_database.cc;l=688;drc=977dc02c431b4979e34c7792bc3d646f649dacb4

So based on this, and also a discussion with the storage team, we believe extension storage with that permission should be exempt from quota/storage pressure eviction. If anyone sees consistently that it is behaving differently I'd definitely love to know.

Longer term, I think it would be great if extension storage was just durable by default. We haven't discussed that much though.

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/2d635608-2410-46ac-b24d-59c52f5d5af1n%40chromium.org.

[Jackie Han]

Thank you for clarifying this! In the past, the documentation (both Chrome and MDN) only said that it was used to exempt the chrome.storage.local's 5MB limitation, so many people were not sure whether it would also work for Web storage (IndexedDB, CacheStorage, etc).

I see you updated Declare permissions for the "unlimitedStorage" permission.

[Peter Coles]

Thanks, Oliver! Is the discussion in this CR bug consistent with that? https://bugs.chromium.org/p/chromium/issues/detail?id=1209236 If I’m misunderstanding, then that’s good news!

[Oliver Dunk]

Hi Peter,

At a glance everything in that bug seems accurate - let me know if anything stuck out.

The one thing that isn't clear in that bug is that `unlimitedStorage` does already result in durable storage.

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

[piyush gupta]

Hi Oliver,

Thanks for the write up with source code.

Had a few questions regarding navigator.storage.persist if you can help answer :

• So do we need to run navigator.storage.persist() in extensions to get persistent storage for OPFS or the unlimitedStorage permission is enough ?
• navigator.storage.persist() returns false even with unlimitedStorage permission, should it not return true in that case ?
• the doc mentioned earlier mentions that `Call navigator.storage.persist() for protection against eviction.` Is this part of the doc correct, considering this function call always return false in extension pages ?
  

[Oliver Dunk]

Hi Piyush,

I spoke to the team and confirmed that in the usual case OPFS data is stored in the "default" storage bucket. With that in mind, I would expect it to behave the same, and unlimited storage should protect it from eviction.

`navigator.storage.persist()` can actually return true for extension pages, if you meet certain heuristics which identify pages important to the user. This isn't reliable for extensions though and we don't yet have the full set of heuristics written down anywhere. Similarly, for unlimited storage, this doesn't get reflected in the return value. I'm unsure if this is intended.

Hopefully we can provide some more clarity on all of the above soon, unfortunately there are lots of interlinked parts and we want to make sure what we document is accurate, so it can be tricky. I know this is very useful information to have as a developer though.

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

[piyush gupta]

Thanks Oliver for talking to the team.

When you say `Hopefully we can provide some more clarity on all of the above soon`, do you mean team is working on updating the docs regarding persistence in extensions, if so, is there any timeline on when it'll be live ? 

[Oliver Dunk]

There aren't currently any updates planned there. I'll certainly make sure the team knows it is wanted though.

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

[piyush gupta]

So I recently encountered a scenario where extension data was completely wiped out even with 'unlimitedStorage' permission and then I started getting 'invalidstateerror' when trying to access opfs files/folders.

Extension would have been under heavy eviction pressure, considering estimated storage left was around 0 or may have crossed 0 and navigator.storage.persist() was returning false.

Is there any way currently to ensure that extension storage is persisted and won't be deleted automatically ?

[Oliver Dunk]

To clarify, do you mean data stored in web APIs or using `chrome.storage`?

If you have the `unlimitedStorage` permission, I wouldn't expect the behavior you mentioned for web APIs.

I don't think eviction applies at all to the extension APIs but I haven't looked closely at that.

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

[David Murdoch]

Hey Olivier,

I work on MetaMask (https://chromewebstore.google.com/detail/metamask/nkbihfbeogaeaoehlefnkodbefgpgknn?hl=en) and we’ve been running into corruption issues when storing large JSON blobs in a single `chrome.storage.local` key.

Our data for this key can often exceed 75 MB. When that key becomes part of a corrupted LevelDB block, the entire value is lost — sometimes with Chromium/LevelDB returning `undefined` silently. We’ve also seen unrecoverable database corruption ([Chromium issue](https://issues.chromium.org/issues/432497646)).  
We update this key very frequently (every few seconds when the app is active), so I suspect the combination of size and write frequency increases corruption likelihood.

We’re exploring three options:

1. **Stay on `chrome.storage.local`**, but split into many smaller keys, writing only changed keys (lower write volume per key, same frequency).  
2. **Switch to IndexedDB**, keep a single large key (same volume and frequency).  
3. **Switch to IndexedDB**, split into many smaller keys (as in #1).

IndexedDB is appealing because it’s been actively developed, is more robust, and includes automatic recovery mechanisms that `chrome.storage.local` lacks.  

Our hesitation is persistence guarantees -- and I'd feel much more comfortable with **documented** guarantees before migrating.
From the [docs](https://developer.chrome.com/docs/extensions/reference/api/storage):

> "`chrome.storage.local` is similar to the web platform's storage APIs (`IndexedDB`, and `Storage`), but was designed to meet the storage needs of extensions. [...] Even if the user clears the cache and browsing history, the data persists."

To me, that implies this persistence *is not* guaranteed for IndexedDB.  

The docs also say:  
> "We recommend using `storage.local` to store larger amounts of data."

Since our data seems like it may qualify as “larger,” this reads as a recommendation *over* `IndexedDB` for our use case.

**Questions:**

1. Which API has the **strongest persistence guarantees** for extensions, especially under heavy write load and eviction scenarios?  
2. Would switching from one huge key to many smaller keys meaningfully reduce corruption risk in `chrome.storage.local` (or IndexedDB, if that is the ultimate recommendation here)?

Thanks for any insight you can share.

[al]

I'm not Oliver, but I know 1Password (Unaffiliated)  uses IndexedDB storage for sensitive data at least and they use storage.local for other data.

Perhaps they might have some insight into this as well.

[piyush gupta]

Hey Oliver, don't know how I missed replying to your ques :) 

• do you mean data stored in web APIs or using `chrome.storage`?

• • data stored using OPFS apis, so i think that would be web apis

• If you have the `unlimitedStorage` permission, I wouldn't expect the behavior you mentioned for web APIs

• • yes but this does happen, with eviction pressure i think even unlimitedstorage permission is not able to protect user data, and then 'invalidstate' comes which is only fixed with browser restart

Can we please check with team again ? I guess earlier you mentioned OPFS data falls into default storage, so it should be protected

[Oliver Dunk]

Apologies for the delay in replying to this one.

[...] To me, that implies this persistence *is not* guaranteed for IndexedDB.

My understanding and the understanding of folks working on storage in Chromium is that with the unlimitedStorage permission, IndexedDB data for an extension origin should be exempt from eviction, even under storage pressure. piyush's message implies otherwise - unfortunately I can't speak to that, but it would be considered a bug if that happened.

Since our data seems like it may qualify as “larger,” this reads as a recommendation *over* `IndexedDB` for our use case.

I expect this was purely intended to mean in comparison to other storage areas like sync storage. IndexedDB is honestly preferable, since as you mention, it is actively maintained and much more heavily optimized. 

Which API has the **strongest persistence guarantees** for extensions, especially under heavy write load and eviction scenarios?

The intention is that with the unlimitedStorage permission, all storage areas have the same level of persistence (data is never evicted). 

Would switching from one huge key to many smaller keys meaningfully reduce corruption risk in `chrome.storage.local` (or IndexedDB, if that is the ultimate recommendation here)?

I don't know, unfortunately.

yes but this does happen, with eviction pressure i think even unlimitedstorage permission is not able to protect user data, and then 'invalidstate' comes which is only fixed with browser restart

Could you clarify what you mean here? Is this something you have seen based on reports from end users, or do you have steps to reproduce?

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB