chrome.storage.sync: Best practices to safeguard for quotas

[Nikita Vasilyev]

chrome.storage.sync has various quotas, making it non-trivial to use.

1. chrome.storage.sync.MAX_ITEMS is only 512. Every time I add an item, I need to make sure it fits. 

There's no built-in efficient way to get the key count. I can:

— get all the data with chrome.storage.sync.get() and count the keys before every operation. This is unnecessarily slow.

— store keyCount property, and keep it updated on every set/remove/clear. This adds more complexity.

Alternatively, I can use try/catch and rely on the error message to remove the oldest items and retry the add operation (similar example below).

2. chrome.storage.sync.QUOTA_BYTES is only 100KB. So every time I add a new value, I have to ensure enough space is available.

So far, I ended up with this:

chrome.storage.sync.set(newItem).then(function(a) {
    // Ok
}).catch(function(er) {
    if (er.message === "QUOTA_BYTES quota exceeded") {

        // Calculate the approx size of the new item, then remove the oldest items until the new item fits, then retry sync.set.

    }
});

Is that what extension developers are expected to do? To rely on error messages?

[Oliver Dunk]

Would you be able to share more about your use case?

In general, my advice would be to only use the sync storage for small amounts of data to the point where the quota is not going to be a problem. If you're regularly (or even infrequently) hitting the limit, it might be worth considering alternatives.

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/f0e173ef-6f42-4617-921e-eea40a82bcebn%40chromium.org.

[Nikita Vasilyev]

My extension archives old tabs. It stores the tab's URL/title/favicon, some tab state (scroll position, text selected, video playback position), tab groups, and window size and position.

Over time, users hit the sync storage quotas. So, instead of saving everything in it, I'm using it as an LRU cache for added/removed operations. I truncate the data into 1KiB chunks as per QUOTA_BYTES_PER_ITEM. It works fine, but it's fairly complex for what it is.

[Simeon Vincent]

This use case sounds like it's more complicated than what the API was designed for. While it is possible to use it as a general purpose data storage mechanism, sync storage was designed to synchronize a user's settings across devices. For example, whether the user prefers light or dark mode, their prefered language, whether or not notifications are enabled, etc.

The use case you described is specific enough that it may be more appropriate to use a 3rd party service or to stand up your own database to store this data. 

Simeon - @dotproto

incremental.software

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/8e0b8999-cac1-4174-9e70-4c4a317def35n%40chromium.org.

[wOxxOm]

Your data is probably highly compressible, so you can try using CompressionStream('gzip') and convert the output to 15-bit unicode to skip the code ranges 00-1F (generates long escape sequences in JSON used internally by the quota enforcer) and D800-DFFF (can't be used for a character as it's reserved for surrogate pairs).

[Robbi]

If the problem occurs frequently, you might consider saving the data to a specific file on the user's GDrive.

This json file will be read and transferred to the local storage that doesn't have these limits.

This binds the use of the extension to the user's Google account. More specifically I want to say that the extension will always have to work with a connected account otherwise the local storage misalignment between the various devices.

There are various extensions that save to Drive, but in all cases that I could see it's always for backup\restore purpose and there is therefore no automatic synchronization.

Perhaps for you this may be too extreme a solution.

[Nikita Vasilyev]

I wasn't aware of CompressionStream API — this is actually useful!

(In case somebody else needs it, the MDN docs are currently sparse, but the spec has some examples.)

[Nikita Vasilyev]

GDrive is a bit out of the scope of what I'm trying to do, considering that chrome.storage.sync already works (granted, with a fair amount of boilerplate code).

[Nikita Vasilyev]

On Thursday, June 1, 2023 at 12:50:26 PM UTC-7 wOxxOm wrote:

Your data is probably highly compressible, so you can try using CompressionStream('gzip') and convert the output to 15-bit unicode to skip the code ranges 00-1F (generates long escape sequences in JSON used internally by the quota enforcer) and D800-DFFF (can't be used for a character as it's reserved for surrogate pairs).

Could you please explain the "convert the output to 15-bit unicode" part?

So far, I haven't figured out how to encode the gzipped array buffer properly.
https://gist.github.com/NV/6222a34e5c09d7ce2d7c8e09b8e61fc4

[wOxxOm]

You would do it in JS using an algorithm for converting bit depth. Hopefully there's an existing algo you can adapt. You would take the source as Uint8Array or Uint16Array and the destination would be a string to which you append String.fromCharCode of each converted 15-bit number carrying over the remaining 1 bit to the next number.

[Robbi]

In my opinion, the road of compression is like playing "Kick-The-Can".

I am not saying that it does not worth experimenting, indeed I think that in your specific case it is a good way, but I don't think it is the definitive one.
If you do not want to think an alternative storage service, you should notify the user of the possibility of a quota limits  reachment  and therefore give him the chance to erase something at his discretion or with an automatism (for example deleting the oldest items).