Preferred storage technology for internal code

[Kai Engert]

For a storage need in Thunderbird, a new application functionality,
we're evaluating which storage technology we should use, either SQLite
or IndexedDB.

My initial thought was to use SQLite.

Can we assume that SQLite APIs (resource://gre/modules/Sqlite.jsm) will
remain available for internal application code, including Thunderbird?

Are there strong reasons why either SQLite or IndexedDB should be preferred?

I see that IndexedDB doesn't guarantee that data will remain available,
rather the browser/platform may decide at any time to delete it when
storage is slow.

IndexedDB allows origin specific data, but our data isn't bound to an
origin, it's global application data.

Also it seems that IndexedDB gets mapped to raw sqlite storage? If true,
then IndexedDB could be seen as an unnecessary layer.

However, the argument has been brought up that in general, it could be
seen as a best practice for Thunderbird to prefer the use of
standardized web technologies, as it's less likely to get
deprecated/removed from the platform.

Thanks in advance for your thoughts,
Kai

[Jan Varga]

On 1/18/22 12:52 PM, Kai Engert wrote:
> Are there strong reasons why either SQLite or IndexedDB should be
> preferred?

I assume that the storage will be used mainly from JS, in that case
IndexedDB might be a better option, because you can store JS objects in
it easily (it's called structured cloning). Even DOM blob objects can be
easily stored in IndexedDB.

> I see that IndexedDB doesn't guarantee that data will remain
> available, rather the browser/platform may decide at any time to
> delete it when storage is slow.

If you use IndexedDB from chrome (using the system principal), the
stored data won't ever be automatically deleted by the browser. I can
provide you more details about that if you are interested.

> Also it seems that IndexedDB gets mapped to raw sqlite storage? If
> true, then IndexedDB could be seen as an unnecessary layer.

Yes, IndexedDB uses SQLite internally, but that should be considered as
an implementation detail only.
I would recommend direct use of SQLite only for highly optimized storage
and queries (full control over the underlying database system).

> However, the argument has been brought up that in general, it could be
> seen as a best practice for Thunderbird to prefer the use of
> standardized web technologies, as it's less likely to get
> deprecated/removed from the platform.

IndexeDB was the primary storage for Firefox OS (IIRC, almost for all
system and app code), that is probably still true in KaiOS.

Lastly, it would be good for our IndexedDB implementation itself if we
had a consumer which is closely tight with mozilla-central.

Please let me know if you have any questions.

Jan

[Gabriele Svelto]

On 21/01/22 13:40, Jan Varga wrote:
> IndexeDB was the primary storage for Firefox OS (IIRC, almost for all
> system and app code), that is probably still true in KaiOS.

Yes, it is still true in KaiOS and recommended in their devdocs.

Gabriele

[Julian Descottes]

For what it's worth, Firefox DevTools use IndexedDB to persist Debugger breakpoints, user information, preferences and other things...

- implementation at https://searchfox.org/mozilla-central/source/devtools/shared/async-storage.js

- call sites at https://searchfox.org/mozilla-central/search?q=async-stor%28age%7Ce-helper%29&regexp=true&path=devtools

Julian

[Nick Alexander]

Hi,

On Fri, Jan 21, 2022 at 1:46 AM Kai Engert <ka...@kuix.de> wrote:

For a storage need in Thunderbird, a new application functionality,
we're evaluating which storage technology we should use, either SQLite
or IndexedDB.

My initial thought was to use SQLite.

 It's worth noting the (now ancient) document counseling to avoid SQLite at https://wiki.mozilla.org/Performance/Avoid_SQLite_In_Your_Next_Firefox_Feature.

I personally don't agree with much of what is written, but the general thrust at that time was that we had a growing number of (independent) SQLite databases that could be made into simple JSON files.  Almost all were made into JSON files, helping performance significantly.

Without context on the shape, size, and access patterns of your data, you're very unlikely to get impactful suggestions that address your use case.  I will say that `SQLite.jsm` is very unlikely to go away in a reasonable time frame.  The key-value store `rkv` may be an option for you, but it is not as sure a bet as `SQLite.jsm`.

Best,

Nick