Intent to Ship in Early Beta or Earlier: persist Sqlite auxiliary files

[Marco Bonardo]

Summary

An open Sqlite database connection may use auxiliary files to achieve its tasks, the most common ones are the shared memory file (*-shm), used to coordinate concurrent access, and the journal file (*-journal, *-wal) that contains transactions to be merged.

These files are commonly created when the connection is opened, and deleted when the connection is closed.

Those creations and deletions have a cost, I indeed could identify async shutdown crashes where the stack pointed to this file removal operation.

I intend to test, in Early Beta and Nightly, an option to persist those auxiliary files. That means even when Firefox is closed, -shm, -journal and -wal files will remain in the profile folder, alongside the .sqlite files.

Once this change will demonstrate to be stable, I will let it ride to Release.

This change won't make ESR115, it's more likely to be in 116 or 117.

To limit the likelihood of users moving around a .sqlite file without its auxiliary files, or mismatching files (that may either cause dataloss, or corruption), I am also setting SQLITE_DEFAULT_JOURNAL_SIZE_LIMIT, that will ensure Sqlite truncates the journal file when closing the connection (plus ensures Sqlite consumers don't forget about limiting the journal file).

The journal file should thus be a persisted 0-bytes file, unless a crash or power outage happened.

If you have questions or concerns about this change, please reach out to me asap.

Notes

a. The journal size limit is only enforced on idle and empty journals, a journal can grow over the set limit, and will be truncated once emptied, thus when everything has been merged into the main database.

b. Having a journal size limit is not a way to avoid checkpointing your wal journal, indeed checkpointing is what allows the journal to be emptied, that is what in the end will enforce the limit and truncate the journal.

c. due to our default large page_size of 32768, the default checkpoint of 1000 pages is not good, that'd end up generating a 32MiB journal. As a rule of thumb you can autocheckpoint at about 1/3 of your journal file size limit (e.g. 1.5MiB size limit => 512KiB checkpoint => 16 pages).

References

https://bugzilla.mozilla.org/show_bug.cgi?id=1820478

https://sqlite.org/walformat.html

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlpersistwal

https://sqlite.org/compile.html#default_journal_size_limit

https://sqlite.org/wal.html#ckpt

https://sqlite.org/howtocorrupt.html

[Marco Bonardo]

Since we didn't notice any particular regressions or concerns with auxiliary Sqlite files persistence in Nightly and Early Beta, I plan to enable it for Release 117.

The enabling will happen in https://bugzilla.mozilla.org/show_bug.cgi?id=1834043.

Cheers,

Marco