IndexedDB transactions are no longer durable by default, and other changes

[ben turner (bent)]

Hi folks,

Just a heads up that some big-ish changes to IndexedDB landed today on m-c.

1. The main change that affects users is bug 1112702, which switched IndexedDB databases to be non-durable by default.
2. The the schema version for all databases has changed, so once you open a database on trunk builds and it upgrades you will not be able to open that same database in an aurora/beta/release build.

Details follow!

1. IndexedDB databases are now non-durable by default. This means that "versionchange" and "readwrite" transactions will now fire "complete" events after the transaction has committed but potentially before all data has been written to disk. If a crash or power loss occurs at just the right moment then the transaction will be lost/rolled back. It should still be impossible to ever see database corruption though. This will mean faster delivery of "complete" events, and more closely aligns with the performance vs. stability tradeoffs other browser vendors have chosen.

The IndexedDB API does not currently have a way to say "no, really, I want to make sure that this important data is saved to disk before I continue". Some examples might be things like saving contacts, reservation confirmations, one-time download tokens, etc. We will work with the spec authors to get something in v2 of the IndexedDB specification for this.

In the meantime, if the "dom.indexedDB.experimental" pref is set (defaults to |false| in Firefox and |true| in B2G, I think), you can specify the "readwriteflush" transaction mode if you want to ensure that data is written to disk before the "complete" event is delivered. E.g.:

var transaction1 = db.transaction("foo", "readwrite");
// Use transaction1...
transaction1.oncomplete = event => {
// Data may not yet be written to disk! A crash or power loss
// here could roll this transaction back.
};

// With "dom.indexedDB.experimental" set to |true|:
var transaction2 = db.transaction("foo", "readwriteflush");
// Use transaction2...
transaction2.oncomplete = event => {
// Data is guaranteed to be written to disk. A crash or power
// loss here will retain this transaction.
};

Waiting for data to be flushed to disk can take a long time, so unless the data you're saving is critical/non-recoverable then you should probably be fine continuing to use the "readwrite" transaction mode.

2. The database schema and journal mode have changed, and it should dramatically decrease the size on disk of databases for desktop builds and should increase performance of many operations due to decreased I/O. To do this we have to rebuild the schema of the database on first load and that process might take a little while, so be on the lookout for slowly opening databases. Thankfully this is only required once, but thereafter these databases won't work in an older build! Jumping back and forth between nightly/aurora/beta/release builds will not work.

Please file any issues you see in Core:DOM::IndexedDB. Thanks!

-bent

[Andrew Sutherland]

On Wed, Apr 1, 2015, at 03:02 PM, ben turner (bent) wrote:
> In the meantime, if the "dom.indexedDB.experimental" pref is set
> (defaults to |false| in Firefox and |true| in B2G, I think)

I don't think it's set to true for B2G right now. I don't see such a
mapping in
https://dxr.mozilla.org/mozilla-central/source/b2g/app/b2g.js, or
elsewhere with a search of
https://dxr.mozilla.org/mozilla-central/search?q=dom.indexedDB.experimental&case=false.
And there doesn't seem to be code in gaia/build that messes with the
pref either. I also attached the WebIDE to my trunk Flame device and
did "Runtime...Preferences" and searched for the pref, and I found it to
have the value false there.

Andrew

[Andrew Sutherland]

On Wed, Apr 1, 2015, at 03:02 PM, ben turner (bent) wrote:

> If a crash or power loss occurs at just
> the right moment then the transaction will be lost/rolled back. It should
> still be impossible to ever see database corruption though. This will
> mean faster delivery of "complete" events, and more closely aligns with
> the performance vs. stability tradeoffs other browser vendors have
> chosen.

Can you clarify the risk profile a little more? Specifically:

- Crash-wise, are we talking about only the parent process crashing, or
are we talking about the child process crashing too?

- For power loss for B2G on mobile, battery-powered devices, the
following sources of power loss seem to come to mind:
1. Device shutdown via the UI
2. User pulls the battery. (More likely, QA pull the battery :)
3. Hardware-shutdown via long-hold of the power button. (Because of
apparent device lockup, user is QA, or accidental triggering due to
pockets/device placement.)
4. Device runs out of power.

It seems like we can probably avoid the problems during 1: normal device
shutdown. Is it your/anyone's understanding that gecko and IndexedDB
will get a chance to cleanly shutdown and an fsync will happen? I don't
think we can do anything for 2: battery pulled (although if there's a
back-cover-sensor...) For 3: long-power-button-hold it seems like
depending on how extensively things are wedged, we could notice at say 6
seconds that we're headed for a shutdown and try to trigger an fsync and
put a stop to new transactions. For 4: low-power, ideally there's just
a point that the device decides it's not safe to operate and shuts
itself down, and that's slightly before it would result in IndexedDB
badness.

My main question is what is a realistic risk model for power loss/crash
and are there mitigations we can put in place to reduce the risk to
users?

The best example I have of this is when a QA tester was adding a contact
to the contacts app and then pulling the battery on the device. This
did find real bugs, but in terms of use-cases to engineer for, I don't
think it's a valid use-case to add a contact and then immediately pull
the battery. I do think it's a valid use-case to add a contact and then
shutdown the device via the UI once the user has seen implied visual
confirmation that the operation has completed. (I think in the case of
that bug, the window of vulnerability was shockingly long.)

On the flip side, desktop users on mains power are usually going to have
a failure model that looks like 2: QA pulling the battery when the mains
go out. But at the same time, it's understood that if your computer
loses power, you very well may lose some work and a UPS may be advisable
on unstable power. The issue there is how long the IndexedDB window of
vulnerability is.

Andrew

[Andrew Sutherland]

On Wed, Apr 1, 2015, at 05:12 PM, Andrew Sutherland wrote:
> On Wed, Apr 1, 2015, at 03:02 PM, ben turner (bent) wrote:

> > If a crash or power loss occurs at just
> > the right moment then the transaction will be lost/rolled back. It should
> > still be impossible to ever see database corruption though. This will
> > mean faster delivery of "complete" events, and more closely aligns with
> > the performance vs. stability tradeoffs other browser vendors have
> > chosen.
>

> Can you clarify the risk profile a little more? Specifically:

And I forgot one other thing:

- Wake lock interaction. Could there be a problem if an application
drops its wake-lock in the oncomplete notification for the transaction?
(Or does IndexedDB hold a wake-lock to cover this and/or the kernel take
care to flush dirty pages to disk before suspending/etc.?)

Andrew

[ben turner (bent)]

On Wednesday, April 1, 2015 at 1:40:07 PM UTC-7, somb...@gmail.com wrote:
> I don't think it's set to true for B2G right now.

Oops, you're right. It looks like it's false everywhere currently.

-bent

[ben turner (bent)]

On Wednesday, April 1, 2015 at 2:12:40 PM UTC-7, somb...@gmail.com wrote:
> - Crash-wise, are we talking about only the parent process crashing, or
> are we talking about the child process crashing too?

I was talking just about the parent process. If the child process crashes then whether or not the transaction is durable depends on whether or not the parent received the commit message and processed it (otherwise we automatically abort any outstanding transactions). That behavior is unchanged.

> 1. Device shutdown via the UI

This should be fine, we will flush to disk before actually powering down.

> 2. User pulls the battery.

> 3. Hardware-shutdown via long-hold of the power button.

> 4. Device runs out of power.

These three cases may fail to flush the data to disk, and if so when we restart the transaction will be rolled back.

> For 3: long-power-button-hold it seems like
> depending on how extensively things are wedged, we could notice at say 6
> seconds that we're headed for a shutdown and try to trigger an fsync and
> put a stop to new transactions.

That sounds reasonable, assuming we get that notification at the gecko layer? I am not sure what kind of info we get when long-presses happen.

> For 4: low-power, ideally there's just
> a point that the device decides it's not safe to operate and shuts
> itself down, and that's slightly before it would result in IndexedDB
> badness.

I don't know, but if it goes through normal shutdown then we will be fine.

> My main question is what is a realistic risk model for power loss/crash
> and are there mitigations we can put in place to reduce the risk to
> users?

> ...

> The issue there is how long the IndexedDB window of vulnerability is.

Flushing happens automatically after the database is idle (i.e. no active transactions) for 2 seconds at present. If the database never goes idle for that long then we continue to journal until the WAL size exceeds 20MB on desktop or 10MB on mobile.

> - Wake lock interaction.

Good question. I don't know of any testing in this area. IndexedDB does not hold an explicit wake lock at present.

-bent

[Jonas Sicking]

On Thu, Apr 2, 2015 at 3:00 AM, ben turner (bent)
<bent.m...@gmail.com> wrote:
> On Wednesday, April 1, 2015 at 2:12:40 PM UTC-7, somb...@gmail.com wrote:
>> - Crash-wise, are we talking about only the parent process crashing, or
>> are we talking about the child process crashing too?
>
> I was talking just about the parent process. If the child process crashes then whether or not the transaction is durable depends on whether or not the parent received the commit message and processed it (otherwise we automatically abort any outstanding transactions). That behavior is unchanged.

That doesn't sound entirely right.

As soon as the child process received the last "success" event for a
given request in a transaction, and we've sent the "go ahead and
commit" message to the parent, then I would expect that it's ok for
the child to crash at any point. This might match what you are saying.

However the more important question that I believe Andrew is asking,
is if we receive a "commit" event, what can then crash without there
being a dataloss. My understanding is that at that point both the
child and the parent can crash. As long as the kernel doesn't panic,
or the battery runs out or is removed, the data will be written.

I.e. once the "commit" event fires, the data has been transferred to
the OS, and as long as the OS shuts down cleanly, or has time to
flush, the data will be safe.

>> 1. Device shutdown via the UI
>
> This should be fine, we will flush to disk before actually powering down.
>
>> 2. User pulls the battery.
>> 3. Hardware-shutdown via long-hold of the power button.
>> 4. Device runs out of power.
>
> These three cases may fail to flush the data to disk, and if so when we restart the transaction will be rolled back.
>
>> For 3: long-power-button-hold it seems like
>> depending on how extensively things are wedged, we could notice at say 6
>> seconds that we're headed for a shutdown and try to trigger an fsync and
>> put a stop to new transactions.
>
> That sounds reasonable, assuming we get that notification at the gecko layer? I am not sure what kind of info we get when long-presses happen.

I would think that when long-press happens, the CPU simply receives a
reset signal. So it's equivalent to pulling the battery and putting it
back.

But maybe the OS is sent a signal a few seconds prior to that which
would encourage it to flush. I'm not sure.

>> For 4: low-power, ideally there's just
>> a point that the device decides it's not safe to operate and shuts
>> itself down, and that's slightly before it would result in IndexedDB
>> badness.
>
> I don't know, but if it goes through normal shutdown then we will be fine.

I'm pretty sure this results in the OS getting shut down cleanly, and
so will flush any data it still holds. I.e. the data should be safe
given my answer at the top.

>> My main question is what is a realistic risk model for power loss/crash
>> and are there mitigations we can put in place to reduce the risk to
>> users?
>> ...
>> The issue there is how long the IndexedDB window of vulnerability is.
>
> Flushing happens automatically after the database is idle (i.e. no active transactions) for 2 seconds at present. If the database never goes idle for that long then we continue to journal until the WAL size exceeds 20MB on desktop or 10MB on mobile.

But prior to that flushing the data has been write()ten, right? So the
OS will take care of flushing it eventually as long as it's shut down
cleanly.

/ Jonas

[Alex Webster]

Please unsubscribe me

On Wed, Apr 1, 2015 at 9:26 PM Jonas Sicking <jo...@sicking.cc> wrote:

> On Thu, Apr 2, 2015 at 3:00 AM, ben turner (bent)
> <bent.m...@gmail.com> wrote:

> > On Wednesday, April 1, 2015 at 2:12:40 PM UTC-7, somb...@gmail.com
> wrote:
> >> - Crash-wise, are we talking about only the parent process crashing, or
> >> are we talking about the child process crashing too?
> >
> > I was talking just about the parent process. If the child process
> crashes then whether or not the transaction is durable depends on whether
> or not the parent received the commit message and processed it (otherwise
> we automatically abort any outstanding transactions). That behavior is
> unchanged.
>

> That doesn't sound entirely right.
>
> As soon as the child process received the last "success" event for a
> given request in a transaction, and we've sent the "go ahead and
> commit" message to the parent, then I would expect that it's ok for
> the child to crash at any point. This might match what you are saying.
>
> However the more important question that I believe Andrew is asking,
> is if we receive a "commit" event, what can then crash without there
> being a dataloss. My understanding is that at that point both the
> child and the parent can crash. As long as the kernel doesn't panic,
> or the battery runs out or is removed, the data will be written.
>
> I.e. once the "commit" event fires, the data has been transferred to
> the OS, and as long as the OS shuts down cleanly, or has time to
> flush, the data will be safe.
>

> >> 1. Device shutdown via the UI
> >
> > This should be fine, we will flush to disk before actually powering down.
> >
> >> 2. User pulls the battery.
> >> 3. Hardware-shutdown via long-hold of the power button.
> >> 4. Device runs out of power.
> >
> > These three cases may fail to flush the data to disk, and if so when we
> restart the transaction will be rolled back.
> >
> >> For 3: long-power-button-hold it seems like
> >> depending on how extensively things are wedged, we could notice at say 6
> >> seconds that we're headed for a shutdown and try to trigger an fsync and
> >> put a stop to new transactions.
> >
> > That sounds reasonable, assuming we get that notification at the gecko
> layer? I am not sure what kind of info we get when long-presses happen.
>

> I would think that when long-press happens, the CPU simply receives a
> reset signal. So it's equivalent to pulling the battery and putting it
> back.
>
> But maybe the OS is sent a signal a few seconds prior to that which
> would encourage it to flush. I'm not sure.
>

> >> For 4: low-power, ideally there's just
> >> a point that the device decides it's not safe to operate and shuts
> >> itself down, and that's slightly before it would result in IndexedDB
> >> badness.
> >
> > I don't know, but if it goes through normal shutdown then we will be
> fine.
>

> I'm pretty sure this results in the OS getting shut down cleanly, and
> so will flush any data it still holds. I.e. the data should be safe
> given my answer at the top.
>

> >> My main question is what is a realistic risk model for power loss/crash
> >> and are there mitigations we can put in place to reduce the risk to
> >> users?
> >> ...
> >> The issue there is how long the IndexedDB window of vulnerability is.
> >
> > Flushing happens automatically after the database is idle (i.e. no
> active transactions) for 2 seconds at present. If the database never goes
> idle for that long then we continue to journal until the WAL size exceeds
> 20MB on desktop or 10MB on mobile.
>

> But prior to that flushing the data has been write()ten, right? So the
> OS will take care of flushing it eventually as long as it's shut down
> cleanly.
>
> / Jonas

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Daniel Holbert]

You've now sent 3 "please unsubscribe me" posts -- I don't think those
have any effect, aside from spamming everyone else on the list.

If you want to unsubscribe, you can do so via this link (which is
included at the bottom of every email you receive from this list):

https://lists.mozilla.org/listinfo/dev-platform

Unsubscription instructions are available at the bottom of that page.

~Daniel

On 04/01/2015 06:39 PM, Alex Webster wrote:
> Please unsubscribe me
> On Wed, Apr 1, 2015 at 9:26 PM Jonas Sicking <jo...@sicking.cc> wrote:
>
>> On Thu, Apr 2, 2015 at 3:00 AM, ben turner (bent)
>> <bent.m...@gmail.com> wrote:

>>> On Wednesday, April 1, 2015 at 2:12:40 PM UTC-7, somb...@gmail.com
>> wrote:
>>>> - Crash-wise, are we talking about only the parent process crashing, or
>>>> are we talking about the child process crashing too?
>>>
>>> I was talking just about the parent process. If the child process
>> crashes then whether or not the transaction is durable depends on whether
>> or not the parent received the commit message and processed it (otherwise
>> we automatically abort any outstanding transactions). That behavior is
>> unchanged.
>>

>> That doesn't sound entirely right.
>>
>> As soon as the child process received the last "success" event for a
>> given request in a transaction, and we've sent the "go ahead and
>> commit" message to the parent, then I would expect that it's ok for
>> the child to crash at any point. This might match what you are saying.
>>
>> However the more important question that I believe Andrew is asking,
>> is if we receive a "commit" event, what can then crash without there
>> being a dataloss. My understanding is that at that point both the
>> child and the parent can crash. As long as the kernel doesn't panic,
>> or the battery runs out or is removed, the data will be written.
>>
>> I.e. once the "commit" event fires, the data has been transferred to
>> the OS, and as long as the OS shuts down cleanly, or has time to
>> flush, the data will be safe.
>>

>>>> 1. Device shutdown via the UI
>>>
>>> This should be fine, we will flush to disk before actually powering down.
>>>
>>>> 2. User pulls the battery.
>>>> 3. Hardware-shutdown via long-hold of the power button.
>>>> 4. Device runs out of power.
>>>
>>> These three cases may fail to flush the data to disk, and if so when we
>> restart the transaction will be rolled back.
>>>
>>>> For 3: long-power-button-hold it seems like
>>>> depending on how extensively things are wedged, we could notice at say 6
>>>> seconds that we're headed for a shutdown and try to trigger an fsync and
>>>> put a stop to new transactions.
>>>
>>> That sounds reasonable, assuming we get that notification at the gecko
>> layer? I am not sure what kind of info we get when long-presses happen.
>>

>> I would think that when long-press happens, the CPU simply receives a
>> reset signal. So it's equivalent to pulling the battery and putting it
>> back.
>>
>> But maybe the OS is sent a signal a few seconds prior to that which
>> would encourage it to flush. I'm not sure.
>>

>>>> For 4: low-power, ideally there's just
>>>> a point that the device decides it's not safe to operate and shuts
>>>> itself down, and that's slightly before it would result in IndexedDB
>>>> badness.
>>>
>>> I don't know, but if it goes through normal shutdown then we will be
>> fine.
>>

>> I'm pretty sure this results in the OS getting shut down cleanly, and
>> so will flush any data it still holds. I.e. the data should be safe
>> given my answer at the top.
>>

>>>> My main question is what is a realistic risk model for power loss/crash
>>>> and are there mitigations we can put in place to reduce the risk to
>>>> users?
>>>> ...
>>>> The issue there is how long the IndexedDB window of vulnerability is.
>>>
>>> Flushing happens automatically after the database is idle (i.e. no
>> active transactions) for 2 seconds at present. If the database never goes
>> idle for that long then we continue to journal until the WAL size exceeds
>> 20MB on desktop or 10MB on mobile.
>>

[ben turner (bent)]

On Wednesday, April 1, 2015 at 6:26:04 PM UTC-7, Jonas Sicking wrote:
> This might match what you are saying.

Yep!

> My understanding is that at that point both the
> child and the parent can crash.

Well, it's not enough to just receive the commit message from the child. We then have to actually switch threads to the db thread and tell SQLite to commit the transaction. Once that completes I believe both processes could crash and the data would eventually be saved to disk (barring loss of power, etc).

-bent

[Jonas Sicking]

We don't fire the "commit" event until after we've told SQLite to
commit and it's come back to tell us that the commit was successful,
do we?

/ Jonas

[ben turner (bent)]

On Thursday, April 2, 2015 at 5:01:12 PM UTC-7, Jonas Sicking wrote:
> We don't fire the "commit" event until after we've told SQLite to
> commit and it's come back to tell us that the commit was successful,
> do we?

You mean the "complete" event, but yes, that's correct.

-bent

[Gian-Carlo Pascutto]

> The IndexedDB API does not currently have a way to say "no, really, I
> want to make sure that this important data is saved to disk before I
> continue".

Do our internal APIs offer this?

Android can kill Firefox at any time it requires more memory. It seems
prudent to at least commit when we're being backgrounded, or you're
going to be in the dataloss scenario a hell of a lot of times.

--
GCP

[Jonas Sicking]

I tried to explain this, but I don't think it was particularly clear.

Once the "complete" event has fired, the data has been completely
transferred to the OS. At that point, any gecko process crashing or
getting killed will not be a problem. Only if the kernel panics, or if
the battery is pulled out, or if the device is hit by one of these[1],
will there be dataloss.

[1] http://www.hutchinsontransmission.com/sites/default/files/8steamroller-poly-v.png

/ Jonas

[Gian-Carlo Pascutto]

On 4/04/2015 12:22, Jonas Sicking wrote:

> I tried to explain this, but I don't think it was particularly clear.
>
> Once the "complete" event has fired, the data has been completely
> transferred to the OS. At that point, any gecko process crashing or
> getting killed will not be a problem.

Then this is probably the source of the confusion:

"If a crash or power loss occurs at just the right moment then the
transaction will be lost/rolled back"

"crash" in that sentence means "kernel panic" not "Gecko crash", right?

--
GCP

[Jonas Sicking]

That's correct as far as I understand it. Ben should confirm.

/ Jonas

[ben turner (bent)]

On Saturday, April 4, 2015 at 9:02:32 AM UTC-7, Jonas Sicking wrote:
> > "crash" in that sentence means "kernel panic" not "Gecko crash", right?
>
> That's correct as far as I understand it. Ben should confirm.

Yep, should have said "system crash" or "OS crash" there.

-bent

[Botond Ballo]

> 2. The the schema version for all databases has changed, so once you open a database on trunk builds and it upgrades you will not be able to open that same database in an aurora/beta/release build.

Have we considered issuing a warning saying that after upgrading the
profile, it will not be compatible with older versions?

Cheers,
Botond

[J. Ryan Stinnett]

On Wed, Apr 8, 2015 at 2:30 PM, Botond Ballo <bba...@mozilla.com> wrote:
> Have we considered issuing a warning saying that after upgrading the
> profile, it will not be compatible with older versions?

I would really like to have something like this. WebIDE makes use of
IndexedDB today, and it seems to trigger uncatchable errors in some
cases when an older client is used with newer profile (see WebIDE bug
1117129), so it's hard to handle at the application level today.

Some people (at least at Mozilla) move their profiles between up and
down versions regularly, so we should at least look for a way to offer
a nice warning.

- Ryan

[Jan Varga]

This is a good idea.

On 08/04/15 21:30, Botond Ballo wrote:
>> 2. The the schema version for all databases has changed, so once you open a database on trunk builds and it upgrades you will not be able to open that same database in an aurora/beta/release build.

> Have we considered issuing a warning saying that after upgrading the
> profile, it will not be compatible with older versions?
>

> Cheers,
> Botond

[Jonas Sicking]

On Wed, Apr 8, 2015 at 12:40 PM, J. Ryan Stinnett <jry...@gmail.com> wrote:
> On Wed, Apr 8, 2015 at 2:30 PM, Botond Ballo <bba...@mozilla.com> wrote:

>> Have we considered issuing a warning saying that after upgrading the
>> profile, it will not be compatible with older versions?
>

> I would really like to have something like this. WebIDE makes use of
> IndexedDB today, and it seems to trigger uncatchable errors in some
> cases when an older client is used with newer profile (see WebIDE bug
> 1117129), so it's hard to handle at the application level today.
>
> Some people (at least at Mozilla) move their profiles between up and
> down versions regularly, so we should at least look for a way to offer
> a nice warning.

For what it's worth, I doubt that this is an IndexedDB specific issue.
Any time we change the format for anything that we store, cookies,
addons registry, session history, Fx account details, form fill data,
etc, we make it impossible to downgrade.

That said, I also fairly regularly switch between nightly and release.
So I agree that it'd be good to have some form of global profile
version which at least gives a warning if you're running an "old"
firefox on a "new" profile.

/ Jonas