OS.File and shutdown

[Felipe Gomes]

Hi, does OS.File guarantees that write tasks that have started will be completed if a shutdown occurs? My use case is for writeAtomic but I'm interested about the behavior of both write and writeAtomic.
Corner case: what if I call write/writeAtomic from an xpcom-shutdown observer?

Another question: are the write tasks queued and completed in order, or can two writeAtomic calls to the same file race each other and the 2nd call finish first (only to have the 1st call finish and write older data)

Thanks,
Felipe

[Benjamin Smedberg]

On 5/10/13 4:45 PM, Felipe Gomes wrote:
> Hi, does OS.File guarantees that write tasks that have started will be completed if a shutdown occurs? My use case is for writeAtomic but I'm interested about the behavior of both write and writeAtomic.
> Corner case: what if I call write/writeAtomic from an xpcom-shutdown observer?

I'm not sure about the rest of this question. But you should not be
performing any I/O after profile-before-change. See
https://wiki.mozilla.org/XPCOM_Shutdown and note that after
profile-before-change, we are working on immediately exiting the browser
and skipping XPCOM shutdown. xpcom-shutdown is definitely too late!

--BDS

[David Rajchenbach-Teller]

On 5/10/13 10:45 PM, Felipe Gomes wrote:
> Hi, does OS.File guarantees that write tasks that have started will be completed if a shutdown occurs? My use case is for writeAtomic but I'm interested about the behavior of both write and writeAtomic.
> Corner case: what if I call write/writeAtomic from an xpcom-shutdown observer?

In theory:
- every call to OS.File queued *before* xpcom-shutdown will be completed;
- every call to OS.File queued after xpcom-sthudown will throw an
asynchronous exception (once bug 845190 has landed).

Note, however, that this has not been thoroughly tested yet.

> Another question: are the write tasks queued and completed in order, or can two writeAtomic calls to the same file race each other and the 2nd call finish first (only to have the 1st call finish and write older data)

All tasks to OS.File are queued and completed in order.

Cheers,
David

--
David Rajchenbach-Teller, PhD
Performance Team, Mozilla

[David Rajchenbach-Teller]

On 5/13/13 10:20 PM, David Rajchenbach-Teller wrote:
> On 5/10/13 10:45 PM, Felipe Gomes wrote:

>> Hi, does OS.File guarantees that write tasks that have started will be completed if a shutdown occurs? My use case is for writeAtomic but I'm interested about the behavior of both write and writeAtomic.
>> Corner case: what if I call write/writeAtomic from an xpcom-shutdown observer?
>

> In theory:
> - every call to OS.File queued *before* xpcom-shutdown will be completed;
> - every call to OS.File queued after xpcom-sthudown will throw an
> asynchronous exception (once bug 845190 has landed).

Actually, what I wrote was misleading. Calls to OS.File queued during
xpcom-shutdown will be honoured *if the OS.File thread has already been
initialized* before xpcom-shutdown � this is the case in Firefox itself,
but not necessarily in xpcshell tests.

[Felipe Gomes]

On Friday, May 10, 2013 7:53:01 PM UTC-3, Benjamin Smedberg wrote:
> I'm not sure about the rest of this question. But you should not be
>
> performing any I/O after profile-before-change. See
>
> https://wiki.mozilla.org/XPCOM_Shutdown and note that after
>
> profile-before-change, we are working on immediately exiting the browser
>
> and skipping XPCOM shutdown. xpcom-shutdown is definitely too late!

Okay. The code i'm touching does that on xpcom-shutdown but since it'll be wrong in the future I'll change it while I'm there. (it finishes some pending async sqlite stmts and rolls back any ongoing transactions).


What prompted the question is that I'm working on a conversion from SQLite storage to a JSON file. OS.File.writeAtomic provides a good guarantee of data consistency against crashes etc, and I'm now looking how to guarantee a proper full flush of the data to disk during shutdown.

Should profile-before-change then be my call to stop accepting changes to the data and call writeAtomic to flush it? I've seen some code nearby doing it at quit-application-granted. Or perhaps there's no "correct" answer and it varies case by case (or anything goes that works and is early enough..)

[Gregory Szorc]

This sounds very similar to what I ran into with Firefox Health Report.

There is some discussion in
https://bugzilla.mozilla.org/show_bug.cgi?id=722648 and
https://bugzilla.mozilla.org/show_bug.cgi?id=850712.

Generally speaking, you have a queued async operation or set of async
operations that need to complete. However, profile-before-change is
observed before or during those operations. There is a race between
xpcom-shutdown and those async operations finishing.

FHR's solution to this problem is to initiate the shutdown procedure
early during shutdown. We chose quit-application. If this shutdown
sequence (which consists of waiting for async operations initiated or
queued before shutdown started) has not completed by the time
profile-before-change (the last observer before the profile goes away)
is observed, FHR creates a nested event loop during the
profile-before-change handler. This delays shutdown. But it eliminates
race conditions and prevents data corruption. Code at
https://hg.mozilla.org/mozilla-central/file/26ab72bfa9df/services/healthreport/healthreporter.jsm#l445

Performance *really* doesn't like this solution because it can delay
shutdown. However, the last I checked Telemetry, we only had 121 reports
on Nightly and Aurora of this occurring in the past month. This is
because we only create a nested event loop if shutdown hasn't finished
by profile-before-change and unless a major FHR operation was initiated
just before shutdown, there will be no work to do except close the
database handle. If we were buffering data and persistently flushing,
we'd likely be spinning much more since we'd be more likely to have
queued I/O.

I'll conclude by saying that every scenario is different. I highly
recommend you connect with someone on the Perf team for a tailored
recommendation.

[David Rajchenbach-Teller]

On 5/14/13 8:35 PM, Felipe Gomes wrote:
> Should profile-before-change then be my call to stop accepting changes to the data and call writeAtomic to flush it? I've seen some code nearby doing it at quit-application-granted. Or perhaps there's no "correct" answer and it varies case by case (or anything goes that works and is early enough..)

profile-before-change should be good. Any OS.File call posted before
xpcom-shutdown will be completed before we exit Firefox.

[Robert Kaiser]

Felipe Gomes schrieb:

> What prompted the question is that I'm working on a conversion from SQLite storage to a JSON file. OS.File.writeAtomic provides a good guarantee of data consistency against crashes etc, and I'm now looking how to guarantee a proper full flush of the data to disk during shutdown.

Just make sure you do (lazily but still) also flush that data out to
disk every now and then during normal operation. For one thing, people
leave Firefox running for hours, days and even weeks and if they run
into a crash, they shouldn't lose state over such long periods, and for
the other, we want shutdown to be fast and the ideal situation is that
you already have written everything when the shutdown request comes and
you don't need to care. ;-)

Robert Kaiser