leveldb db corruption on brownout and shutdown?

[Joel de Guzman]

Hello,

We're using leveldb in an embedded system (linux) that may be powered off without shutting down the OS.

The system may also restart when some failure is detected. Although it is not 100% confirmed yet, it seems 

we're experiencing db corruption when that happens. Is leveldb resilient on unexpected power outages/

brownouts? is there any other key-value store library that is resilient on such in the first place? Or is it not

a good idea to use a database like leveldb in such a use-case?

Thoughts?

Thanks in advance,

--Joel

[Florian Weimer]

* Joel de Guzman:

> we're experiencing db corruption when that happens. Is leveldb
> resilient on unexpected power outages/ brownouts?

I have forgotten what my review of the leveldb sources for this aspect
yielded a while back, sorry (and fixes may have went in since then).

The larger problem is cheap storage lying about write completion, or
kernels which do not even attempt to implement barriers correctly.

Most key-value stores/databases assume that some amount of data is
written atomically. This is supposed to match spinning disk behavior
(unless you have IBM DTLAs, which may require a sector overwrite to
revive a sector which was written during drive power-down). With
other kinds of storage (such as degraded RAID-5), a one-sector write
can wipe out much more than just a single sector during a power fault.

Berkeley DB assumes that entire (multi-sector) pages are written
atomically, which is probably not true in general. On the other hand,
SQLite and PostgreSQL should cope rather well with a storage subsystem
which implements barriers properly, without requiring anything
further.

(But some versions of SQLite skip a barrier before returning success
to the application, and this is very difficult to change without
recompiling SQLite. As a result, one transaction can be rolled back
after a power outage, even if its successful completion was reported
to the application. SQLite database consistency is not affected,
though.)

[Robert Escriva]

You may find some answers in this paper and its related research:
https://www.usenix.org/node/186195

The authors managed to find real application-level bugs in various data
storage backends on various filesystems.

The answer to your question of what is resilient, it turns out, is very
heavily dependent upon both the storage backend you use, and the
filesystem it accesses.

-Robert

[Joel de Guzman]

Thank you all for your replies!