Suitability of mvstore (stand-alone) in Android : robustness, processes killed, etc

[alexr...@gmail.com]

Hello, I would like to use mvstore in my Android application.

In Android, it is very common that your app (process) will get killed by the OS (no longer needed, even if it has a Service is running) or by the user (when swiped away from Recents).

The bundled sqlite is extremely robust. In fact, if you look at most Android apps that use sqlite - it is opened / initialized in a Singleton or ContentProvider and is never closed.

This is because the app usually doesn't have a chance to shut things down cleanly, the process is simply killed.

So I would like to know, how well would mvstore do in this scenario? I would open the store, and leave it open for the app to use, but never really find a good moment to close() it.

If data from in-flight transactions is lost, that is fine of course - but will the integrity of the file be OK?

Thanks

-Alex

[Thomas Mueller]

Hi,

The MVStore should work well with Android. The main differences to SQLite are: 

* SQLite overwrites / writes blocks of the size 4 KB. With flash or solid state disks this is not that great, as internally (by the firmware or OS) those writes are translated to much larger writes, for example 128 KB (this is called "write amplification" I think). H2 MVStore overwrites / writes chunks of size 1 MB. This is good for flash / solid state disks.

* SQLite first writes to the transaction log, then calls fsync, then writes to the main data area. H2 MVStore only writes to one file (one write per change).

* SQLite reclaims disk space much earlier; almost immediately. The H2 MVStore leaves old data for at least 45 seconds, which will result in higher (visible) disk space usage. On a lower, invisible level, for solid state disks and flash memory, the space usage is very similar.

* In a crash, for SQLite, the last committed transaction is typically stored. For MVStore, the last second of transactions is typically lost.

* The code of the MVStore is simpler, but SQLite is much older and more mature. The risk of bugs in SQLite is therefore smaller.

Regards,

Thomas

--
You received this message because you are subscribed to the Google Groups "H2 Database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to h2-database...@googlegroups.com.
To post to this group, send email to h2-da...@googlegroups.com.
Visit this group at http://groups.google.com/group/h2-database.
For more options, visit https://groups.google.com/d/optout.

[alexr...@gmail.com]

Hi Thomas, thank you very much for your response. That is very encouraging.

I would like clarification on one item: can I use mvstore reliably without calling .close().

There are no hooks in an Android application to cleanly dispose resources. This is why sqlite databases are opened and shared among components, but not closed ... because when the application is truly shut down by OS, the process is simply killed (and the timing of this is unpredictable).

Thanks & regards

-Alex

[Thomas Mueller]

Hi,

I would like clarification on one item: can I use mvstore reliably without calling .close().

Yes, that should work. However, please note that this scenario is not fully tested yet (as much as with SQLite). At the moment, I would either ensure it works for you with your own tests, or don't store very important data unless you do regular backups.

Regards,

Thomas

[Klaus Wuestefeld]

> At the moment, I would either ensure it works for you with your own tests,
> or don't store very important data unless you do regular backups.

Hi,

I am sad because I think I have to stop using mvstore.

I love mvstore as a product and I am using it in a web server and also in an Android p2p app platform called Sneer.

My continuous deploy solution on the web server pkills the process (regular pkill, not -9).

I have just had an mvstore file get currupted. :(

Please let us know when mvstore is ready for robust production use.

Thanks, Klaus

[Nicolas Fortin (OrbisGIS)]

Hi,

Using https://docs.oracle.com/javase/6/docs/api/java/util/concurrent/Future.html#cancel%28boolean%29 this method I was able to trigger corrupted MVStore. I could help by building unit test in this case. When statement.cancel( ) can't do the job, this kind of Thread kill can be somewhat useful.

Regards,

--

Nicolas Fortin
IRSTV FR CNRS 2488
GIS        http://orbisgis.org
Spatial DB http://h2gis.org
Noise      http://noisemap.orbisgis.org

[Thomas Mueller]

Hi,

That's very interesting. 

So far I know about a potential problem in case of power failure, and I'm working on fix for that. This is related to write re-ordering. Recovery is affected, and truncating the file needs to be delayed, but both should be quite simple to fix.

However, the problem you describe is not related to power failure. I would be very interested in a test case!

Regards,

Thomas

[Nicolas Fortin (OrbisGIS)]

Hi,

I try to write the unit test with thread killing. As I can't reconnect to the DB I'm facing some problem to check if the database is corrupted. (see attached traceback)

https://github.com/nicolas-f/h2database/tree/thread_kill_mvstore

As I said before, I want to reproduce a problem of corrupted db we have in production when using this kind of "brutal" thread kill.

regards,

--

Nicolas Fortin
IRSTV FR CNRS 2488
GIS        http://orbisgis.org
Spatial DB http://h2gis.org
Noise      http://noisemap.orbisgis.org

Le jeudi 28 mai 2015 22:04:22 UTC+2, Thomas Mueller a écrit :

Hi,

[Thomas Mueller]

Hi,

The test interrupts the thread, which will close the database file (as always with FileChannel). Subsequent close calls will fail, this is a bug that I need to fix. However, I don't think this will corrupt the database file.

Interrupting the thread is one way to test it, killing the process is another. There are already such tests, and so far I couldn't reproduce the problem.

Additional tests are with a file system implementation that simulates re-ordering writes, and power failure. Simulating power failures I have, but re-ordering writes is still missing.

Then, real power failure tests are needed. This will require some work, but I think it is important. I will work on that.

Regards,

Thomas

[Klaus Wuestefeld]

Thanks Thomas! I believe simulation is the way to go. It can give you
more coverage than trying to perform real power failures.

> You received this message because you are subscribed to a topic in the

> Google Groups "H2 Database" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/h2-database/FPyYtNPbgJQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

> h2-database...@googlegroups.com.
> To post to this group, send email to h2-da...@googlegroups.com.
> Visit this group at http://groups.google.com/group/h2-database.
> For more options, visit https://groups.google.com/d/optout.

--
Valeu, Klaus.

[Thomas Mueller]

Hi,

Thanks Thomas! I believe simulation is the way to go. It can give you
more coverage than trying to perform real power failures.

Sure. But real power failures help understand what needs to be simulated.

Regards,

Thomas

[Thomas Mueller]

Hi,

I have now a test case where a MVStore file can not be opened after an out of memory. It doesn't look like the file is corrupt, it's just that the recovery code can not deal with this case yet. The stack trace is:

Caused by: java.lang.IllegalStateException: Chunk 2 is invalid [1.4.187/6]
  at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:773)
  at org.h2.mvstore.MVStore.readStoreHeader(MVStore.java:695)
  at org.h2.mvstore.MVStore.<init>(MVStore.java:348)

If you have stack traces of your failures (even better MVStore files), that would be great.

Regards,
Thomas

[Thomas Mueller]

Hi,

By the way, the "corruption after out of memory" is now fixed in the trunk.

Regards,

Thomas

[art...@amuse.io]

It's an old post but for anyone still searching i might have some insight about the viability of mvstore in android.

For the past year i've been using nitrite-db which is a nosql db that uses mvstore. The library itself is fine but i always have issues with mvstore corrupting the db file.

This mainly occurs during background operations (async tasks, threads, etc). After implementing multiple checks to not use the db if a thread is interrupted and making sure the db closes properly when the app closes i still get many crashes in the wild.

So my verdict is that it's unfit for android, at least when dealing with background operations.