In memory snapshots with MVStore

[Max Englander]

Hello!

I am using H2 as an in-memory substitute for MySQL to run a portion of my application's test suite. The way I am using it currently looks something like this:

1. Before test suite runs
1. Start H2 server
2. Run Liquibase migrations
2. Before each test runs
1. Seed database with data specific to test
3. Run test
4. After each test runs
1. Remove test-specific seed data (to avoid unique constraint violations, etc.)
5. After test suite finishes running
1. Shut down H2

Setting up and tearing down seed data is a bit of a hassle. I've thought that one way I might ease this burden is to eliminate the seed tear-down step. I've considered running the Liquibase migrations before each test runs, but that would be very time consuming. I've also thought of backing up the database after running the Liquibase migrations, and restoring the backup after each test. This would involve a lot of disk writes, and I imagine would be similarly time consuming.

One option I'm interested in exploring is using the snapshot feature of the underlying MVStore. My questions are:

1. Is it currently possible to have my test suite run against the in-memory TcpServer, and, before each test runs, access the underlying MVStore to restore to a snapshot version?
2. If it is currently possible..is it safe?

Thanks for reading, and for the fantastic tool!

Best,

Max 

[Noel Grandin]

I suspect you might have more luck by creating a custom FilePath that wraps the existing FilePathMem (i.e. the jdbc h2:mem: functionality) which you can use to copy/restore the db.

You'd still need to open/close H2, but you could avoid the seed data step.

With some real cleverness, you could define a snapshot point, and store modifications on top of that, and then just throw the modifications away.

​

[Max Englander]

Interesting, thanks for the suggestion! 

Looking through the code now I see that MVStore uses FileStore which uses FilePath which maintains a publicly accessible registry of FilePath implementations, where I could register, for example, a CustomFilePathMem. Cool!

Looking into the fs package, I see that ByteBuffer is used to pass data to and from FileBase, and that, in the case of FilePathMem, data in ByteBuffers makes ultimately makes its way into a byte array in FileMemData. I imagine, for my purposes, that I should be able to simply copy all of the underlying byte data stored in FileMemData after running my liquibase migrations and then, between each test, close/open H2 and restore the backed up byte data into CustomFileMemPath.

Thanks, Noel!

[Max Englander]

I finally got around to working on this, and ended up using the steps outline here, which worked great for my purposes.

Using "SET EXCLUSIVE 2" and "CHECKPOINT SYNC" turns out to be necessary; without, I run into sporadic issues, e.g. tables not being found, missing triggers, etc.

There are still some things I don't quite understand, such as:

1. Why exactly the absence of "SET EXCLUSIVE" and "CHECKPOINT SYNC" occasionally results in failures? I think that I am being diligent about terminating existing operations to the primary database before copying. Perhaps there's no easy/obvious answer here.
2. What exactly is the different between the "*FS" class of databases and the "mem" database?
3. Is it possible to backup and restore to a "mem" database similar to "*FS" databases, e.g. with {input,output} streams? What part of the code base is responsible for "mem" databases?

Thanks for the great tool :-)

Best,

Max

[Noel Grandin]

On 25 October 2017 at 22:07, Max Englander <max.en...@gmail.com> wrote:

I finally got around to working on this, and ended up using the steps outline here, which worked great for my purposes.

Using "SET EXCLUSIVE 2" and "CHECKPOINT SYNC" turns out to be necessary; without, I run into sporadic issues, e.g. tables not being found, missing triggers, etc.

There are still some things I don't quite understand, such as:

1. Why exactly the absence of "SET EXCLUSIVE" and "CHECKPOINT SYNC" occasionally results in failures? I think that I am being diligent about terminating existing operations to the primary database before copying. Perhaps there's no easy/obvious answer here.

there is a background writer which has probably not finished flushing it's state to the file system 

1. What exactly is the different between the "*FS" class of databases and the "mem" database?

in-memory vs on-disk 

1. Is it possible to backup and restore to a "mem" database similar to "*FS" databases, e.g. with {input,output} streams? What part of the code base is responsible for "mem" databases?

yes. It's in the same place as the other FilePath stuff. Although there are a couple of places where we special-case in-memory stuff.

[Max Englander]

Interesting - I couldn't quite figure out how "mem" mapped to any FilePath implementation. I didn't see any "mem" scheme or any logic mapping it to "memFS", but maybe I missed it.

Thanks for the answers!

Best,
Max

--
You received this message because you are subscribed to a topic in the Google Groups "H2 Database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/h2-database/0J5ZHlxVptY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to h2-database...@googlegroups.com.
To post to this group, send email to h2-da...@googlegroups.com.
Visit this group at https://groups.google.com/group/h2-database.
For more options, visit https://groups.google.com/d/optout.

[Noel Grandin]

FilePathMem and related classes.​