Common snapshot support

[Martin Krasser]

Devs,

on branch wip-reusable-snapshotting, I'm currently working on a common snapshotting infrastructure for all journals using the Hadoop FileSystem abstraction (with implementations for local filesystem, HDFS, FTP, S3 ... etc ). Here's a brief description how to use the new features to implement snapshotting for a journal (links go to HBase journal):

- A journal must extend HadoopFilesystemSnashotting and call initSnapshotting in its start method.
- The journal props must implement HadoopFilesystemSnashotting

The same has been done for LevelDB and Journal.IO. I'll implement it for the Dynamodb journal soon.

@Duncan: I'm not sure if you want to go for the Hadoop FileSystem abstraction for snapshotting. If yes, I'm happy to implement that for the two MongoDB journals as well (should be done in a few minutes).

Further changes on this branch:

- Journal creation via Journal(JournalProps) deprecated - use JournalProps.createJournal instead
- Several refactorings

Please let me know if you see any issues with the changes on this branch. All changes and new features will be documented later in detail.

Thanks,
Martin

-- 
Martin Krasser

blog:    http://krasserm.blogspot.com
code:    http://github.com/krasserm
twitter: http://twitter.com/mrt1nz

[Martin Krasser]

Just wanted to let you know that I extended the mongodb-casbah journal (on branch wip-reusable snapshotting) so that is passes now the PersistentReplaySpec. To pass these tests a journal must support upper replay limits and snapshotting. At the moment snapshotting is implemented based on Hadoop FileSystem but we can see that implementation as preliminary should you want to switch to another snapshotting implementation.

I did not implement the PersistentReplaySpec for the mongodb-reactive journal and the dynamodb-journal. At a first glance, these journals already seem to support upper replay limits but this needs to be verified. Any help is highly appreciated here. Please let me know if you have any questions.

Thanks,
Martin

[scott.clasen]

Hey Martin

   The dynamo journal should be able to handle the upper replay limits pretty easily.

I'll try and take a crack at wiring up the reusable snapshotting when I get a chance

Cheers!

SC

[ddevore]

On Saturday, June 15, 2013 5:10:38 AM UTC-4, Martin Krasser wrote:

Just wanted to let you know that I extended the mongodb-casbah journal (on branch wip-reusable snapshotting) so that is passes now the PersistentReplaySpec. To pass these tests a journal must support upper replay limits and snapshotting. At the moment snapshotting is implemented based on Hadoop FileSystem but we can see that implementation as preliminary should you want to switch to another snapshotting implementation.

Martin, this looks great.  

 

I did not implement the PersistentReplaySpec for the mongodb-reactive journal and the dynamodb-journal. At a first glance, these journals already seem to support upper replay limits but this needs to be verified. Any help is highly appreciated here. Please let me know if you have any questions.

The reactive journal does support upper limits. After I finish the .9 upgrade on reactive (taking longer than I thought), I'll take a look at the reusable snapshotting as soon as I have that done.

[Martin Krasser]

Duncan and Scott,

thanks a lot for taking care of that. I meanwhile merged wip-reusable-snapshotting to master.

Cheers,
Martin

The preceding email message may contain confidential information of Viridity Energy, Inc. It is not intended for transmission to, or receipt by, any unauthorized persons. If you have received this message in error, please (i) do not read it, (ii) reply to the sender that you received the message in error, and (iii) erase or destroy the message.

[ddevore]

Hi Martin,

As it turns out Reactive .9 is still running on Akka 2.1.x, so I can branch Reactive and upgrade it to Akka 2.2.x or wait until they upgrade.

WDYT?

[Martin Krasser]

On 19.06.13 02:06, ddevore wrote:

Hi Martin,

As it turns out Reactive .9 is still running on Akka 2.1.x, so I can branch Reactive and upgrade it to Akka 2.2.x or wait until they upgrade.

creating a branch sounds good to me (unless they plan to upgrade within a week or two). I can upload the artifacts to the Eligotech Maven repo.

The upgrade is hopefully not a big deal if all the deprecation warnings are ignored - which is acceptable for a temporary solution IMO.

[ddevore]

Hi Martin,

It looks it may be a few weeks or longer before they up to 2.2 so I'll go ahead and branch. I'll let you know as soon as I have a working fork of Reactive on Akka 2.2, then as you mentioned we can upload them to the Eligotech repo. Once that's done, I can finish the changes to the journal.

Sound good?

[Martin Krasser]

+1

[ddevore]

Hi Martin,

I've updated (locally) ReactiveMongo to AKKA 2.2. Additionally I have upgraded the ReactiveMongoJournal to use this new version of the ReactiveMongo driver. All test pass including the EventsourcingSpec test. 

I'd like to go ahead and issue a pull request, but before I do that, I'll need to publish my local version of the ReactiveMongo driver to the Eligosource repository. How do you want me to go about that?

NOTE: I have not implemented snapshotting yet in the ReactiveMongo Journal, but I will handle that next.

[scott.clasen]

Starting on dynamo, and seeing some unexpected things in the test, this is probably the upper limit stuff that Im not totally grokking.

In the JournalSpec 'recover its counter' journal test, the journal is receiving a ReplayInMsgs with a StandardReplayParams(processorId = 1, fromSequenceNr = 4, toSequenceNr= 1)

What are the semantics there? shouldnt the toSequenceNr always be greater than the fromSequenceNr?  If not how should it behave?

[Martin Krasser]

Hi Scott,

On 26.06.13 01:42, scott.clasen wrote:

Starting on dynamo, and seeing some unexpected things in the test, this is probably the upper limit stuff that Im not totally grokking.

In the JournalSpec 'recover its counter' journal test, the journal is receiving a ReplayInMsgs with a StandardReplayParams(processorId = 1, fromSequenceNr = 4, toSequenceNr= 1)

This is the case when the journal doesn't properly restore the latest sequence number. The journal must be able to find the highest stored sequence number across all stored in and out messages during initialization. For this test, storedCounter should return 3 (and initialize _counter to 4, after the test re-creates a journal instance). If this is properly done, then the resequencer will receive a StandardReplayParams(processorId = 1, fromSequenceNr = 4, toSequenceNr= 4) and a single message (with sequenceNr 4) is replayed.

[scott clasen]

Aah, I think Im having some trouble in my spec journalProps  method.

Since there isnt a fast way to delete a journal in Dynamo, this is what the journalProps looks like, see the System.currentTimeMillis.

  def journalProps = {

    val key = sys.env("AWS_ACCESS_KEY_ID")

    val secret = sys.env("AWS_SECRET_ACCESS_KEY")

    val table = sys.env("TEST_TABLE")

    val app = System.currentTimeMillis().toString

    DynamoDBJournalProps(table, app, key, secret, counterShards = 10, system = ActorSystem("dynamo-test"))

  }

So in tests that recreate the journal ("anotherJournal")  they are getting a fresh journal and not the one expected with messages.

If I tweak the JournalSpec to call journalProps only once and cache in a val I get some different failures...

Ill play around with it some more.

[scott clasen]

Ok seems to be mostly working now. My internet connection is a bit dodgy so the only failures I am seeing are timeouts.

One thing I have done which will change all the JournalSpec impls is to change the signature of journalProps to 

    def journalProps(implicit sys:ActorSystem): JournalProps

Anyone have any issue with that? (The actorSystem used for testing is beign passed to and used by my spray dynamodb client)

 

[Martin Krasser]

Hi Scott,

On 26.06.13 21:43, scott clasen wrote:

Ok seems to be mostly working now. My internet connection is a bit dodgy so the only failures I am seeing are timeouts.

One thing I have done which will change all the JournalSpec impls is to change the signature of journalProps to 

    def journalProps(implicit sys:ActorSystem): JournalProps

Anyone have any issue with that? (The actorSystem used for testing is beign passed to and used by my spray dynamodb client)

why can't you (re)use context.system where context is that of the DynamoDBJournal actor? You're anyway creating new DynamoDBClient instances only inside the journal actor. This would also free DynamoDBJournalProps users from providing an ActorSystem.

[scott clasen]

Good call!

Added that commit.

[scott clasen]

Martin, Great job on the snapshotting! It was very straightforward to add support.

[Martin Krasser]

On 27.06.13 19:00, scott clasen wrote:

Martin, Great job on the snapshotting! It was very straightforward to add support.

Thanks, glad that it was easy to add.

[ddevore]

Martin,

Added snapshot support. Really like how you implemented common support, very nice.

Cheers,

[ddevore]

Hi Martin,

I've added write concern support (fsync) and updated the pull request.

Cheers,

[Martin Krasser]

Hi Duncan,

On 29.06.13 07:51, ddevore wrote:

Hi Martin,

I've added write concern support (fsync) and updated the pull request.

Great. I just added a comment on the PR, regarding closing connections redundnantly. Otherwise the PR looks good to me.

Cheers,

On Saturday, June 29, 2013 1:00:19 AM UTC-4, ddevore wrote:

Martin,

Added snapshot support. Really like how you implemented common support, very nice.

Thanks.

[ddevore]

Thanks Martin, I've made the suggested change and updated the PR.

[Martin Krasser]

On 29.06.13 09:26, ddevore wrote:

Thanks Martin, I've made the suggested change and updated the PR.

Just merged. Thanks a lot!

[ddevore]

Martin,

Do you think we should update all the Journal Readme's to reflect snapshotting or is the general Readme good enough?

[Martin Krasser]

On 29.06.13 17:08, ddevore wrote:

Martin,

Do you think we should update all the Journal Readme's to reflect snapshotting or is the general Readme good enough?

We should at least link from the jorunal docs to this section (and optionally extend it a bit). Sooner or later, I want to give the journal documentation a bit more love (e.g. have it better organized, extended and less scattered) but I have only very vague ideas how this should look like. Any suggestions are welcome.

[ddevore]

Okay I'll add that to the Mongo Readme's.