Persisting entirely in JSON

[Alan Burlison]

I'm trying to write a persistence journal plugin (AsyncWriteJournal)
that saves entirely in JSON. I've created a serializer that serializes
to JSON and hooked that up along with my custom journal plugin, and
whilst the objects I'm saving are being persisted as JSON, the
PersistentRepr that wraps them isn't. Here's what I have:

def asyncWriteMessages(messages: immutable.Seq[AtomicWrite]):
Future[immutable.Seq[Try[Unit]]] = {
val ser = SerializationExtension(context.system)
for (m <- messages; p <- m.payload) {
val bytes = ser.findSerializerFor(p).toBinary(p)
// XXX Real thing would write to persistent storage here
println(new String(bytes))
}
Future.successful(Nil)
}

How do I get the *entire* payload that's stored in the journal saves
in JSON via my custom serializer?

Thanks,

--
Alan Burlison
--

[Konrad Malawski]

Pick a Journal which does this, simple.

Akka has absolutely no control over how a given Akka Persistence Journal Plugin decides to persist things.

Research the available journals and which allows to save things like this.

You did not mention which datastore you're using, which is very very relevant to this question.

-- 
Konrad `ktoso` Malawski
Akka @ Lightbend

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Patrik Nordwall]

You should not serialize PersistentRepr, unless you are using a key-value store that can only store one single value per key. Instead, grab the meta data fields from PersistentRepr and save them in separate columns. Grab the payload, serialize it, and save its bytes (the json) in a separate column.

It's also possible to not use a serializer (bytes) at all and instead use an EventAdapter to transform the domain events to json representation (e.g. I would imagine this would be efficient for MongoDB and crating BSON instead of using bytes).

/Patrik

[Alan Burlison]

On 16/01/17 20:45, Konrad Malawski wrote:

> Pick a Journal which does this, simple.
> Akka has absolutely no control over how a given Akka Persistence Journal
> Plugin decides to persist things.
>
> Research the available journals and which allows to save things like this.

I have, I've been unable to find one.

> You did not mention which datastore you're using, which is very very
> relevant to this question.

Flat file, JSON content.

--
Alan Burlison
--

[Alan Burlison]

On 16/01/17 21:22, Patrik Nordwall wrote:

> You should not serialize PersistentRepr, unless you are using a key-value
> store that can only store one single value per key.

It's a flat file so the concept of columns only exists insofar as I
define it - which is why I was intending to serialize to JSON. Isn't
PersistentRepr saved inside the datastore? If so, how is it normally
serialized, eg in the LevelDB datastore?

> Instead, grab the meta data fields from PersistentRepr and save them
> in separate columns.

Or I could just save the required fields as an outer JSON 'wrapper' over
the application event that's been serialized, perhaps?

> Grab the payload, serialize it, and save its bytes (the json) in a
> separate column.

That doesn't seem to be possible - findSerializerFor can't be passed the
payload as PersistentRepr.payload returns an Any, e.g.

// p is a PersistentRepr
val bytes = ser.findSerializerFor(p.payload).toBinary(p.payload)

does not compile as findSerializerFor expects an AnyRef.

> It's also possible to not use a serializer (bytes) at all and instead use
> an EventAdapter to transform the domain events to json representation (e.g.
> I would imagine this would be efficient for MongoDB and crating BSON
> instead of using bytes).

I did look at EventAdapter but it looked like you still needed to feed
that through a Journal implementation, which is what I'm struggling with
here.

The docs really don't give you much help on how to write a
AsyncWriteJournal - they tell you about the API but I haven't been able
to find any information about how you hook it up with the serializer,
for example.

--
Alan Burlison
--

[Patrik Nordwall]

On Mon, Jan 16, 2017 at 11:57 PM, Alan Burlison <alan.b...@gmail.com> wrote:

On 16/01/17 21:22, Patrik Nordwall wrote:

You should not serialize PersistentRepr, unless you are using a key-value
store that can only store one single value per key.

It's a flat file so the concept of columns only exists insofar as I define it - which is why I was intending to serialize to JSON. Isn't PersistentRepr saved inside the datastore? If so, how is it normally serialized, eg in the LevelDB datastore?

LevelDB is a key-value store with a single value per key, so there PersistentRepr is serialized as the value. Cassandra is not, and there the event (payload) is saved in a separate column.

 

Instead, grab the meta data fields from PersistentRepr and save them
in separate columns.

Or I could just save the required fields as an outer JSON 'wrapper' over the application event that's been serialized, perhaps?

Yes, that sounds good for Json.

 

Grab the payload, serialize it, and save its bytes (the json) in a
separate column.

That doesn't seem to be possible - findSerializerFor can't be passed the payload as PersistentRepr.payload returns an Any, e.g.

// p is a PersistentRepr
val bytes = ser.findSerializerFor(p.payload).toBinary(p.payload)

does not compile as findSerializerFor expects an AnyRef.

You can safely cast it to AnyRef, .asInstanceOf[AnyRef]

I think you can use ser.serialize(event).get

If you need manifest and such, here is an example: https://github.com/akka/akka-persistence-cassandra/blob/master/src/main/scala/akka/persistence/cassandra/journal/CassandraJournal.scala#L380

 

It's also possible to not use a serializer (bytes) at all and instead use
an EventAdapter to transform the domain events to json representation (e.g.
I would imagine this would be efficient for MongoDB and crating BSON
instead of using bytes).

I did look at EventAdapter but it looked like you still needed to feed that through a Journal implementation, which is what I'm struggling with here.

Yes, but then the EventAdapter would be responsible for creating the JSON and the journal would not user serialization. Not saying that it's needed or better in your case.

 

The docs really don't give you much help on how to write a AsyncWriteJournal - they tell you about the API but I haven't been able to find any information about how you hook it up with the serializer, for example.

--
Alan Burlison
--

--

Patrik Nordwall
Akka Tech Lead
Lightbend -  Reactive apps on the JVM
Twitter: @patriknw

[Alan Burlison]

On 17/01/2017 06:49, Patrik Nordwall wrote:

>> Or I could just save the required fields as an outer JSON 'wrapper' over
>> the application event that's been serialized, perhaps?
>
> Yes, that sounds good for Json.

Thanks, good to know I'm not completely off base :-)

>> That doesn't seem to be possible - findSerializerFor can't be passed the
>> payload as PersistentRepr.payload returns an Any, e.g.
>>
>> // p is a PersistentRepr
>> val bytes = ser.findSerializerFor(p.payload).toBinary(p.payload)
>>
>> does not compile as findSerializerFor expects an AnyRef.
>
> You can safely cast it to AnyRef, .asInstanceOf[AnyRef]
> I think you can use ser.serialize(event).get
> If you need manifest and such, here is an example:
> https://github.com/akka/akka-persistence-cassandra/blob/master/src/main/scala/akka/persistence/cassandra/journal/CassandraJournal.scala#L380

That's very useful, the manifest part in particular - just what I was
looking for as an example, thanks.

I did wonder why payload is an Any rather than an AnyRef - casting was
my first thought but it kinda smells...

>> I did look at EventAdapter but it looked like you still needed to feed
>> that through a Journal implementation, which is what I'm struggling with
>> here.
>
> Yes, but then the EventAdapter would be responsible for creating the JSON
> and the journal would not user serialization. Not saying that it's needed
> or better in your case.

I think I can just do it in the AsyncWriteJournal implementation in my
case, as what I want is simple.

One further question - I see mention in the docs and source of tags, I
don't think it's something I'll need, at least not initially. I assume
it's OK to just leave it all unimplemented?

Thanks again for the help and pointers,

--
Alan Burlison
--