--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
-- Martin Krasser blog: http://krasserm.blogspot.com code: http://github.com/krasserm twitter: http://twitter.com/mrt1nz
Hi Ashley,
thanks for bringing up these questions. Here are some general comments:
as you already mentioned (in different words) akka-persistence is currently optimized around write models rather than read models (= Q in CQRS) i.e it is optimized for fast, scalable persistence and recovery of stateful actors (= PersistentActor).
For full CQRS support, the discussions so far (in several other threads) make the assumption that both write and read models are backed by the same backend store (assuming read models are maintained by PersistentView actor, receiving a stream of events from synthetic or physical "topics").
On Tue, Aug 19, 2014 at 8:51 AM, Martin Krasser <kras...@googlemail.com> wrote:
Hi Ashley,
thanks for bringing up these questions. Here are some general comments:
as you already mentioned (in different words) akka-persistence is currently optimized around write models rather than read models (= Q in CQRS) i.e it is optimized for fast, scalable persistence and recovery of stateful actors (= PersistentActor).
For full CQRS support, the discussions so far (in several other threads) make the assumption that both write and read models are backed by the same backend store (assuming read models are maintained by PersistentView actor, receiving a stream of events from synthetic or physical "topics").
That is not my view of it, at all. PersistentView is a way to replicate the events to the read side, which typically will store a denormalized representation optimized for queries. That query store is typically not the same as the event store, because the requirements are very different.
I agree, but recent discussions were about how to join events from several topics/streams that a PersistentView receives (e.g. all events of an aggregate type or based on a user-defined join/query)...
For full CQRS support, the discussions so far (in several other threads) make the assumption that both write and read models are backed by the same backend store (assuming read models are maintained by PersistentView actor, receiving a stream of events from synthetic or physical "topics"). This is a severe limitation, IMO. As Greg already mentioned elsewhere, some read models may be best backed by a graph database, for example. Although a graph database may be good for backing certain read models, it may have limitations for fast logging of events (something where Kafka or Cassandra are very good at). Consequently, it definitely makes sense to have different backend stores for write and read models.
If akka-persistence should have support for CQRS in the future, its design/API should be extended to allow different backend stores for write and read models (of course, a provider may choose to use the same backend store to serve both which may be a reasonable default). This way PersistentActors log events to one backend store and PersistentViews (or whatever consumer) generate read models from the other backend store. Data transfer between these backend stores can be implementation-specific for optimization purposes.
For example
- Cassandra (for logging events) => Spark (to batch-process logged events) => Graph database XY (to store events processed with Spark), or
- Kafka (for logging events) => Spark Streaming (to stream-process logged events) => Database XY (to store events processed with Spark Streaming)
- ...
These are just two examples how read model backend stores can be populated in a highly scalable way (both in batch and streaming mode). Assuming akka-persistence provides an additional plugin API for storage backends on the read model side (XY in the examples above) a wide range of CQRS applications could be covered with whatever scalability and/or ordering requirements needed by the respective applications. In case you want to read more about it, take a look at akka-analytics (it is very much work in progress as I'm waiting for Spark to upgrade to Akka 2.3 and Kafka to Scala 2.11)
WDYT?
journal -> akka actor(s) -> read model datastore
when I can do this much more efficiently via
journal -> spark -> read model datastore
directly, for example
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
On Tuesday, 19 August 2014 16:53:46 UTC+8, Martin Krasser wrote:
journal -> akka actor(s) -> read model datastore
when I can do this much more efficiently via
journal -> spark -> read model datastore
directly, for exampleI am confused, are you suggesting that spark is talking to the journal data store directly, without any involvement of Akka / Akka Persistence?
If so, it sounds like a great solution but why would that require an extension to the Akka Persistence design/API?
19 aug 2014 kl. 11:28 skrev Ashley Aitken <amai...@gmail.com>:
On Tuesday, 19 August 2014 16:53:46 UTC+8, Martin Krasser wrote:
journal -> akka actor(s) -> read model datastore
when I can do this much more efficiently via
journal -> spark -> read model datastore
directly, for exampleI am confused, are you suggesting that spark is talking to the journal data store directly, without any involvement of Akka / Akka Persistence? If so, it sounds like a great solution but why would that require an extension to the Akka Persistence design/API?
Well, another comment is that spark uses Akka actors in its implementation, so I don’t see why it would magically be “much more efficient”. I think we are mixing up two concerns here, will reply later when I can type properly again.
This is a misunderstanding. As mentioned in my previous message, scaling reads through a single journal actor doesn't work, it's not about that I see a general performance issue with Akka actors.
On 19.08.14 13:00, Roland Kuhn wrote:
19 aug 2014 kl. 11:28 skrev Ashley Aitken <amai...@gmail.com>:
On Tuesday, 19 August 2014 16:53:46 UTC+8, Martin Krasser wrote:
journal -> akka actor(s) -> read model datastore
when I can do this much more efficiently via
journal -> spark -> read model datastore
directly, for exampleI am confused, are you suggesting that spark is talking to the journal data store directly, without any involvement of Akka / Akka Persistence? If so, it sounds like a great solution but why would that require an extension to the Akka Persistence design/API?
Well, another comment is that spark uses Akka actors in its implementation, so I don’t see why it would magically be “much more efficient”. I think we are mixing up two concerns here, will reply later when I can type properly again.
On Tue, Aug 19, 2014 at 1:35 PM, Martin Krasser <kras...@googlemail.com> wrote:
This is a misunderstanding. As mentioned in my previous message, scaling reads through a single journal actor doesn't work, it's not about that I see a general performance issue with Akka actors.
On 19.08.14 13:00, Roland Kuhn wrote:
19 aug 2014 kl. 11:28 skrev Ashley Aitken <amai...@gmail.com>:
On Tuesday, 19 August 2014 16:53:46 UTC+8, Martin Krasser wrote:
journal -> akka actor(s) -> read model datastore
when I can do this much more efficiently via
journal -> spark -> read model datastore
directly, for exampleI am confused, are you suggesting that spark is talking to the journal data store directly, without any involvement of Akka / Akka Persistence? If so, it sounds like a great solution but why would that require an extension to the Akka Persistence design/API?
Well, another comment is that spark uses Akka actors in its implementation, so I don’t see why it would magically be “much more efficient”. I think we are mixing up two concerns here, will reply later when I can type properly again.
I think the integration "akka persistence -> kafka -> spark -> whatever" looks great, but not everybody has that infrastructure, and therefore we provide PersistentView as a simple way to replicate events to the read side, and then a de-normalized representation can be stored in whatever makes sense for the queries.
Martin, what do you suggest? Removing PersistentView altogether?
Of course, that should be possible too, and as I already said, backend store providers can choose to use the very same backend for both plugins. There is absolutely no need that applications must use Spark as part of their infrastructure infrastructure. But if it is needed in large-scale applications, a seconds plugin API for on the read side would make things much more flexible. For users, who just want to have a single backend store, they just have to configure one additional line (plugin) in their application conf.
On 19.08.14 13:40, Patrik Nordwall wrote:
On Tue, Aug 19, 2014 at 1:35 PM, Martin Krasser <kras...@googlemail.com> wrote:
This is a misunderstanding. As mentioned in my previous message, scaling reads through a single journal actor doesn't work, it's not about that I see a general performance issue with Akka actors.
On 19.08.14 13:00, Roland Kuhn wrote:
19 aug 2014 kl. 11:28 skrev Ashley Aitken <amai...@gmail.com>:
On Tuesday, 19 August 2014 16:53:46 UTC+8, Martin Krasser wrote:
journal -> akka actor(s) -> read model datastore
when I can do this much more efficiently via
journal -> spark -> read model datastore
directly, for exampleI am confused, are you suggesting that spark is talking to the journal data store directly, without any involvement of Akka / Akka Persistence? If so, it sounds like a great solution but why would that require an extension to the Akka Persistence design/API?
Well, another comment is that spark uses Akka actors in its implementation, so I don’t see why it would magically be “much more efficient”. I think we are mixing up two concerns here, will reply later when I can type properly again.
I think the integration "akka persistence -> kafka -> spark -> whatever" looks great, but not everybody has that infrastructure, and therefore we provide PersistentView as a simple way to replicate events to the read side, and then a de-normalized representation can be stored in whatever makes sense for the queries.No, not at all, with an additional plugin, PersistentViews should have the option to read transformed/joined/... streams from a backend store that is optimized for that.
Martin, what do you suggest? Removing PersistentView altogether?
On Tue, Aug 19, 2014 at 1:46 PM, Martin Krasser <kras...@googlemail.com> wrote:Of course, that should be possible too, and as I already said, backend store providers can choose to use the very same backend for both plugins. There is absolutely no need that applications must use Spark as part of their infrastructure infrastructure. But if it is needed in large-scale applications, a seconds plugin API for on the read side would make things much more flexible. For users, who just want to have a single backend store, they just have to configure one additional line (plugin) in their application conf.
On 19.08.14 13:40, Patrik Nordwall wrote:
On Tue, Aug 19, 2014 at 1:35 PM, Martin Krasser <kras...@googlemail.com> wrote:
This is a misunderstanding. As mentioned in my previous message, scaling reads through a single journal actor doesn't work, it's not about that I see a general performance issue with Akka actors.
On 19.08.14 13:00, Roland Kuhn wrote:
19 aug 2014 kl. 11:28 skrev Ashley Aitken <amai...@gmail.com>:
On Tuesday, 19 August 2014 16:53:46 UTC+8, Martin Krasser wrote:
journal -> akka actor(s) -> read model datastore
when I can do this much more efficiently via
journal -> spark -> read model datastore
directly, for exampleI am confused, are you suggesting that spark is talking to the journal data store directly, without any involvement of Akka / Akka Persistence? If so, it sounds like a great solution but why would that require an extension to the Akka Persistence design/API?
Well, another comment is that spark uses Akka actors in its implementation, so I don’t see why it would magically be “much more efficient”. I think we are mixing up two concerns here, will reply later when I can type properly again.
I think the integration "akka persistence -> kafka -> spark -> whatever" looks great, but not everybody has that infrastructure, and therefore we provide PersistentView as a simple way to replicate events to the read side, and then a de-normalized representation can be stored in whatever makes sense for the queries.No, not at all, with an additional plugin, PersistentViews should have the option to read transformed/joined/... streams from a backend store that is optimized for that.
Martin, what do you suggest? Removing PersistentView altogether?
ah, now I understand what you mean. That makes sense.
19 aug 2014 kl. 13:49 skrev Patrik Nordwall <patrik....@gmail.com>:
On Tue, Aug 19, 2014 at 1:46 PM, Martin Krasser <kras...@googlemail.com> wrote:
Of course, that should be possible too, and as I already said, backend store providers can choose to use the very same backend for both plugins. There is absolutely no need that applications must use Spark as part of their infrastructure infrastructure. But if it is needed in large-scale applications, a seconds plugin API for on the read side would make things much more flexible. For users, who just want to have a single backend store, they just have to configure one additional line (plugin) in their application conf.
On 19.08.14 13:40, Patrik Nordwall wrote:
On Tue, Aug 19, 2014 at 1:35 PM, Martin Krasser <kras...@googlemail.com> wrote:
This is a misunderstanding. As mentioned in my previous message, scaling reads through a single journal actor doesn't work, it's not about that I see a general performance issue with Akka actors.
On 19.08.14 13:00, Roland Kuhn wrote:
19 aug 2014 kl. 11:28 skrev Ashley Aitken <amai...@gmail.com>:
On Tuesday, 19 August 2014 16:53:46 UTC+8, Martin Krasser wrote:
journal -> akka actor(s) -> read model datastore
when I can do this much more efficiently via
journal -> spark -> read model datastore
directly, for exampleI am confused, are you suggesting that spark is talking to the journal data store directly, without any involvement of Akka / Akka Persistence? If so, it sounds like a great solution but why would that require an extension to the Akka Persistence design/API?
Well, another comment is that spark uses Akka actors in its implementation, so I don’t see why it would magically be “much more efficient”. I think we are mixing up two concerns here, will reply later when I can type properly again.
I think the integration "akka persistence -> kafka -> spark -> whatever" looks great, but not everybody has that infrastructure, and therefore we provide PersistentView as a simple way to replicate events to the read side, and then a de-normalized representation can be stored in whatever makes sense for the queries.
No, not at all, with an additional plugin, PersistentViews should have the option to read transformed/joined/... streams from a backend store that is optimized for that.
Martin, what do you suggest? Removing PersistentView altogether?
ah, now I understand what you mean. That makes sense.
I’m not completely there yet: in which way does this require changes to Akka Persistence? The only thing we need is to support multiple Journals in the same ActorSystem, and a way for PersistentView and PersistentActor to select between them, is this what you mean?
Or do you mean that the read-side would be a new kind of plugin?
OTOH this would not solve the read-side concerns raised by Greg: building a View on top of an incoming event stream is precisely not what he wants, unless I got that wrong. The idea behind CQRS/ES is that the events from the write-side drive updates of the read-side which is then queried (i.e. actively asked instead of passively updating) in whatever way is appropriate (e.g. graph searches).
Hi guys, really interesting thread. However, it follows from this discussion that Akka Persistence is not currently 100% ready for a full CRQS/ES implementation. A little bit frustrating but, to be honest, it's true that it's still an experimental feature. As users, we're assuming this.
Anyway, and thinking about how to solve the Query part, what do you think about using some distributed in-memory data grid solution such as Hazelcast or GridGain?.
Regards,Juanjo
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
"and therefore we provide PersistentView as a simple way to replicate events to the read side, and then a de-normalized representation can be stored..."
If I understand this right, this means:
PersistenActor persists event
PersistentView queries EventStore e.g.: every second and forwards new events to the read side, e.g.: EventListener which then updates the ReadModel
What is the advantage of using the PersistenView here, instead of just emitting the event to the read side from the PersistentActor directly?
thanks,
michael
19 aug 2014 kl. 15:39 skrev Juan José Vázquez Delgado <jvaz...@tecsisa.com>:Hi guys, really interesting thread. However, it follows from this discussion that Akka Persistence is not currently 100% ready for a full CRQS/ES implementation. A little bit frustrating but, to be honest, it's true that it's still an experimental feature. As users, we're assuming this.Akka Persistence is about persistent actors, using Event Sourcing to achieve this goal. This makes it a perfect fit for the C in CQRS. The Q on the other hand does not actually need to have anything to do with Akka or actors at all, per se. If we can provide nice things then we will, or course :-)
Then the PersistentView is not used as a "middle-man" to replicate events to the read side, but it is the read side(meaning if a client sends a query a PersistentView creates the response)?
That's how I understood PersistentViews until now - but maybe that was wrong, so I' am asking...
thank's for your answer
On Tuesday, 19 August 2014 16:11:33 UTC+2, Patrik Nordwall wrote:On Tue, Aug 19, 2014 at 2:33 PM, delasoul <michael...@gmx.at> wrote:
"and therefore we provide PersistentView as a simple way to replicate events to the read side, and then a de-normalized representation can be stored..."
If I understand this right, this means:
PersistenActor persists event
PersistentView queries EventStore e.g.: every second and forwards new events to the read side, e.g.: EventListener which then updates the ReadModel
What is the advantage of using the PersistenView here, instead of just emitting the event to the read side from the PersistentActor directly?The PersistentView (read side) can process the events in its own pace, it is decoupled from the write side. It can be down without affecting the write side, and it can be started later and catch up.Also, you can have multiple PersistentView instances consuming the same stream of events, maybe doing different things with them./Patrik
thanks,
michael
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
If so, it sounds like a great solution but why would that require an extension to the Akka Persistence design/API?
Because transformed/joined/... event streams in backend store on the read side must be consumable by PersistentViews (for creating read models). I still see this backend store to maintain changes (= transformed/joined/... events) instead of current state.
On Tuesday, 19 August 2014 19:33:55 UTC+8, Martin Krasser wrote:If so, it sounds like a great solution but why would that require an extension to the Akka Persistence design/API?
Because transformed/joined/... event streams in backend store on the read side must be consumable by PersistentViews (for creating read models). I still see this backend store to maintain changes (= transformed/joined/... events) instead of current state.
I am sorry I still don't see this.
This suggests to me that spark is talking directly to the read model datastore (e.g. graph database, MongoDB, SQL database).
So are you suggesting:
1. journal -> spark -> Akka actors (like PersistentView) -> read model data store
or
2. journal -> spark -> read model data store (like graph database, MongoDb, SQL database) -> Akka actors <- Queries
I see PersistentView(for generalised topics) as the glue between the Akka journal (write store) and read stores (1.).
Thanks for your patience.
Cheers,Ashley.
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
As I am no Spark expert - will it be used only as kind of messaging(streaming) middleware to sync write and read store or also to somehow change/merge/
filter the events it gets/pulls from the write store
Team,Consider I have domain modelBank
Customer'sAccount'sBank object will have many customers.Every customer will have multiple accounts.Consider I implement the above model using akka persistence. For the sake of discussion consider I make each as different Aggregate Roots using PersistentActor's.I want to implement a query : Give my all customers whose balance is less then 100$.Do you mean,1. Create one PersistentView XYZ.2. This XYZ will listen from all 3 AR's.3. PersistentView XYZ will replicate these events as state to some DB consider cassandra.4. Client will query directly cassandra to find "all customers whose balance is less then 100$".Is my understanding correct?
On 21 August 2014 at 20:28:35, Prakhyat Mallikarjun (prakh...@gmail.com) wrote:
Patrik,Thanks.You have given good directions to me.Most of the apps will do queries or searches specific to domain or business. If business specific complex queries/searches are better solved in the backend data store, why akka persistence "PersistenceView" is required and should be still part of akka persistence?I am not doubting "PersistenceView" but want to understand why one would choose "PersistenceView" in design over simpler data store?
You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/4kbYcwWS2OI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.