How to listen to events on CassandraJournalProvider ?

[kant kodali]

Hi All,

I am trying to understand event sourcing in detail and I am new to Akka. One of book I was reading says a Persistence Actor will write events to a Journal so in my case the Journal is Cassandra and one can have Persistence Query Actor listen to the events from the Journal and this is where I don't understand how it works. More specifically, How does Persistence Query Actor know they are new events that it needs to Query for? I assume Persistence Actor and Persistence Query Actor are two separate things (so How does Persistence Query Actor gets notified that there are new events written by Persistence Actor)

I would like to get live stream from Cassandra but I as far as I know I don't think there is any push based mechanism in Cassandra where Cassandra itself will push new events to some message bus or application. please correct me if I am wrong.

Thanks,

kant

[Tal Pressman]

Hi,

I think the name for Persistence Query is a little misleading - when I first heard it I imagined something like an SQL query that can run over the data source, but that's not what it is. What it actually does is set up an Akka stream source that handles the events from the data store. The stream can be finite (like currentEventsByPersistenceId will stop once it has caught up with all the events) or infinite (like eventsByPersistenceId). In the second case, whenever there a new message is written to the data store (and the stream isn't back-pressured / busy) the source will emit the new message for the stream to handle.

Now, as you've mentioned, there is no "push" mechanism for Cassandra, but that doesn't mean that everything breaks down. It simply means that when you use the CassandraReadJournal there is a thread that polls the DB and when there are new events there passes them down the stream.

HTH,

Tal

[Patrik Nordwall]

Tal describes it very well. In the future we'll hopefully be able to get notifications from Cassandra when something has changed instead of using the primitive poll approach. There is already Change Data Capture (CDC) in Cassandra but it's an extremely low level api so far. Improvement suggestions based on CDC or something else very much welcome.

/Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Michal Borowiecki]

Also, there's a setting called "cassandra-journal.pubsub-minimum-interval", which if set will cause the journal to notify the persistence query side of new writes, so it can only poll when needed instead of doing so periodically.

Cheers,
Michal

--

	Michal Borowiecki
	Senior Software Engineer L4
	T: +44 208 742 1600 +44 203 249 8448   E: michal.b...@openbet.com W: www.openbet.com	OpenBet Ltd Chiswick Park Building 9 566 Chiswick High Rd London W4 5XT UK

This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postm...@openbet.com and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by OpenBet for employment and security purposes. To protect the environment please do not print this e-mail unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company registered in England and Wales. Registered no. 3134634. VAT no. GB927523612

[kant kodali]

Hi!

@Tal  say The actor that writes to a Cassandra data store is different from an Actor/Streaming source that polls from Cassandra. And if I understand this correctly you are saying the Actor that writes to Cassandra data store notifies to the Actor/streaming source that the write has been successful so it can poll from Cassandra. correct? 2) Can the streaming source be Spark? 3) Does Akka streaming source is a single process? I know that one can form clusters with Akka but I just want to know it is done typically when it comes to distribution and scalability? because if a single process has to catch up with a large stream I assume that may not scale well.

@Patrick Can I see a ticket or something I can follow?

Thanks much!

[Patrik Nordwall]

On Tue, Mar 14, 2017 at 2:04 AM, kant kodali <kant...@gmail.com> wrote:

Hi!

@Tal  say The actor that writes to a Cassandra data store is different from an Actor/Streaming source that polls from Cassandra. And if I understand this correctly you are saying the Actor that writes to Cassandra data store notifies to the Actor/streaming source that the write has been successful so it can poll from Cassandra. correct? 2) Can the streaming source be Spark? 3) Does Akka streaming source is a single process? I know that one can form clusters with Akka but I just want to know it is done typically when it comes to distribution and scalability? because if a single process has to catch up with a large stream I assume that may not scale well.

@Patrick Can I see a ticket or something I can follow?

ok, I created issue https://github.com/akka/akka-persistence-cassandra/issues/161

 

Thanks much!

On Sunday, March 12, 2017 at 7:09:02 AM UTC-7, kant kodali wrote:

Hi All,

I am trying to understand event sourcing in detail and I am new to Akka. One of book I was reading says a Persistence Actor will write events to a Journal so in my case the Journal is Cassandra and one can have Persistence Query Actor listen to the events from the Journal and this is where I don't understand how it works. More specifically, How does Persistence Query Actor know they are new events that it needs to Query for? I assume Persistence Actor and Persistence Query Actor are two separate things (so How does Persistence Query Actor gets notified that there are new events written by Persistence Actor)

I would like to get live stream from Cassandra but I as far as I know I don't think there is any push based mechanism in Cassandra where Cassandra itself will push new events to some message bus or application. please correct me if I am wrong.

Thanks,

kant

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Akka Tech Lead
Lightbend -  Reactive apps on the JVM
Twitter: @patriknw

[kant kodali]

Hi Michael,

Any idea how that that is implemented? I understand this setting "cassandra-journal.pubsub-minimum-interval" is for Akka-Cassandra-Persistence but I wonder what exactly it leverages underneath from Cassandra or Cassandra driver?

Thanks,

kant 

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/PM3UwnnsK5w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+unsubscribe@googlegroups.com.

[Michal Borowiecki]

Hi kant,

You can browse the sources on github. The implementation doesn't seem to leverage anything from Cassandra underneath, the journal plugin does the work itself. When asked to persist events, it emits (throttled) notifications via akka pubsub which the CassandraReadJournal subscribes to.

Cheers,

Michal

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.