How do people handle "delayed" events in eventsByTag in Cassandra?

557 views
Skip to first unread message

lutzh

unread,
Oct 10, 2017, 5:33:06 AM10/10/17
to Akka User List

eventsByTag in the CassandraReadJournal uses a materialized view to read the events. It can be configured to ignore the latest events (events younger than "eventual-consistency-delay") to give the materialized view some time to converge.

This seems to work well enough for most cases, if the "eventual-consistency-delay" is generous, at least I haven't heard of any problems. Generally the events will be read in the order they have been written, no matter on which node etc., regardless if one "arrived" a bit late in the materialized view (i.e. a younger event could appear in the materialized view before the older one, but if the older one also appears within "eventual-consistency-delay", that's no problem).

But - there's no guarantee for that. The documentation makes clear that this is a "best effort" and at least in theory, the actual eventual consistency delay can still be bigger than the one that is anticipated by configuration. If I understand correctly, the following scenario could still happen:

  1. Read eventsByTag with start offset "7" (obviously 7 is not a valid time-based uuid.. I simplify for the sake of the example) 
  2. get some events, the time-based uuid of the latest is "10"
  3. after the query, a new event with time-based uuid "9" appears

Am I correct in assuming this could occur?

Given a non-terminated stream that is still running after 2, would the event in step 3 appear in the stream?

Assuming there's an issue after step 2, and the stream terminates. A restarted stream would start from offset 10, so the event that appeared in step 3 would be ignored, correct?

People may say the "delayed-event-timeout" can make sure a delayed event is not ignored. But that is "local", in the sense of per persistenceId, correct? So if the event from step 3 is for a new / unseen persistenceId, it might still be ignored?

I hope I'm not just confusing everyone... probably the solution is obvious and I'm just suffering from tunnel vision.

How do people guarantee they never miss a (delayed) event when populating the read side? Do you just rely on eventual-consistency-delay and it never fails you?

If you do miss a delayed event, is there any way to detect it?


For reference, the relevant comment from akka/persistence/cassandra/query/javadsl/CassandraReadJournal.scala:

   * The same stream elements (in same order) are returned for multiple
   * executions of the query on a best effort basis. The query is using a Cassandra Materialized
   * View for the query and that is eventually consistent, so different queries may see different
   * events for the latest events, but eventually the result will be ordered by timestamp
   * (Cassandra timeuuid column). To compensate for the the eventual consistency the query is
   * delayed to not read the latest events, see `cassandra-query-journal.eventual-consistency-delay`
   * in reference.conf. However, this is only best effort and in case of network partitions
   * or other things that may delay the updates of the Materialized View the events may be
   * delivered in different order (not strictly by their timestamp).


Thanks,
Lutz


Patrik Nordwall

unread,
Oct 10, 2017, 7:46:11 AM10/10/17
to akka...@googlegroups.com
Hi Lutz,

On Tue, Oct 10, 2017 at 11:33 AM, lutzh <lutz.h...@gmail.com> wrote:

eventsByTag in the CassandraReadJournal uses a materialized view to read the events. It can be configured to ignore the latest events (events younger than "eventual-consistency-delay") to give the materialized view some time to converge.

This seems to work well enough for most cases, if the "eventual-consistency-delay" is generous, at least I haven't heard of any problems. Generally the events will be read in the order they have been written, no matter on which node etc., regardless if one "arrived" a bit late in the materialized view (i.e. a younger event could appear in the materialized view before the older one, but if the older one also appears within "eventual-consistency-delay", that's no problem).

But - there's no guarantee for that. The documentation makes clear that this is a "best effort" and at least in theory, the actual eventual consistency delay can still be bigger than the one that is anticipated by configuration. If I understand correctly, the following scenario could still happen:

  1. Read eventsByTag with start offset "7" (obviously 7 is not a valid time-based uuid.. I simplify for the sake of the example) 
  2. get some events, the time-based uuid of the latest is "10"
  3. after the query, a new event with time-based uuid "9" appears

Am I correct in assuming this could occur?

Given a non-terminated stream that is still running after 2, would the event in step 3 appear in the stream?

Assuming there's an issue after step 2, and the stream terminates. A restarted stream would start from offset 10, so the event that appeared in step 3 would be ignored, correct?

Don't use it without setting delayed-event-timeout if you care about retrieving all events. It was probably a mistake to support anything else and not having it enabled by default. The reason is that it requires that all events from a persistentId must be tagged, not only some of them. Otherwise it would be gap in the sequence numbers, which are use to detect missing events.
 

People may say the "delayed-event-timeout" can make sure a delayed event is not ignored. But that is "local", in the sense of per persistenceId, correct? So if the event from step 3 is for a new / unseen persistenceId, it might still be ignored?

True that there is a risk for, but the query is looking back for such missed/new persistenceId so it shouldn't be a big problem. You could add another layer of verification for such by periodically query for all persistenceId and see if you are missing any and then restart the eventsByTag query from an earlier offset.
 

I hope I'm not just confusing everyone... probably the solution is obvious and I'm just suffering from tunnel vision.

It's a valid question.
 

How do people guarantee they never miss a (delayed) event when populating the read side? Do you just rely on eventual-consistency-delay and it never fails you?

If you do miss a delayed event, is there any way to detect it?


For reference, the relevant comment from akka/persistence/cassandra/query/javadsl/CassandraReadJournal.scala:

   * The same stream elements (in same order) are returned for multiple
   * executions of the query on a best effort basis. The query is using a Cassandra Materialized
   * View for the query and that is eventually consistent, so different queries may see different
   * events for the latest events, but eventually the result will be ordered by timestamp
   * (Cassandra timeuuid column). To compensate for the the eventual consistency the query is
   * delayed to not read the latest events, see `cassandra-query-journal.eventual-consistency-delay`
   * in reference.conf. However, this is only best effort and in case of network partitions
   * or other things that may delay the updates of the Materialized View the events may be
   * delivered in different order (not strictly by their timestamp).


Thanks,
Lutz


--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.



--

Patrik Nordwall
Akka Tech Lead
Lightbend -  Reactive apps on the JVM
Twitter: @patriknw

lutzh

unread,
Oct 11, 2017, 4:57:55 AM10/11/17
to Akka User List
Hi Patrik,

Thanks, that's very helpful!

So when delayed-event-timeout is used, would you even care to set the eventual-consistency-delay to > 0? To me it still seems to make sense to give the materialized view some time to catch up, if only for efficiency.. or doesn't it?

About missing events from new persistenceIDs:

> True that there is a risk for, but the query is looking back for such missed/new persistenceId so it shouldn't be a big problem. 

So if an event from an unseen persistenceID is encoutered, the code will check if there are any previous events for this persistenceID?


Regards,
Lutz



Hi Lutz,

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Patrik Nordwall

unread,
Oct 11, 2017, 5:08:55 AM10/11/17
to akka...@googlegroups.com
On Wed, Oct 11, 2017 at 10:57 AM, lutzh <lutz.h...@gmail.com> wrote:
Hi Patrik,

Thanks, that's very helpful!

So when delayed-event-timeout is used, would you even care to set the eventual-consistency-delay to > 0? To me it still seems to make sense to give the materialized view some time to catch up, if only for efficiency.. or doesn't it?

yes, I'd set it to something low, such as 200 ms
 

About missing events from new persistenceIDs:

> True that there is a risk for, but the query is looking back for such missed/new persistenceId so it shouldn't be a big problem. 

So if an event from an unseen persistenceID is encoutered, the code will check if there are any previous events for this persistenceID?


yes
 
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages