How do you persist state of Sagas?

[Rinat Abdullin]

Hi all,

Just a quick question on saga persistence - how do you persist saga
state and dispatch messages while avoiding transactions and 2PC?

Long story: I'm trying to reason out the logic behind sagas, in order
to understand everything better (and map concepts back to the reactive
programming)

Basically a saga is an entity, that is used to coordinate some long-
running process. It can subscribe to events (UserAccountCreated), keep
track of time (i.e.: user should activate his account within 24 hours)
and send commands (CancelUserRegistration).

Additionally, since saga is an entity and could be addressed in the
scalable world, we can send command directly to the saga
(StopRegistrationProcess). Sagas can be modeled and perceived as
finite state machines. So far - so good and rather straightforward.

However just a quick question: how do you persist saga state and send
messages out of it?

Logically, in order to avoid 2PC and transactions you would need to
join state transition and publication in one atomic operation (just
like with the aggregate roots and event sourcing) and reuse message
dispatching mechanism that catches up with the history (append-only
persistence scales much better anyway)

This feels like more sensible and simple operation, than introducing
relational DBs or any kind of transactions into the system. However,
as I recall, I've never heard of using event sourcing for the saga
state persistence. Is there a reason for this? How do you implement
your sagas and persist their state?

All feedback would be appreciated!

Best regards,
Rinat Abdullin

[João Bragança]

I will be running into this brick wall shortly - first it was YAGNI
but now it is YIGNI. I am thinking will just persist saga state as
json and have the saga explicitly load the state. If this becomes too
unmaintainable I can always throw an interceptor around it and have
that load the saga state based on convention.

[Jonathan Oliver]

Rinat,

There are a few options two avoid 2PC. One of the easiest ways is to
simply have the saga entity store a list of all command IDs
internally. Rarely will you have sagas that exist beyond even several
dozen commands/events. That being the case, you can effectively treat
the saga as a kind of aggregate root using event sourcing. (More on
this in a minute.)

By storing the command IDs internal to the saga, you can avoid 2PC by
having two completely separate transactions--an outer as well as an
inner transaction. The inner transaction is related to committing the
saga "aggregate" to the event store. The outer transaction is related
to removing the message from the message queue. If the message queue
doesn't support TransactionScope, it's not a big deal--it will attempt
to deliver the message at least once and you can easily detect it as a
duplicate and drop it because it's already been handled.

Let the event store do the publishing for you asynchronously.

I've outlined a few of these concepts in some blog posts I wrote a few
months back (one of which you commented on):
http://jonathan-oliver.blogspot.com/2010/04/extending-nservicebus-avoiding-two.html
http://jonathan-oliver.blogspot.com/2010/04/idempotency-patterns.html
http://jonathan-oliver.blogspot.com/2010/04/message-idempotency-patterns-and-disk.html

The other part of your question is how to leverage event sourcing to
take care of sagas. It's not unlike your typically aggregate root.
Some kind of stimulus comes in (either a command or event), you
transition the state (this being the part that's distinct from DDD
aggregates), which results in a message being "raised". Then, you
commit the new state to the event store and let it perform the message
dispatch asynchronously.

Jonathan

[Rinat Abdullin]

Ah, thanks a lot guys.

So basically for the saga persistence we can have either event
sourcing (command is saved along with the events in the transaction)
or simple state storage (command is saved along with the latest state
and possible outgoing events). Dispatcher could dispatch in async
later in both cases.

Once we have command info persisted atomically with the resulting
changes, we can have all the idempotence we need (still staying away
from the 2PC). Consistency is 100% even if process dies between the
commit and ACK.

So technically sagas are just like the aggregates (they are entities),
and the primary difference is in the intent (similar to the difference
between commands and events) and life span expectations.

This way everything that happens in saga between the handler and
message dispatch is rather straightforward, reliable and simple (and
similar to the aggregate behavior).

Thanks again for helping to think though the logic of this part of CQRS!

Best regards,
Rinat

[Jonathan Oliver]

The only thing that I would add is that sagas should be more like a
state machine which is about *process*, whereas our aggregates are
more about *logic* (if statements and flow control).

> >http://jonathan-oliver.blogspot.com/2010/04/extending-nservicebus-avo...
> >http://jonathan-oliver.blogspot.com/2010/04/idempotency-patterns.html
> >http://jonathan-oliver.blogspot.com/2010/04/message-idempotency-patte...

[Rinat Abdullin]

I agree. Sagas and aggregates have different intent plus resulting
differences in behavior, life cycle and persistence. Ignoring this in
the project might kick in the natural selection process for it.

However, implementation logic of command handlers outside of these
"inner" specifics seem to be similar for both cases (i.e.: questions
of reliability, 2PC, transactions and message dispatch). Don't you
think?

Best regards,
Rinat

[Pedro Henriques dos Santos Teixeira]

hi Rinat,

I'm using Esper for my "sagas" and currently I can rebuild it's state
by replaying events at startup.

Esper allows one send timetick events to control the flow of time when
replaying in isolation, and it's pretty awesome!

(regards)

[Rinat Abdullin]

Hi Pedro,

Thanks a lot for the lead!

Esper definitely looks interesting. Although I'm not sure that's a
perfect fit for sagas in my projects, but the ability to run
continuous event queries is really something I would love to try
(currently we cache all events locally and run manual LINQ queries
printing results to Excel via OpenXML).

Best regards,
Rinat

[Bhoomi Kakaiya]

Hi,

Can anybody provide with a pseudo code for saga?

That would make things more clear.

Best regards,

Bhoomi.

[Rinat Abdullin]

Bhoomi,

I've published an article that goes into some deeper on Sagas (as per
discussions in this thread and outside of it).

Although there is still no source code, but it might help to
understand everything.

http://abdullin.com/journal/2010/9/26/theory-of-cqrs-command-handlers-sagas-ars-and-event-subscrip.html

Just a caveat: I'm sorry for going into deep details about the
partitioning logic (this was needed by the specifics). In practice
implementations will probably skip this part completely in 95% of
cases (and go lightly on a few other explicit constraints as well).

Best regards,
Rinat

[Bhoomi Kakaiya]

Thanks a lot Rinat.