Handling of SQL deadlocks for parallel messages returning to a saga in a fork and merge process

[Simon Davis]

I'm trying to handle multiple messages of the same type returning to my saga, which then need to update the saga state to track the number of the messages which have completed.

In my specific case I could trigger up to nine messages to be processed in parallel which need to be then need to be merged and update the saga state, how ever I am currently getting SQL connection deadlocks when testing with just two messages.

I'm using the EntityFrameworkSagaRepository to manage the Entity Framework DbContext for the saga.

I'm going to experiment with adding a retry on the for the connection, but I don't see this as a very scalable solution when a larger number of messages return at the same time.

Had anyone had to solve a similar issue and have an idea how to implement this use case?

[Ben Poelstra]

I had this happen to me too. Here is the best solution that I have come up with. 

Create a table where each row in the table is the forked messages and stores the result of the processing. I created this table outside the SagaRepository, but it stores the saga correlation and the individual message correlation.

Insert the rows before the fork.

Update the row when the fork has completed processing in the consumer.

Publish a message for merging when each fork has finished

Validate that all the forked messages are finished and then publish a transition message.

I had issues where it would publish more than one transition message because they were happening in parallel, so I had to mark the whole thing as complete on the saga instance and validate that it wasn't set before publishing the transition message.

Hope that helps.

[Alexey Zimarev]

I have similar issue with using RavenDb saga repository. It happens with just two messages and CombinedEvent, the waiting bit was overriding in a race condition. When I turned on RavenDb optimistic concurrency, it started to fail on the wrong instance version. I do have the retry policy configured but this seems to have no effect on saga. At least for me it does not retry, just crashes the whole thing. I do not have any solution for this though, we are now using in memory repository since we are not in production yet but this is not acceptable.

My view on this that messages that are related to one saga instance should not be processed in parallel but rather put in a sequence or queue in memory and not ACKed until saga actually processed them. When saga state gets updated (this includes joins) and messages come fast, parallel processing causes version conflicts.

[Chris Patterson]

You should be able to use the retry policy specifically for this situation. It's one of the key features of MT.

endpoint.UseRetry(Retry.Interval(...));

endpoint.StateMachineSaga(sagaRepository, stateMachine);

The order matters, the retry must come before the saga/consumer/etc.

On Wed, Jul 13, 2016 at 10:32 AM, Alexey Zimarev <azim...@gmail.com> wrote:

I have similar issue with using RavenDb saga repository. It happens with just two messages and CombinedEvent, the waiting bit was overriding in a race condition. When I turned on RavenDb optimistic concurrency, it started to fail on the wrong instance version. I do have the retry policy configured but this seems to have no effect on saga. At least for me it does not retry, just crashes the whole thing. I do not have any solution for this though, we are now using in memory repository since we are not in production yet but this is not acceptable.

My view on this that messages that are related to one saga instance should not be processed in parallel but rather put in a sequence or queue in memory and not ACKed until saga actually processed them. When saga state gets updated (this includes joins) and messages come fast, parallel processing causes version conflicts.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/f4301245-3e36-4bf2-8dc8-e866083b04b9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alexey Zimarev]

I didn’t know about configuration order, will try that. I thought about retries after looking at MongoDbSagaRepository since it throws an exception when it finds an override.

You received this message because you are subscribed to a topic in the Google Groups "masstransit-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/masstransit-discuss/IJrc1BG_f80/unsubscribe.
To unsubscribe from this group and all its topics, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAHX4WYcKL8aH9LKW2qcxj_sk8VPxePDvEZ%2BSWaB6ntnP8tfpkA%40mail.gmail.com.

[Alexey Zimarev]

Yes, we have it wrong:

            ec.LoadConsumers(context, contextName);
            ec.LoadStateMachineSagas(context, contextName);

            if (serviceBusSettings.Retries > 0)
            {
                ec.UseRetry(Retry.Exponential(
                    serviceBusSettings.Retries,
                    serviceBusSettings.RetryMinInterval,
                    serviceBusSettings.RetryMaxInterval,
                    serviceBusSettings.RetryIntervalDelta));
            }

[Chris Patterson]

Yeah, that does essentially nothing, since the UseRetry() is after the consumers and state machines are configured on the pipeline.

Moving the UseRetry above the other lines will resolve your issue.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/8665fd9d-98b4-4a56-9abc-1b0a6ad5bcf3%40googlegroups.com.

[Alexey Zimarev]

I actually found what my problem is.

I have no composite event status update because this struct has no set. RavenDb cannot deserialize it back.
Saving works but reading from the database always return the initial state.

How critical is it to change from no set to private set? I can submit a PR.

[Chris Patterson]

Just use an int, no reason to use that struct any longer. And int serializes much better into every data store known to man.

--

You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/59d80faa-d644-4812-a876-2d1d44b2502a%40googlegroups.com.

[Alexey Zimarev]

OK, this makes sense. Will do that, thanks for the suggestion.

You received this message because you are subscribed to a topic in the Google Groups "masstransit-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/masstransit-discuss/IJrc1BG_f80/unsubscribe.
To unsubscribe from this group and all its topics, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAHX4WYdJrHotAoTqg-C8kK2%2BSwVawWLiov%3D8mGVUFLdS4%3DqBRw%40mail.gmail.com.