Activities duplicated when running 2 instances of StateMachine

[ryan....@royaljay.net]

Hoping this is an easy one as I am still not 100% sure what is going on here. I have a routingslip workflow that reports events and state to a SagaStateMachine.

Behind the scenes, the routingslip activities are hosted in 3 different windows services spread over multiple hosts.

The StateMachine is hosted on it's own windows service on 3 hosts. 

There are 2 things that have recently started happening and I am not sure if they're related:

1. When multiple StateMachine service processes are running, activities get duplicated (run multiple times per workflow)

2. routing_slip_state_skipped queue is filling with compensation events, but we don't compensate any activities nor do we listen/handle that event in the state machine.

Any direction and help is greatly appreciated.

Thanks,

Ryan

[Chris Patterson]

Code would really help but first.

Are all the state machine services using the same queue?

Are all activities on unique queues?

Are any activities adding a log entry with a compensation address that was specified when the execute activity host was configured?

__

Chris Patterson

---

From: masstrans...@googlegroups.com <masstrans...@googlegroups.com> on behalf of ryan....@royaljay.net <ryan....@royaljay.net>
Sent: Saturday, October 7, 2017 4:41:51 PM
To: masstransit-discuss
Subject: [masstransit-discuss] Activities duplicated when running 2 instances of StateMachine

 

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/a4aa2a5d-44ba-4614-a772-1fbcfa35ab5f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ryan....@royaljay.net]

Chris,

Sorry code included as well as answers I hope.

All state machine services do use the same queue

All activities use unique queues

I do not believe we are adding a log entry with a compensation address, though if my code samples say otherwise, please excuse my ignorance.

Activity Configs (I included 1 activity and it's compensate, but there are multiple all in the same fasion):

cfg.ReceiveEndpoint(host, SendActivities.SendFaxActivity, e =>
{
 e.EnableExpress = false;
 e.EnablePartitioning = true;
 var compUri = new Uri(serviceBusBaseUri, SendActivities.SendFaxActivity_Compensate);
 e.ExecuteActivityHost<SendFaxActivity, SendFaxArguments>(compUri, context.Resolve<ILifetimeScope>());
});


cfg.ReceiveEndpoint(host, SendActivities.SendFaxActivity_Compensate, e =>
{
 e.EnableExpress = false;
 e.EnablePartitioning = true;
 e.CompensateActivityHost<SendFaxActivity, ActivityMeta>(context.Resolve<ILifetimeScope>(), x => x.UseRetry(y => y.Exponential(3, TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(5))));
});

State Machine service config:

builder.Register(context =>
 {
     var busControl = Bus.Factory.CreateUsingAzureServiceBus(cfg =>
 {
     var host = cfg.Host(ConnectionString(), c => { });


     cfg.EnableExpress = false;
     cfg.UseJsonSerializer();
     cfg.UseInMemoryOutbox();


 cfg.ReceiveEndpoint(host, SendWorkflowStateMachine.QueueName, e =>
 {
     e.UseInMemoryOutbox();
     e.PrefetchCount = 8;
     e.UseConcurrencyLimit(1);


 e.StateMachineSaga(context.Resolve<SendWorkflowStateMachine>(), context.Resolve<ILifetimeScope>(), x =>
 {


 });
 });


 cfg.ReceiveEndpoint(host, ReceiveWorkflowStateMachine.QueueName, e =>
 {
     cfg.EnableExpress = false;
     cfg.UseJsonSerializer();
     cfg.UseInMemoryOutbox();
     e.StateMachineSaga(context.Resolve<ReceiveWorkflowStateMachine>(), context.Resolve<ILifetimeScope>(),
   x =>
   {


   });
 });
 });


     return busControl;
 })
 .SingleInstance()
 .As<IBusControl>()
 .As<IBus>();

Better formatted samples with more activities as well: https://gist.github.com/ryankelley/0d581234a99a8e947b684b5b81a605c1

Again, really appreciate the help here, hoping I can figure out what's going on.

On Sunday, October 8, 2017 at 8:31:12 AM UTC-6, Chris Patterson wrote:

Code would really help but first.

Are all the state machine services using the same queue?

Are all activities on unique queues?

Are any activities adding a log entry with a compensation address that was specified when the execute activity host was configured?

__

Chris Patterson

---

From: masstrans...@googlegroups.com <masstrans...@googlegroups.com> on behalf of ryan....@royaljay.net <ryan....@royaljay.net>
Sent: Saturday, October 7, 2017 4:41:51 PM
To: masstransit-discuss
Subject: [masstransit-discuss] Activities duplicated when running 2 instances of StateMachine

 

Hoping this is an easy one as I am still not 100% sure what is going on here. I have a routingslip workflow that reports events and state to a SagaStateMachine.

Behind the scenes, the routingslip activities are hosted in 3 different windows services spread over multiple hosts.

The StateMachine is hosted on it's own windows service on 3 hosts. 

There are 2 things that have recently started happening and I am not sure if they're related:

1. When multiple StateMachine service processes are running, activities get duplicated (run multiple times per workflow)

2. routing_slip_state_skipped queue is filling with compensation events, but we don't compensate any activities nor do we listen/handle that event in the state machine.

Any direction and help is greatly appreciated.

Thanks,

Ryan

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

[Chris Patterson]

So while your activities have a compensate method, and are configured with a compensate API, it's hard without seeing the activity code if you're actually logging entries. But it does appear that if you're seeing compensated events, you are compensating routing slips.

It's likely a subscription has been added between the RoutingSlipActivityCompensated topic and your state machine service queue. Perhaps it was being handled at once point, but no longer processed. Or you are executing with a specific FaultAddress in the execute headers, or a subscription added to the routing slip for the compensated event, when that event isn't actually handled (this would be in the creation of the routing slip where activities and subscriptions are added.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsubscribe...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/a4aa2a5d-44ba-4614-a772-1fbcfa35ab5f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstransit-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/47d42b07-33da-4570-ad4d-8d084a7f913a%40googlegroups.com.

[Chris Patterson]

By subscriptions in the routing slip, I mean these: http://masstransit-project.com/MassTransit/advanced/courier/subscriptions.html

To post to this group, send email to masstransit-discuss@googlegroups.com.

[ryan....@royaljay.net]

Yes, in the routing slip build up we do add a subscription to the state machine for all events:

builder.AddSubscription(_endpointProvider.GetEndpointAddress(SendWorkflowStateMachine.QueueName), RoutingSlipEvents.All);

So the basic layout of each of our activities, is the execute does something and the compensate returns just returns that compensation is completed:  

return Task.Run(() => context.Compensated());

In looking at the Azure Service Bus skipped queue, it looks like compensation is being sourced from 1 of 2 activities in the workflow so I am adding some logging there to first off try and figure out what is causing the step to fail and therefore be compensated. 

Biggest head scratcher is how I am getting the duplicate activities to show up.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/a4aa2a5d-44ba-4614-a772-1fbcfa35ab5f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.