mt3 executing numerous activities causes some to not get acked eventually

202 views
Skip to first unread message

Brian Weeres

unread,
May 6, 2015, 5:50:02 PM5/6/15
to masstrans...@googlegroups.com
Using mt3 3.04. Quick test on 3.05 indicates same issue.

I am running into an issue where activities do not get acked and eventually block my queues. 
Pretty simple to reproduce. 

        for (int i = 0; i < 1000; i++)
        {
            Bus.Execute(CreateRoutingSlip());
        }

        private RoutingSlip CreateRoutingSlip()
        {
            var builder = new RoutingSlipBuilder(Guid.NewGuid());
            ActivityMessageOneCmd cmd = new ActivityMessageOneCmd();
            cmd.Data = "Msg One in Routing Slip.";
            builder.AddActivity("TestActivityOne", new Uri("rabbitmq://localhost/mt3/Exec_ActivityMessageOneCmd"), cmd);
            RoutingSlip slip = builder.Build();
            return slip;
        }


The activity does nothing but context.Completed();

(NOTE: Does not matter if Bus.Excecute is awaited or not)

I cannot reproduce this issue with consumers just a routingslip.

At one point I thought I had a good snapshot of the call stacks of the stuck threads but I can't seem to get that back so I am sceptical about that run however here are the details. Take it with a grain of salt, not sure those threads are related to the issue.


Six threads were here waiting on the lock
 public void ExchangeDeclare(string exchange, string type, bool durable, bool autoDelete, IDictionary<string, object> arguments)
        {
            lock (_rpcLock)   // 6 threads are here
                _model.ExchangeDeclare(exchange, type, durable, autoDelete, arguments); // one thread here
        }


One thread was executing  _model.ExchangeDeclare
The one in _model.ExchangeDeclare had a call stack of 
Worker Thread APPL_981060609_92 MassTransit.RabbitMqTransport.Integration.HaModel.ExchangeDeclare Normal
  mscorlib.dll!System.Threading.Monitor.Wait(object obj) + 0x1e bytes  
  RabbitMQ.Client.dll!RabbitMQ.Util.BlockingCell.Value.get() + 0x70 bytes  
  RabbitMQ.Client.dll!RabbitMQ.Client.Impl.SimpleBlockingRpcContinuation.GetReply() + 0x56 bytes  
  RabbitMQ.Client.dll!RabbitMQ.Client.Impl.ModelBase.ModelRpc(RabbitMQ.Client.Impl.MethodBase method, RabbitMQ.Client.Impl.ContentHeaderBase header, byte[] body) + 0xd1 bytes  
  RabbitMQ.Client.dll!RabbitMQ.Client.Framing.Impl.Model._Private_ExchangeDeclare(string exchange, string type, bool passive, bool durable, bool autoDelete, bool internal, bool nowait, System.Collections.Generic.IDictionary<string,object> arguments) + 0x141 bytes  
  RabbitMQ.Client.dll!RabbitMQ.Client.Impl.ModelBase.ExchangeDeclare(string exchange, string type, bool durable, bool autoDelete, System.Collections.Generic.IDictionary<string,object> arguments) + 0x86 bytes  
  RabbitMQ.Client.dll!RabbitMQ.Client.Impl.AutorecoveringModel.ExchangeDeclare(string exchange, string type, bool durable, bool autoDelete, System.Collections.Generic.IDictionary<string,object> arguments) + 0x141 bytes  
  MassTransit.RabbitMqTransport.dll!MassTransit.RabbitMqTransport.Integration.HaModel.ExchangeDeclare(string exchange, string type, bool durable, bool autoDelete, System.Collections.Generic.IDictionary<string,object> 

The other thread that seemed related was publishing a RoutingSlipCompleted. It was here:
Task SendUsingNewConnection(IPipe<ModelContext> modelPipe, CancellationToken cancellationToken)
        {
            IPipe<ConnectionContext> connectionPipe = Pipe.New<ConnectionContext>(x =>
            {
                x.ExecuteAsync(async connectionContext =>
                {
                    IModel model;
                    lock(connectionContext.Connection)
                        model = connectionContext.Connection.CreateModel(); // ON THIS LINE
                    var modelContext = new RabbitMqModelContext(connectionContext, model, connectionContext.CancellationToken);

Here is the stack trace for that one.
97 Worker Thread AMQP Connection amqp://localhost:5672 MassTransit.RabbitMqTransport.Integration.RabbitMqModelCache.SendUsingNewConnection Normal
  mscorlib.dll!System.Threading.Monitor.Wait(object obj) + 0x1e bytes  
  RabbitMQ.Client.dll!RabbitMQ.Util.BlockingCell.Value.get() + 0x70 bytes  
  RabbitMQ.Client.dll!RabbitMQ.Client.Impl.SimpleBlockingRpcContinuation.GetReply() + 0x56 bytes  
  RabbitMQ.Client.dll!RabbitMQ.Client.Impl.ModelBase.ModelRpc(RabbitMQ.Client.Impl.MethodBase method, RabbitMQ.Client.Impl.ContentHeaderBase header, byte[] body) + 0xd1 bytes  
  RabbitMQ.Client.dll!RabbitMQ.Client.Framing.Impl.Model._Private_ChannelOpen(string outOfBand) + 0x95 bytes  
  RabbitMQ.Client.dll!RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.CreateNonRecoveringModel() + 0xb7 bytes  
  RabbitMQ.Client.dll!RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.CreateModel() + 0x56 bytes  
  MassTransit.RabbitMqTransport.dll!MassTransit.RabbitMqTransport.Integration.RabbitMqModelCache.SendUsingNewConnection(MassTransit.RabbitMqTransport.ConnectionContext connectionContext) Line 64 + 0x67 bytes  
  MassTransit.dll!MassTransit.Pipeline.Filters.AsyncDelegateFilter<MassTransit.RabbitMqTransport.ConnectionContext>.Send(MassTransit.RabbitMqTransport.ConnectionContext context, 

If there is someway I can get more / better information let me know. 

Thanks,
Brian

Travis Smith

unread,
May 6, 2015, 6:48:47 PM5/6/15
to masstrans...@googlegroups.com
Did you create an issue for this on GitHub? It's easy to lose track of an email and this is simple enough repo steps that should be reasonable to track down. 

-Travis

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/73e2d66b-04a0-4c2b-bb0b-4a7beb759c28%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Chris Patterson

unread,
May 6, 2015, 9:02:20 PM5/6/15
to masstrans...@googlegroups.com
This seems to be related to faults, and moving messages to the error queue. I need to dig into this more.


To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAAqUTbnbvUy-f7GTyhmERk-hvi5BSTn2hWO0ULWN31SOWu4UfA%40mail.gmail.com.

Brian Weeres

unread,
May 7, 2015, 9:36:43 AM5/7/15
to masstrans...@googlegroups.com
No, I have not been posting the issues on github. I will do that as well from now on.

-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

Chris Patterson

unread,
May 7, 2015, 10:19:45 AM5/7/15
to masstrans...@googlegroups.com
I have a theory that this has something to do with threads and locking on the exchange declaration code. Since there is an _rpcLock around it, I think it's blocking some callback or something. Strangely I'm getting a null reference exception deep inside the RabbitMQ.Client - which makes me think it has a bug (3.5.1) or something else is amiss. I'm going to work on replacing the locking with a different approach that should not block other threads, before getting the RMQ client built and tested in debug.



-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/7c2148ce-42c0-4bb5-ae87-b52389cc1d69%40googlegroups.com.

Brian Weeres

unread,
May 7, 2015, 2:44:46 PM5/7/15
to masstrans...@googlegroups.com
Sounds good. Let me know if you want me to put it through the paces before you package it up.

-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

Chris Patterson

unread,
May 8, 2015, 2:30:30 PM5/8/15
to masstrans...@googlegroups.com
So this simple test fails, with the debug log shown. It seems to hang on creating the second publish endpoint. A simple consumer publishing two different types.




-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/341a1a6c-5596-4d1b-9e6d-bfa7f29b9a69%40googlegroups.com.

Chris Patterson

unread,
May 8, 2015, 6:25:34 PM5/8/15
to masstrans...@googlegroups.com
It's hanging trying to create the model for the second endpoint it builds for the second event being published. Something is causing the connection to lock up, clearly some interaction of threads happening here.

Chris Patterson

unread,
May 13, 2015, 8:30:10 PM5/13/15
to masstrans...@googlegroups.com
So I branched off my investigation, and left it aside for now.

I've upgraded to RabbitMQ 2.5.2 with the latest develop branch if you care to pull it down and try it. I'm not optimistic, however. But at this point, I think it has something to do with an async thread blocking on a synchronization context.

Brian Weeres

unread,
May 14, 2015, 11:26:36 AM5/14/15
to masstrans...@googlegroups.com
Thanks for the update. I noticed if I run in release mode it pretty much happens all the time. If in debug mode it still happens just not as often. I am able to reproduce the same thread calls stacks that I showed earlier in this post by running my application in release and then attaching to it once I noticed some messages not being acked. 

There is always one thread here:
  mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout, bool exitContext) + 0x16 bytes
  RabbitMQ.Client.dll!RabbitMQ.Util.BlockingCell.Value.get() + 0x4e bytes
  RabbitMQ.Client.dll!RabbitMQ.Client.Impl.SimpleBlockingRpcContinuation.GetReply() + 0xf bytes
  RabbitMQ.Client.dll!RabbitMQ.Client.Impl.ModelBase.ModelRpc(RabbitMQ.Client.Impl.MethodBase method, RabbitMQ.Client.Impl.ContentHeaderBase header, byte[] body) + 0x8e bytes
  RabbitMQ.Client.dll!RabbitMQ.Client.Framing.Impl.Model._Private_ExchangeDeclare(string exchange, string type, bool passive, bool durable, bool autoDelete, bool internal, bool nowait, System.Collections.Generic.IDictionary<string,object> arguments) + 0x7b bytes
  RabbitMQ.Client.dll!RabbitMQ.Client.Impl.AutorecoveringModel.ExchangeDeclare(string exchange, string type, bool durable, bool autoDelete, System.Collections.Generic.IDictionary<string,object> arguments) + 0x6b bytes
  MassTransit.RabbitMqTransport.dll!MassTransit.RabbitMqTransport.Integration.HaModel.ExchangeDeclare(string exchange, string type, bool durable, bool autoDelete, System.Collections.Generic.IDictionary<string,object> arguments) + 0x58 bytes
  MassTransit.RabbitMqTransport.dll!MassTransit.RabbitMqTransport.Pipeline.SendExchangeBindingModelFilter.MassTransit.Pipeline.IFilter<MassTransit.RabbitMqTransport.ModelContext>.Send(MassTransit.RabbitMqTransport.ModelContext context, MassTransit.Pipeline.IPipe<MassTransit.RabbitMqTransport.ModelContext> next) + 0x83 bytes


The other threads are here:
  MassTransit.RabbitMqTransport.dll!MassTransit.RabbitMqTransport.Integration.HaModel.ExchangeDeclare(string exchange, string type, bool durable, bool autoDelete, System.Collections.Generic.IDictionary<string,object> arguments) + 0x3d bytes
  MassTransit.RabbitMqTransport.dll!MassTransit.RabbitMqTransport.Pipeline.SendExchangeBindingModelFilter.MassTransit.Pipeline.IFilter<MassTransit.RabbitMqTransport.ModelContext>.Send(MassTransit.RabbitMqTransport.ModelContext context, MassTransit.Pipeline.IPipe<MassTransit.RabbitMqTransport.ModelContext> next) + 0x83 bytes
  MassTransit.dll!MassTransit.Pipeline.Pipes.FilterPipe<MassTransit.RabbitMqTransport.ModelContext>.Send(MassTransit.RabbitMqTransport.ModelContext context) + 0x40 bytes
  MassTransit.RabbitMqTransport.dll!MassTransit.RabbitMqTransport.Pipeline.PrepareSendExchangeFilter.MassTransit.Pipeline.IFilter<MassTransit.RabbitMqTransport.ModelContext>.Send(MassTransit.RabbitMqTransport.ModelContext context, MassTransit.Pipeline.IPipe<MassTransit.RabbitMqTransport.ModelContext> next) + 0x4d bytes
  MassTransit.dll!MassTransit.Pipeline.Pipes.FilterPipe<MassTransit.RabbitMqTransport.ModelContext>.Send(MassTransit.RabbitMqTransport.ModelContext context) + 0x40 bytes
  MassTransit.RabbitMqTransport.dll!MassTransit.RabbitMqTransport.Integration.RabbitMqModelCache.SendUsingExistingConnection.MoveNext() + 0x66 bytes


Someone reported the same issue from a .Net client against Rabbit in 2010. http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-January/005863.html. Unfortunately no resolution was supplied in the thread.

Brian

-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

Chris Patterson

unread,
May 14, 2015, 10:37:22 PM5/14/15
to masstrans...@googlegroups.com
So it's a threading issue when dealing with connections or models, wonderful. The RPC layer of the AMQP channel is getting, confused. Ugh.



-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/387ad6b9-31f2-41e8-974a-84adbefe2883%40googlegroups.com.

Brian Weeres

unread,
May 15, 2015, 9:29:26 AM5/15/15
to masstrans...@googlegroups.com
I don't think it matters but I just realized I better clean up the terminology I used. In all cases I am using a debug build for my code but not Mass Transit which just uses the nuget packages. 3.0.5.  The failed acks happen all the time when I just run the app and less often when I debug the app from visual studio. (Not surprised.)  Sorry for my confusing use of the term release mode.

-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

Chris Patterson

unread,
May 15, 2015, 10:19:20 AM5/15/15
to masstrans...@googlegroups.com
Oh, i know what you're saying. The problems I'm seeing are really just TPL timing related, which is really a bitch to deal with since you have limited scheduler control.

If you could somehow determine if there are SynchronizationContext's that are somehow getting in the way of operations, that would help. That's my next issue to determine, to see if that's part of the problem and if somehow a ConfigureAwait() might help avoid that concern.



-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/9d0ee5e5-4b6e-4a40-aaba-2cc670fb6dc5%40googlegroups.com.

Brian Weeres

unread,
May 15, 2015, 1:13:20 PM5/15/15
to masstrans...@googlegroups.com
I will take a deeper dive into that area early next week.

-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

Chris Patterson

unread,
May 15, 2015, 3:17:50 PM5/15/15
to masstrans...@googlegroups.com
Well a sprinkled an epic ton of ConfigureAwait() and it still hangs. so I'm not sure what to do next. I also changed all the internal publishes to no longer await, since they are adding their Task's to the pendingTask collection in the consume context, which is awaited as a whole before the message is ack'd (which makes all those publishes happen concurrently as scheduled). I'll continue to plug away at it.


-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/8432adc1-0eec-4d18-8e49-55e96c16ecdc%40googlegroups.com.

Chris Patterson

unread,
May 19, 2015, 12:00:15 AM5/19/15
to masstrans...@googlegroups.com
What version of RabbitMQ are you using on the server? I switched to another machine and can't get it to fail with RabbitMQ v3.4.2 server (using the same 3.5.2 client library).

Brian Weeres

unread,
May 19, 2015, 10:05:42 AM5/19/15
to masstrans...@googlegroups.com
3.3.0. I will upgrade to 3.4.2 on a different machine and let you know. Wouldn't that be nice if that was the issue.

-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

Chris Patterson

unread,
May 19, 2015, 10:11:52 AM5/19/15
to masstrans...@googlegroups.com
Well, on my physical machine running RMQ 3.4.2 server, I can't get it to fail. I'm going to downgrade my mac to the same version later today and see if the issue goes away. The fact you're seeing it with 3.3 isn't encouraging though. It could by that physical vs VM, it doesn't thread lock on physical due to real hardware or just bad luck with timing.



-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/95ead62a-646f-4ffa-9b7a-3a65cd3e5988%40googlegroups.com.

Brian Weeres

unread,
May 19, 2015, 1:23:46 PM5/19/15
to masstrans...@googlegroups.com
I have the same issue on a physical machine using RMQ 3.4.2. server. Happens right away if I don't debug. 

Three unacked messages

Two threads here:
Not Flagged 7552 93 Worker Thread Worker Thread RabbitMQ.Util.BlockingCell.Value.get() Normal


One thread here:
Not Flagged 9128 82 Worker Thread Worker Thread MassTransit.RabbitMqTransport.Integration.HaModel.ExchangeDeclare() Normal







-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

Chris Patterson

unread,
May 19, 2015, 5:48:08 PM5/19/15
to masstrans...@googlegroups.com
Ugh, I hate threading.


-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/75bed452-2d5d-4797-aa8b-dd5d89b6ad46%40googlegroups.com.

Brian Weeres

unread,
May 27, 2015, 5:58:45 PM5/27/15
to masstrans...@googlegroups.com
 
I have the same issue with 3.06 and which uses RMQ 3.5.3. I am using Rabbit MQ 3.3 so perhaps that is a difference. Still only on activities.
Maybe I'm doing something wrong like registeriing my activities or the code in my activities.

I did find something interesting that may be related  In my activity I do this:
 public async Task<MassTransit.Courier.ExecutionResult> Execute(MassTransit.Courier.ExecuteContext<IActivityOne> context)
        {
            Console.WriteLine("TestActivityOne");
            await Task.Delay(10);
            return context.Completed();
        }

If I take out the await Task.Delay(10) OR if I do not await it then all of my messages get acked. (Tried up to 10,000 numerous times.)
If I keep the await Task.Delay in I will always get a few that sit in my queue unacked if I run about 2,000.
 
Here is my entire code that I run. Just create a console app, add the MQ nugets and make this your program.cs. (Needs virtual host MT3). 


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using MassTransit.Courier;
using MassTransit.Courier.Factories;
using MassTransit.Courier.Contracts;
using MassTransit;
using System.Threading;
 
namespace TestActivityMT3
{
    class Program
    {
        static IBusControl BUS;
        static BusHandle BUSHANDLE;
        static void Main(string[] args)
        {
            CreateBus();
            SendActivities();
            Console.WriteLine("Hit Enter");
            Console.ReadLine();
            BUSHANDLE.Stop();
        }
 
        private async static void SendActivities()
        {
            for (int i = 0; i < 2000; i++)
            {
                Console.WriteLine("sending: " + i);
                RoutingSlip slip = CreateRoutingSlip();
                await BUS.Execute(slip);
            }
        }
 
        private static RoutingSlip CreateRoutingSlip()
        {
            var builder = new RoutingSlipBuilder(Guid.NewGuid());
            ActivityOne cmd = new ActivityOne();
            cmd.Data = "Msg One in Routing Slip.";
            builder.AddActivity("TestActivityOne", new Uri("rabbitmq://localhost/mt3/Exec_ActivityOne"), cmd);
            RoutingSlip slip = builder.Build();
            return slip;
        }
 
 
        private static void CreateBus()
        {
                var hostAddress = new Uri("rabbitmq://localhost/mt3");
 
                BUS = Bus.Factory.CreateUsingRabbitMq(x =>
                {
                    var host = x.Host(hostAddress, h =>
                    {
                        h.Username("guest");
                        h.Password("guest");
                    });
 
                    var activityFactoryMessageOne = new FactoryMethodActivityFactory<TestActivityOne, IActivityOne, ILogActivity>(_ => new TestActivityOne(), _ => new TestActivityOne());
                    Uri uri = new Uri("rabbitmq://localhost/mt3/Comp_ActivityOne");
                    x.ReceiveEndpoint(host, "Exec_ActivityOne", e =>
                    {
                        e.ExecuteActivityHost<TestActivityOne, IActivityOne>(uri, activityFactoryMessageOne);
                    });
 
                    x.ReceiveEndpoint(host, "Comp_ActivityOne", e =>
                    {
                        e.CompensateActivityHost<TestActivityOne, ILogActivity>(() => new TestActivityOne());
                    });
 
 
                });
 
                BUSHANDLE = BUS.Start();
            
 
        }
 
    }
 
 
    public class TestActivityOne : MassTransit.Courier.Activity<IActivityOne, ILogActivity>
    {
 
        public async Task<MassTransit.Courier.ExecutionResult> Execute(MassTransit.Courier.ExecuteContext<IActivityOne> context)
        {
            Console.WriteLine("TestActivityOne");
            await Task.Delay(10);
            return context.Completed();
        }
 
        public async Task<MassTransit.Courier.CompensationResult> Compensate(MassTransit.Courier.CompensateContext<ILogActivity> context)
        {
            return context.Compensated();
        }
    }
 
    public interface ILogActivity
    {
    }
 
    public interface IActivityOne
    {
        string Data { get; }
    }
 
    public class ActivityOne : IActivityOne
    {
        public string Data { get; set; }
 
    }
 
}

Chris Patterson

unread,
May 28, 2015, 2:22:25 PM5/28/15
to masstrans...@googlegroups.com
Yeah, something is still dying in the TPL deadlocking on the RabbitMQ client.


--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/a845cf59-6d74-4801-b918-8ce1920ad051%40googlegroups.com.

Chris Patterson

unread,
May 31, 2015, 8:19:43 PM5/31/15
to masstrans...@googlegroups.com
Another update posted in org/mt3 if you want to try it. I added a few things, including waiting for pending confirms, that will hopefully help out.

Reply all
Reply to author
Forward
0 new messages