Exception when publishing on startup

702 views
Skip to first unread message

Karl Atkinson

unread,
May 16, 2016, 6:31:47 AM5/16/16
to masstransit-discuss
We're seeing occasional issues when publishing messages MT3 using RabbitMq.

With logging turned on we get the following exception appear. This does not throw an exception but silently fails to publish the event.
Further messages publishes using this bus fail silently without any log entries.

DEBUG 2016-05-16 10:18:05 - The model usage threw an exception - System.OperationCanceledException: The supervisor is stopping, no additional participants can be created
   at MassTransit.Util.TaskSupervisor.CreateParticipant(String tag)
   at MassTransit.RabbitMqTransport.Contexts.SharedModelContext..ctor(ModelContext context, CancellationToken cancellationToken, ITaskScope scope)
   at MassTransit.RabbitMqTransport.Integration.RabbitMqModelCache.ModelScope.<Attach>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MassTransit.RabbitMqTransport.Integration.RabbitMqModelCache.<SendUsingExistingModel>d__11.MoveNext()


We're seeing this intermittently on our production on application startup (we generally see it after IIS has recycled our app pool).
We register our bus as a singleton (using Unity IOC) and we see a successful publish followed by nothing (although our app attempts to publish and receives no exception on the publish).

This issue is intermittent but we can replicate it on our QA environment. 
We believe the behaviour may have been introduced in MT 3.2.x. Upgrading to 3.3.2 has not solved the problem.
Investigations have led to us think it's load related, where two (or more) messages are attempted to be published when the connection may still be being established to RabbitMq?

Chris Patterson

unread,
May 16, 2016, 12:16:35 PM5/16/16
to masstrans...@googlegroups.com
So the message fails to publish and does not throw an exception back to the calling method? That would be bad, have you verified that the message does indeed not end up in a queue?

It might retry and use a new channel without any log detail if there were no further errors. 

__
Chris Patterson




--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/6aa481e4-a96c-4749-bb68-f1792c266673%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Dan Fox

unread,
May 17, 2016, 5:08:54 AM5/17/16
to masstransit-discuss
Hi Chris,

The message fails to publish and doesn't appear on the queue.  No further messages appear on the queue and no errors are logged.

Our current working assumption is that there is a problem at startup time: if two similar messages are published while the connections to RabbitMQ are being initialised, then it causes the connection to break (which it doesn't recover from).

I've reproduced it on IIS 6, but I can't reproduce it in IIS 8 which I'm finding strange.



On Monday, May 16, 2016 at 5:16:35 PM UTC+1, Chris Patterson wrote:
So the message fails to publish and does not throw an exception back to the calling method? That would be bad, have you verified that the message does indeed not end up in a queue?

It might retry and use a new channel without any log detail if there were no further errors. 

__
Chris Patterson




On Mon, May 16, 2016 at 3:31 AM -0700, "Karl Atkinson" <karl.a...@gmail.com> wrote:

We're seeing occasional issues when publishing messages MT3 using RabbitMq.

With logging turned on we get the following exception appear. This does not throw an exception but silently fails to publish the event.
Further messages publishes using this bus fail silently without any log entries.

DEBUG 2016-05-16 10:18:05 - The model usage threw an exception - System.OperationCanceledException: The supervisor is stopping, no additional participants can be created
   at MassTransit.Util.TaskSupervisor.CreateParticipant(String tag)
   at MassTransit.RabbitMqTransport.Contexts.SharedModelContext..ctor(ModelContext context, CancellationToken cancellationToken, ITaskScope scope)
   at MassTransit.RabbitMqTransport.Integration.RabbitMqModelCache.ModelScope.<Attach>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MassTransit.RabbitMqTransport.Integration.RabbitMqModelCache.<SendUsingExistingModel>d__11.MoveNext()


We're seeing this intermittently on our production on application startup (we generally see it after IIS has recycled our app pool).
We register our bus as a singleton (using Unity IOC) and we see a successful publish followed by nothing (although our app attempts to publish and receives no exception on the publish).

This issue is intermittent but we can replicate it on our QA environment. 
We believe the behaviour may have been introduced in MT 3.2.x. Upgrading to 3.3.2 has not solved the problem.
Investigations have led to us think it's load related, where two (or more) messages are attempted to be published when the connection may still be being established to RabbitMq?

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

Dan Fox

unread,
May 17, 2016, 5:40:38 AM5/17/16
to masstransit-discuss
My mistake... wrong IIS version quoted...

I've reproduced it on IIS 7.5, but I can't reproduce it in IIS 8.

Chris Patterson

unread,
May 17, 2016, 10:53:28 AM5/17/16
to masstrans...@googlegroups.com
Interesting. Are you making sure that the bus is started before allowing access to it from the container? Once it is started I don't see how this could happen. I'm trying to reproduce it personally but having no success. A full debug log of the bug in action might help me figure it out since I could trace through the code. 


__
Chris Patterson




To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

Karl Atkinson

unread,
May 17, 2016, 11:31:10 AM5/17/16
to masstransit-discuss
We're creating the bus in the constructor of our publisher, calling Start() and not StartAsync() which is then being registered by unity so I would think this shouldn't cause anything to be able to publish to it before then.

Here is a code snippet...

        public MassTransitPublisher(
            QueueConfiguration config, 
            IPublisherLogger logger)
        {
            _config = config;
            _logger = logger;

            _logger.Info("Start the bus...");
            _busControl = Bus.Factory.CreateUsingRabbitMq(sbc =>
            {
                sbc.Host(new Uri(_config.QueueUrl), h =>
                {
                    h.Username(_config.Username);
                    h.Password(_config.Password);
                    h.Heartbeat(_config.Timeout);
                });

                sbc.UseLog4Net();
                sbc.UseRetry(Retry.All().Immediate(RetryLimit));
            });
            
            _busControl.Start();
        }

We'll try and get a complete log for you tomorrow.

Dan Fox

unread,
May 18, 2016, 9:10:14 AM5/18/16
to masstransit-discuss
Zip file with logs attached.  The zip contains the log files from 5 runthroughs:
For runs 1-4, we received one message on the queue and then no more
For run 5 no messages appeared on the queue

Most of the logs have an exception in them which may be a symptom or a cause.  Sometimes there is a disposing message which seems strange so close to application startup.

The outputs all have two messages saying "Starting bus hosts..." which seems odd.  We do have two RabbitMQ buses but only one is configured to use log4net at the moment
mt_log.zip

Chris Patterson

unread,
May 18, 2016, 9:48:07 AM5/18/16
to masstrans...@googlegroups.com
Thanks. I'll look over these today. And once you configure log4net for one bus it's used for all in the AppDomain since it's a domain wide setting on the logger. That's why you see both of them. 

__
Chris Patterson




To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

Dan Fox

unread,
May 19, 2016, 7:07:34 AM5/19/16
to masstransit-discuss
Hi Chris,

I've isolated the issue to a potential threading issue in the LazyMemoryCache.cs

I've raised a PR here: https://github.com/MassTransit/MassTransit/pull/561 so you can see the change I have made that seems to make it perform more reliably on our setup.

Hopefully someone can spot something in this method which could cause a problem at startup like we are seeing.  I imagine the lock I have added will carry an undesirable performance penalty, but it's the best way to show you the issue.

Thanks

Dan Fox

unread,
May 19, 2016, 8:55:35 AM5/19/16
to masstransit-discuss
It is possible though that because of the lock statement throttling throughput, the issue is actually a bit further down the pipeline and this change is just reducing the chances of a problem rather than fixing it completely

Chris Patterson

unread,
May 19, 2016, 11:57:01 AM5/19/16
to masstrans...@googlegroups.com
This is the area I suspected as well, given that it was added during the timeframe you indicated. I think the entire cache needs to get a less smart, and perhaps become single threaded using a separate thread pool to avoid blocking but to keep the system moving quickly without locks (but synchronize on endpoint creation).

I'll look closer shortly.


To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/dd263645-5b7b-4953-9fe2-6a057f34ed04%40googlegroups.com.

Dan Fox

unread,
May 19, 2016, 12:06:11 PM5/19/16
to masstransit-discuss
Thanks Chris. If you need any more help from us please ask and we'll try our best!
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

Chris Patterson

unread,
May 19, 2016, 1:45:19 PM5/19/16
to masstrans...@googlegroups.com
I just pushed a commit to develop that changes how the cache works, if you all could check it out and see if it helps with the issue.


To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/e0f1d7e5-55cc-4e07-84b8-7e65e0c261ee%40googlegroups.com.

Dan Fox

unread,
May 20, 2016, 6:08:43 AM5/20/16
to masstransit-discuss
Good news... looks like that resolved it.  I had an environment under load which hit the issue ~70% of the time on the original code.  When I try using the latest commit built locally and patched onto the server, I haven't reproduced it.

Do you think it will be possible to get a new version in NuGet soon?  Or is there something else in the next release that we'll be waiting on?

Many thanks for the help!
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

Chris Patterson

unread,
May 20, 2016, 8:18:14 AM5/20/16
to masstrans...@googlegroups.com
I'll release the fix in the next day or two once I load test it for performance. 

__
Chris Patterson




To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
Message has been deleted

Anh Bui Trong

unread,
May 23, 2016, 12:02:47 AM5/23/16
to masstransit-discuss
We are having similar issue, our service is normally idle over weekend (when service would not serve any request for about 2 days) then after weekend IRequestClient will throw exception when trying to send request to server. Client Service need to be restarted in order for it to work  again:

{"message":"An error has occurred.","exceptionMessage":"The supervisor is stopping, no additional scopes can be created","exceptionType":"System.OperationCanceledException","stackTrace":"   at MassTransit.Util.TaskSupervisor.CreateScope(String tag)\r\n   at MassTransit.Util.TaskSupervisorExtensions.CreateScope(ITaskSupervisor supervisor, String tag, Func`1 afterStopped)\r\n   at MassTransit.RabbitMqTransport.Integration.RabbitMqModelCache.Send(IPipe`1 connectionPipe, CancellationToken cancellationToken)\r\n   at MassTransit.RabbitMqTransport.RabbitMqSendTransport.<MassTransit-Transports-ISendTransport-Send>d__6`1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at MassTransit.MessageRequestClient`2.<Request>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n

var workflowStatus = await _
combinedWorkflowStatusRequestClient.Request(new CheckWorkflowStatus
{
       AsOfDate = value
}, token);

We are using MassTransit version="3.2.2"

Chris Patterson

unread,
May 23, 2016, 12:08:29 AM5/23/16
to masstrans...@googlegroups.com
The fix is coming, sorry for not getting to it yet. It will be there soon.


To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/4ffb687f-6e15-4ef7-bef7-1fbb79d3099c%40googlegroups.com.

Anh Bui Trong

unread,
May 30, 2016, 9:25:42 PM5/30/16
to masstransit-discuss
we upgraded to version 3.3.3 but still encounter the error after service is idle over the weekend. We have to manually restart service after weekend :(

System.OperationCanceledException: The supervisor is stopping, no additional scopes can be created

   at MassTransit.Util.TaskSupervisor.CreateScope(String tag)

   at MassTransit.Util.TaskSupervisorExtensions.CreateScope(ITaskSupervisor supervisor, String tag, Func`1 afterStopped)

   at MassTransit.RabbitMqTransport.Integration.RabbitMqModelCache.Send(IPipe`1 connectionPipe, CancellationToken cancellationToken)

   at MassTransit.RabbitMqTransport.RabbitMqSendTransport.<MassTransit-Transports-ISendTransport-Send>d__6`1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at MassTransit.MessageRequestClient`2.<Request>d__6.MoveNext()

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

Dan Fox

unread,
May 31, 2016, 3:41:26 AM5/31/16
to masstransit-discuss
I've just looked at our recent logs and since we upgraded our service to v3.3.3 on 24th May we've not experienced any issues.

This might be another issue that appears as the same symptom.

Henry Tonnison

unread,
Jun 21, 2016, 11:55:17 AM6/21/16
to masstransit-discuss
I am experiencing this issue as well, not at startup however.

I have looked over your fix pull request and was wondering whether 
async void CleanupCacheItem(Task<TValue> valueTask, string textKey, CacheEntryRemovedReason reason)


being called from
var cacheItemPolicy = _policyProvider(new CacheExpirationSelector(key)).Policy;
cacheItemPolicy.RemovedCallback = OnCacheItemRemoved;

Should be called within your
_scheduler = new LimitedConcurrencyLevelTaskScheduler(1);


I cannot repro the issue, but from looking around the code I am assuming it happens during a faulty rabbit connection occuring when we attempt a send.

Martin Rapavy

unread,
Mar 2, 2017, 10:56:27 AM3/2/17
to masstransit-discuss
We are having exactly the same issue with version 3.5.5. The idle period necessary for this to happen seems to be approx. 25hrs.

@Chris: is there any resolution to this issue?
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

Chris Patterson

unread,
Mar 2, 2017, 12:17:45 PM3/2/17
to masstrans...@googlegroups.com
Seems to be related to the send endpoint getting stale after a long period of non-usage, during which the broker connection may have been severed and restored.

I'll end an issue for this and see if I can work through a resolution, the model context of RabbitMQ should be able to notify the send endpoint that it is now dead and won't be usable any longer, I'm just not sure if that event is getting pushed all the way to the cached endpoint. I just replaced that cache with a new one, I should be able to bend it to my will to evict the send endpoints if the model context is disconnected.


To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsubscribe...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsubscribe...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsubscribe...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsubscribe...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.
To post to this group, send email to masstransit-discuss@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/1794afc3-34d3-4b00-951c-b99e950cc4e5%40googlegroups.com.

Chris Patterson

unread,
Mar 2, 2017, 12:34:36 PM3/2/17
to masstrans...@googlegroups.com

To post to this group, send email to masstransit-discuss@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages