Graceful shutdown using RabbitMQ and TopShelf
1,995 views
Skip to first unread message
Clay Anderson
Oct 17, 2012, 6:41:10 PM10/17/12
to masstrans...@googlegroups.com
Hi,
We're trying to gracefully shut down our service (i.e. no errors in our log). We're using RabbitMQ, TopShelf, and log4net (including log4net integration packages for TopShelf and MassTransit).
When restarting the service manually in windows, our service shuts down gracefully. However, when rebooting the server, we get TONS for errors from TopShelf and MassTransit. The errors are around RabbitMQ (AMQP) connections being closed. We listed the RabbitMQ service as a dependency for our service using TopShelf. This takes effect when stopping, restarting, and starting services manually, but doesn't seem to take effect when Windows is shutting down.
Is there anything we can do to detect that Windows is shutting down and attempt to shut down / dispose of the MassTransit bus so we don't flood our logs with false errors?
This may be more of a TopShelf than MassTransit question, but I'm hopefully still getting to the right group of people.
Thanks.
Chris Patterson
Oct 18, 2012, 12:22:24 AM10/18/12
to masstrans...@googlegroups.com
There is an optional CanShutdown() or something like that where you
can get a call to your service on shutdown, not sure that's what
you're seeing though. You are disposing of the IServiceBus on service
stop, right?
> --
> You received this message because you are subscribed to the Google Groups
> "masstransit-discuss" group.
> To post to this group, send email to masstrans...@googlegroups.com.
> To unsubscribe from this group, send email to
> masstransit-dis...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/masstransit-discuss/-/SwGOWoSzHB8J.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
can get a call to your service on shutdown, not sure that's what
you're seeing though. You are disposing of the IServiceBus on service
stop, right?
> You received this message because you are subscribed to the Google Groups
> "masstransit-discuss" group.
> To post to this group, send email to masstrans...@googlegroups.com.
> To unsubscribe from this group, send email to
> masstransit-dis...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/masstransit-discuss/-/SwGOWoSzHB8J.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
Clay Anderson
Oct 18, 2012, 11:44:10 AM10/18/12
to masstrans...@googlegroups.com
Chris,
We are utilizing TopShelf's EnableShutdown() option and calling MassTransit.Bus.Shutdown() upon TopShelf's Shutdown event. I think the issue is that when Windows shuts down, it ignores the dependency of our service on the RabbitMQ service and shuts them down at the same time. When RabbitMQ shuts down, we get exceptions from MassTransit right away, perhaps a half second before TopShelf calls our method so we can shutdown the bus.
Program.cs and the generated log file are here. Let me know if you'd like our sample project.
Program.cs and the generated log file are here. Let me know if you'd like our sample project.
Thanks.
Clay
Clay Anderson
Oct 18, 2012, 11:46:13 AM10/18/12
to masstrans...@googlegroups.com
FYI shutdown of Windows occurred at 09:19.
Chris Patterson
Oct 22, 2012, 1:26:37 PM10/22/12
to masstrans...@googlegroups.com
I'm pretty sure this is resolved in 3.0.2, but I've but a couple of
additional checks into the develop branch just to make sure we don't
cause an NRE.
On Mon, Oct 22, 2012 at 8:32 AM, Derek Greer <dbg...@gmail.com> wrote:
> I stumbled upon this and, though I'm not using MassTransit, I thought I'd
> comment since my team is also seeing Topshelf exceptions only when Windows
> reboots:
>
> -------
> FATAL
> 2012-10-21 01:33:02,010 - The service did not shut down gracefully
>
>
> System.NullReferenceException: Object reference not set to an instance of an
> object.
> at
> Topshelf.Builders.DelegateServiceBuilder`1.DelegateServiceHandle.Shutdown(HostControl
> hostControl)
> at Topshelf.Runtime.Windows.WindowsServiceHost.OnShutdown()
>
> -------
>
> We're using Topshelf and Topshelf.Log4net version 3.0.1.
additional checks into the develop branch just to make sure we don't
cause an NRE.
On Mon, Oct 22, 2012 at 8:32 AM, Derek Greer <dbg...@gmail.com> wrote:
> I stumbled upon this and, though I'm not using MassTransit, I thought I'd
> comment since my team is also seeing Topshelf exceptions only when Windows
> reboots:
>
> -------
> FATAL
> 2012-10-21 01:33:02,010 - The service did not shut down gracefully
>
>
> System.NullReferenceException: Object reference not set to an instance of an
> object.
> at
> Topshelf.Builders.DelegateServiceBuilder`1.DelegateServiceHandle.Shutdown(HostControl
> hostControl)
> at Topshelf.Runtime.Windows.WindowsServiceHost.OnShutdown()
>
> -------
>
> We're using Topshelf and Topshelf.Log4net version 3.0.1.
> --
> You received this message because you are subscribed to the Google Groups
> "masstransit-discuss" group.
> To post to this group, send email to masstrans...@googlegroups.com.
> To unsubscribe from this group, send email to
> masstransit-dis...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/masstransit-discuss/-/_orW-O2wP_UJ.
> You received this message because you are subscribed to the Google Groups
> "masstransit-discuss" group.
> To post to this group, send email to masstrans...@googlegroups.com.
> To unsubscribe from this group, send email to
> masstransit-dis...@googlegroups.com.
> To view this discussion on the web visit
Clay Anderson
Oct 31, 2012, 4:58:23 PM10/31/12
to masstrans...@googlegroups.com
Unfortunately, it's 3.0.2 (file version 3.0.105) of TopShelf that this is happening in. Attached is the full VS solution.
Any other ideas would be helpful. We're stuck.
Thanks.
Clay
Clay Anderson
Nov 9, 2012, 4:00:31 PM11/9/12
to masstrans...@googlegroups.com
Just upgraded to TopShelf 3.1.0 and MassTransit 2.7.0 and it is still happening. Here's what the log looked like during Windows shutdown:
2012-11-09 13:35:43,208 WARN MassTransit.Transports.DefaultConnectionPolicy [4] - Invalid Connection when executing callback
System.IO.EndOfStreamException: SharedQueue closed
at RabbitMQ.Util.SharedQueue.EnsureIsOpen()
at RabbitMQ.Util.SharedQueue.Dequeue(Int32 millisecondsTimeout, Object& result)
at MassTransit.Transports.RabbitMq.RabbitMqConsumer.Get(TimeSpan timeout) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqConsumer.cs:line 85
at MassTransit.Transports.RabbitMq.InboundRabbitMqTransport.<>c__DisplayClass1.<Receive>b__0(RabbitMqConnection connection) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\InboundRabbitMqTransport.cs:line 64
2012-11-09 13:35:43,208 WARN MassTransit.Transports.DefaultConnectionPolicy [7] - Invalid Connection when executing callback
System.IO.EndOfStreamException: SharedQueue closed
at RabbitMQ.Util.SharedQueue.EnsureIsOpen()
at RabbitMQ.Util.SharedQueue.Dequeue(Int32 millisecondsTimeout, Object& result)
at MassTransit.Transports.RabbitMq.RabbitMqConsumer.Get(TimeSpan timeout) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqConsumer.cs:line 85
at MassTransit.Transports.RabbitMq.InboundRabbitMqTransport.<>c__DisplayClass1.<Receive>b__0(RabbitMqConnection connection) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\InboundRabbitMqTransport.cs:line 64
2012-11-09 13:35:43,243 DEBUG MassTransit.Transports.DefaultConnectionPolicy [4] - Disconnecting connection handler.
2012-11-09 13:35:43,243 DEBUG MassTransit.Transports.DefaultConnectionPolicy [7] - Disconnecting connection handler.
2012-11-09 13:35:43,276 WARN MassTransit.Transports.RabbitMq.RabbitMqConnection [4] - Exception disposing of RabbitMQ connection
RabbitMQ.Client.Exceptions.OperationInterruptedException: The AMQP operation was interrupted
at RabbitMQ.Client.Impl.ConnectionBase.System.IDisposable.Dispose()
at MassTransit.Transports.RabbitMq.RabbitMqConnection.Disconnect() in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqConnection.cs:line 67
2012-11-09 13:35:43,276 WARN MassTransit.Transports.RabbitMq.RabbitMqConnection [7] - Exception disposing of RabbitMQ connection
RabbitMQ.Client.Exceptions.OperationInterruptedException: The AMQP operation was interrupted
at RabbitMQ.Client.Impl.ConnectionBase.System.IDisposable.Dispose()
at MassTransit.Transports.RabbitMq.RabbitMqConnection.Disconnect() in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqConnection.cs:line 67
2012-11-09 13:35:44,283 DEBUG MassTransit.Transports.DefaultConnectionPolicy [7] - Re-connecting connection handler...
2012-11-09 13:35:44,283 DEBUG MassTransit.Transports.DefaultConnectionPolicy [4] - Re-connecting connection handler...
2012-11-09 13:35:45,850 INFO Topshelf.Runtime.Windows.WindowsServiceHost [5] - [Topshelf] Service is being shutdown
2012-11-09 13:35:45,850 INFO HandleShutdown.MyService [5] - Shutdown
2012-11-09 13:35:45,851 DEBUG MassTransit.Threading.ThreadPoolConsumerPool [5] - Stopping Consumer Pool for rabbitmq://localhost/HandleShutdown
2012-11-09 13:35:45,859 DEBUG MassTransit.Threading.ThreadPoolConsumerPool [5] - Consumer Pool stopped for rabbitmq://localhost/HandleShutdown
2012-11-09 13:35:46,520 WARN MassTransit.Transports.DefaultConnectionPolicy [7] - Failed to reconnect, deferring to connection policy for reconnection
2012-11-09 13:35:46,521 DEBUG MassTransit.Transports.DefaultConnectionPolicy [7] - Retrying callback after reconnect.
2012-11-09 13:35:46,521 WARN MassTransit.Transports.DefaultConnectionPolicy [4] - Failed to reconnect, deferring to connection policy for reconnection
2012-11-09 13:35:46,521 DEBUG MassTransit.Transports.DefaultConnectionPolicy [4] - Retrying callback after reconnect.
2012-11-09 13:35:46,521 ERROR MassTransit.Transports.RabbitMq.InboundRabbitMqTransport [7] - Failed to consume message from endpoint
System.NullReferenceException: Object reference not set to an instance of an object.
at MassTransit.Transports.RabbitMq.RabbitMqConsumer.Get(TimeSpan timeout) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqConsumer.cs:line 85
at MassTransit.Transports.RabbitMq.InboundRabbitMqTransport.<>c__DisplayClass1.<Receive>b__0(RabbitMqConnection connection) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\InboundRabbitMqTransport.cs:line 64
2012-11-09 13:35:46,521 ERROR MassTransit.Context.ServiceBusReceiveContext [7] - Consumer Exception Exposed
System.NullReferenceException: Object reference not set to an instance of an object.
at MassTransit.Transports.RabbitMq.RabbitMqConsumer.Get(TimeSpan timeout) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqConsumer.cs:line 85
at MassTransit.Transports.RabbitMq.InboundRabbitMqTransport.<>c__DisplayClass1.<Receive>b__0(RabbitMqConnection connection) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\InboundRabbitMqTransport.cs:line 123
at MassTransit.Transports.ConnectionHandlerImpl`1.<>c__DisplayClass3.<Use>b__2() in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\ConnectionHandlerImpl.cs:line 87
at MassTransit.Transports.DefaultConnectionPolicy.Execute(Action callback) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\DefaultConnectionPolicy.cs:line 64
at MassTransit.Transports.ConnectionPolicyChainImpl.Next(Action callback) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\ConnectionPolicyChainImpl.cs:line 49
at MassTransit.Transports.ConnectionHandlerImpl`1.Use(Action`1 callback) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\ConnectionHandlerImpl.cs:line 87
at MassTransit.Transports.RabbitMq.InboundRabbitMqTransport.Receive(Func`2 lookupSinkChain, TimeSpan timeout) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\InboundRabbitMqTransport.cs:line 62
at MassTransit.Transports.Transport.Receive(Func`2 callback, TimeSpan timeout) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\Transport.cs:line 59
at MassTransit.Transports.Endpoint.Receive(Func`2 receiver, TimeSpan timeout) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\Endpoint.cs:line 334
at MassTransit.Context.ServiceBusReceiveContext.ReceiveFromEndpoint() in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Context\ServiceBusReceiveContext.cs:line 91
2012-11-09 13:35:46,521 ERROR MassTransit.Transports.RabbitMq.InboundRabbitMqTransport [4] - Failed to consume message from endpoint
System.NullReferenceException: Object reference not set to an instance of an object.
at MassTransit.Transports.RabbitMq.RabbitMqConsumer.Get(TimeSpan timeout) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqConsumer.cs:line 85
at MassTransit.Transports.RabbitMq.InboundRabbitMqTransport.<>c__DisplayClass1.<Receive>b__0(RabbitMqConnection connection) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\InboundRabbitMqTransport.cs:line 64
2012-11-09 13:35:46,592 ERROR MassTransit.Context.ServiceBusReceiveContext [4] - Consumer Exception Exposed
System.NullReferenceException: Object reference not set to an instance of an object.
at MassTransit.Transports.RabbitMq.RabbitMqConsumer.Get(TimeSpan timeout) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqConsumer.cs:line 85
at MassTransit.Transports.RabbitMq.InboundRabbitMqTransport.<>c__DisplayClass1.<Receive>b__0(RabbitMqConnection connection) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\InboundRabbitMqTransport.cs:line 123
at MassTransit.Transports.ConnectionHandlerImpl`1.<>c__DisplayClass3.<Use>b__2() in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\ConnectionHandlerImpl.cs:line 87
at MassTransit.Transports.DefaultConnectionPolicy.Execute(Action callback) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\DefaultConnectionPolicy.cs:line 64
at MassTransit.Transports.ConnectionPolicyChainImpl.Next(Action callback) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\ConnectionPolicyChainImpl.cs:line 49
at MassTransit.Transports.ConnectionHandlerImpl`1.Use(Action`1 callback) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\ConnectionHandlerImpl.cs:line 87
at MassTransit.Transports.RabbitMq.InboundRabbitMqTransport.Receive(Func`2 lookupSinkChain, TimeSpan timeout) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\InboundRabbitMqTransport.cs:line 62
at MassTransit.Transports.Transport.Receive(Func`2 callback, TimeSpan timeout) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\Transport.cs:line 59
at MassTransit.Transports.Endpoint.Receive(Func`2 receiver, TimeSpan timeout) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\Endpoint.cs:line 334
at MassTransit.Context.ServiceBusReceiveContext.ReceiveFromEndpoint() in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Context\ServiceBusReceiveContext.cs:line 91
2012-11-09 13:35:46,592 DEBUG MassTransit.Threading.ThreadPoolConsumerPool [5] - Consumer Pool stopped for rabbitmq://localhost/HandleShutdown
2012-11-09 13:35:46,592 ERROR MassTransit.Transports.RabbitMq.RabbitMqConsumer [6] - Failed to close channel: rabbitmq://localhost/HandleShutdown_control
System.NullReferenceException: Object reference not set to an instance of an object.
at MassTransit.Transports.RabbitMq.RabbitMqConsumer.Unbind(RabbitMqConnection connection) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqConsumer.cs:line 60
2012-11-09 13:35:46,592 ERROR MassTransit.Transports.RabbitMq.RabbitMqConsumer [6] - Failed to dispose channel: rabbitmq://localhost/HandleShutdown_control
System.NullReferenceException: Object reference not set to an instance of an object.
at MassTransit.Transports.RabbitMq.RabbitMqConsumer.Unbind(RabbitMqConnection connection) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqConsumer.cs:line 70
2012-11-09 13:35:46,608 DEBUG MassTransit.Subscriptions.Coordinator.BusSubscriptionEventListener [5] - UnsubscribeFrom: MassTransit.Diagnostics.Tracing.GetMessageTraceList, MassTransit, 08cf8cd0-9c75-2b0e-f0de-f14f3a450000
2012-11-09 13:35:46,609 DEBUG MassTransit.Subscriptions.Coordinator.BusSubscriptionCache [5] - UnsubscribeFrom: MassTransit.Diagnostics.Tracing.GetMessageTraceList, MassTransit, 08cf8cd0-9c75-2b0e-f0de-f14f3a450000
2012-11-09 13:35:46,610 DEBUG MassTransit.Subscriptions.Coordinator.BusSubscription [5] - UnsubscribeFrom: MassTransit.Diagnostics.Tracing.GetMessageTraceList, MassTransit, 08cf8cd0-9c75-4768-f0de-f14f3a450000
2012-11-09 13:35:50,306 ERROR MassTransit.ServiceContainer [5] - Failed to stop service: MassTransit.Diagnostics.Tracing.MessageTraceBusService
None of the specified endpoints were reachable
Endpoints attempted:
------------------------------------------------
endpoint=amqp-0-9://localhost:5672, attempts=1
System.Net.Sockets.SocketException (0x80004005): No connection could be made because the target machine actively refused it 127.0.0.1:5672
at System.Net.Sockets.TcpClient.Connect(String hostname, Int32 port)
at RabbitMQ.Client.Impl.SocketFrameHandler_0_9..ctor(AmqpTcpEndpoint endpoint)
at RabbitMQ.Client.Framing.Impl.v0_9_1.ProtocolBase.CreateFrameHandler(AmqpTcpEndpoint endpoint)
at RabbitMQ.Client.ConnectionFactory.FollowRedirectChain(Int32 maxRedirects, IDictionary connectionAttempts, IDictionary connectionErrors, AmqpTcpEndpoint[]& mostRecentKnownHosts, AmqpTcpEndpoint endpoint)
================================================
Stack trace:
at RabbitMQ.Client.ConnectionFactory.CreateConnection(Int32 maxRedirects)
at RabbitMQ.Client.ConnectionFactory.CreateConnection()
at MassTransit.Transports.RabbitMq.Management.RabbitMqEndpointManagement..ctor(IRabbitMqEndpointAddress address) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\Management\RabbitMQEndpointManagement.cs:line 31
at MassTransit.Transports.RabbitMq.RabbitMqSubscriptionBinder.OnSubscriptionRemoved(SubscriptionRemoved message) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqSubscriptionBinder.cs:line 77
at MassTransit.Subscriptions.Coordinator.SubscriptionRouterService.<>c__DisplayClass10.<OnSubscriptionRemoved>b__e(SubscriptionObserver x) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Subscriptions\Coordinator\SubscriptionRouterService.cs:line 101
at Magnum.Extensions.ExtensionsToEnumerable.Each[T](IEnumerable`1 collection, Action`1 callback)
at MassTransit.Subscriptions.Coordinator.SubscriptionRouterService.OnSubscriptionRemoved(SubscriptionRemoved message) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Subscriptions\Coordinator\SubscriptionRouterService.cs:line 101
at MassTransit.Subscriptions.Coordinator.BusSubscription.OnUnsubscribeFrom(UnsubscribeFrom removed) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Subscriptions\Coordinator\BusSubscription.cs:line 117
at MassTransit.Subscriptions.Coordinator.BusSubscriptionCache.<>c__DisplayClass5.<OnUnsubscribeFrom>b__4(BusSubscription subscription) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Subscriptions\Coordinator\BusSubscriptionCache.cs:line 61
at Magnum.Caching.ConcurrentCache`2.WithValue(TKey key, Action`1 callback)
at MassTransit.Subscriptions.Coordinator.BusSubscriptionCache.OnUnsubscribeFrom(UnsubscribeFrom message) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Subscriptions\Coordinator\BusSubscriptionCache.cs:line 56
at MassTransit.Subscriptions.Coordinator.BusSubscriptionEventListener.Unsubscribe(Guid subscriptionId, String messageName, String correlationId) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Subscriptions\Coordinator\BusSubscriptionEventListener.cs:line 87
at MassTransit.Subscriptions.Coordinator.BusSubscriptionEventListener.<>c__DisplayClass2`1.<Subscribe>b__1() in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Subscriptions\Coordinator\BusSubscriptionEventListener.cs:line 71
at MassTransit.UnsubscribeAction.Invoke()
at MassTransit.Pipeline.Configuration.ExtensionsForPipelineConfiguration.<>c__DisplayClass4`1.<ConnectToRouter>b__3() in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Pipeline\Configuration\ExtensionsForPipelineConfiguration.cs:line 47
at MassTransit.SubscriptionConnectors.InstanceConnector`1.<>c__DisplayClassa.<Connect>b__5() in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Configuration\SubscriptionConnectors\InstanceConnector.cs:line 69
at MassTransit.Diagnostics.Tracing.MessageTraceBusService.Stop() in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Diagnostics\Tracing\MessageTraceBusService.cs:line 83
at MassTransit.ServiceContainer.Stop() in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\ServiceContainer.cs:line 94
2012-11-09 13:35:50,496 DEBUG MassTransit.Threading.ThreadPoolConsumerPool [5] - Stopping Consumer Pool for rabbitmq://localhost/HandleShutdown_control
2012-11-09 13:35:50,497 DEBUG MassTransit.Threading.ThreadPoolConsumerPool [5] - Consumer Pool stopped for rabbitmq://localhost/HandleShutdown_control
2012-11-09 13:35:51,307 ERROR MassTransit.Context.ServiceBusReceiveContext [6] - Consumer Exception Exposed
None of the specified endpoints were reachable
Endpoints attempted:
------------------------------------------------
endpoint=amqp-0-9://localhost:5672, attempts=1
System.Net.Sockets.SocketException (0x80004005): No connection could be made because the target machine actively refused it 127.0.0.1:5672
at System.Net.Sockets.TcpClient.Connect(String hostname, Int32 port)
at RabbitMQ.Client.Impl.SocketFrameHandler_0_9..ctor(AmqpTcpEndpoint endpoint)
at RabbitMQ.Client.Framing.Impl.v0_9_1.ProtocolBase.CreateFrameHandler(AmqpTcpEndpoint endpoint)
at RabbitMQ.Client.ConnectionFactory.FollowRedirectChain(Int32 maxRedirects, IDictionary connectionAttempts, IDictionary connectionErrors, AmqpTcpEndpoint[]& mostRecentKnownHosts, AmqpTcpEndpoint endpoint)
================================================
Stack trace:
at RabbitMQ.Client.ConnectionFactory.CreateConnection(Int32 maxRedirects)
at RabbitMQ.Client.ConnectionFactory.CreateConnection()
at MassTransit.Transports.RabbitMq.RabbitMqConnection.Connect() in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqConnection.cs:line 48
at MassTransit.Transports.ConnectionHandlerImpl`1.Connect() in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\ConnectionHandlerImpl.cs:line 48
at MassTransit.Transports.ReconnectPolicy.Execute(Action callback) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\ReconnectPolicy.cs:line 39
at MassTransit.Transports.ConnectionPolicyChainImpl.Next(Action callback) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\ConnectionPolicyChainImpl.cs:line 49
at MassTransit.Transports.ConnectionHandlerImpl`1.Use(Action`1 callback) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\ConnectionHandlerImpl.cs:line 87
at MassTransit.Transports.RabbitMq.InboundRabbitMqTransport.Receive(Func`2 lookupSinkChain, TimeSpan timeout) in d:\BuildAgent-03\work\8d1373c869590c5b\src\Transports\MassTransit.Transports.RabbitMq\InboundRabbitMqTransport.cs:line 62
at MassTransit.Transports.Transport.Receive(Func`2 callback, TimeSpan timeout) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\Transport.cs:line 59
at MassTransit.Transports.Endpoint.Receive(Func`2 receiver, TimeSpan timeout) in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Transports\Endpoint.cs:line 334
at MassTransit.Context.ServiceBusReceiveContext.ReceiveFromEndpoint() in d:\BuildAgent-03\work\8d1373c869590c5b\src\MassTransit\Context\ServiceBusReceiveContext.cs:line 91
2012-11-09 13:35:51,308 DEBUG MassTransit.Threading.ThreadPoolConsumerPool [5] - Consumer Pool stopped for rabbitmq://localhost/HandleShutdown_control
2012-11-09 13:35:51,321 INFO Topshelf.Runtime.Windows.WindowsServiceHost [5] - [Topshelf] Stopped
wolszakp
Sep 3, 2015, 9:02:52 AM9/3/15
to masstransit-discuss
I have the same issue.
Did you solve it somehow?
After server reboot my service is stopped because of this errors.
Thanks,
Peter
Did you solve it somehow?
After server reboot my service is stopped because of this errors.
Thanks,
Peter
Chris Patterson
Sep 3, 2015, 12:05:55 PM9/3/15
to masstrans...@googlegroups.com
If RabbitMQ is not ready at boot time, it may fail to start. You could add a service dependency on RabbitMQ if it's local, or set a service recovery policy to restart the service on failure.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/masstransit-discuss/-/_orW-O2wP_UJ.
>
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/c4613f1c-7a75-46bd-8b2c-07fa821c93bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
wolszakp
Sep 4, 2015, 3:00:10 AM9/4/15
to masstransit-discuss
Hi Chris,
Thank you for the quick reply.
I add dependency to RabbitMQ but it didn't help.
I think that problem can be with container in my example.
This is my configuration code:
var container = WindsorConfiguration.Configure();
HostFactory.Run(x =>
{
x.Service<ParserHost>(s =>
{
// Workeraund to start service with system reboot. Wait for RabbitMQ initialization.
Thread.Sleep(new TimeSpan(0, 0, 15));
s.ConstructUsing(name => container.Resolve<MyHost>());
s.WhenStarted(wq => wq.Start());
s.WhenStopped(wq =>
{
wq.Stop();
container.Dispose();
});
s.WhenShutdown(wq =>
{
// Shutdown is failling - RabbitMQ already close connection.
// MassTransit.Transports.RabbitMq.RabbitMqTransportFactory: Failed to close connection
wq.Shutdown();
container.Dispose();
});
});
x.EnableServiceRecovery(src =>
{
src.RestartService(5);
src.RestartService(5);
src.RestartService(5);
src.SetResetPeriod(1);
});
x.RunAsLocalService();
x.SetServiceName("MyServiceName");
x.DependsOn("RabbitMQ");
});
MyHost has dependency to IServiceBus.
I gues that ConstructUsing run imediately without waiting for a RabbitMQ service dependency.
My workaround - maybe not graceful - is to "Sleep" before call ConstructUsing.
I still got error during shutdown:
2015-09-04 08:41:27.1071 WARN MassTransit.Transports.DefaultConnectionPolicy: Invalid Connection when executing callback :
EXCEPTION: System.IO.EndOfStreamException: SharedQueue closed
at RabbitMQ.Util.SharedQueue`1.EnsureIsOpen()
at RabbitMQ.Util.SharedQueue`1.Dequeue(Int32 millisecondsTimeout, T& result)
at MassTransit.Transports.RabbitMq.RabbitMqConsumer.Get(TimeSpan timeout) in z:\Builds\work\aa063b4295dfc097\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqConsumer.cs:line 111
at MassTransit.Transports.RabbitMq.InboundRabbitMqTransport.<>c__DisplayClass1.<Receive>b__0(RabbitMqConnection connection) in z:\Builds\work\aa063b4295dfc097\src\Transports\MassTransit.Transports.RabbitMq\InboundRabbitMqTransport.cs:line 68
2015-09-04 08:41:27.1228 WARN MassTransit.Transports.RabbitMq.RabbitMqTransportFactory: Failed to close connection :
EXCEPTION: RabbitMQ.Client.Exceptions.AlreadyClosedException: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=320, text="CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'", classId=0, methodId=0, cause=
at RabbitMQ.Client.Framing.Impl.Connection.Close(ShutdownEventArgs reason, Boolean abort, Int32 timeout)
at RabbitMQ.Client.Framing.Impl.Connection.Close(UInt16 reasonCode, String reasonText)
at MassTransit.Transports.RabbitMq.RabbitMqExtensions.Cleanup(IConnection connection, UInt16 replyCode, String message) in z:\Builds\work\aa063b4295dfc097\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqExtensions.cs:line 66
2015-09-04 08:41:27.1228 WARN MassTransit.Transports.RabbitMq.RabbitMqTransportFactory: Failed to dispose connection :
EXCEPTION: System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at RabbitMQ.Client.Framing.Impl.Connection.System.IDisposable.Dispose()
at MassTransit.Transports.RabbitMq.RabbitMqExtensions.Cleanup(IConnection connection, UInt16 replyCode, String message) in z:\Builds\work\aa063b4295dfc097\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqExtensions.cs:line 75
Thanks,
Peter
Thank you for the quick reply.
I add dependency to RabbitMQ but it didn't help.
I think that problem can be with container in my example.
This is my configuration code:
var container = WindsorConfiguration.Configure();
HostFactory.Run(x =>
{
x.Service<ParserHost>(s =>
{
// Workeraund to start service with system reboot. Wait for RabbitMQ initialization.
Thread.Sleep(new TimeSpan(0, 0, 15));
s.ConstructUsing(name => container.Resolve<MyHost>());
s.WhenStarted(wq => wq.Start());
s.WhenStopped(wq =>
{
wq.Stop();
container.Dispose();
});
s.WhenShutdown(wq =>
{
// Shutdown is failling - RabbitMQ already close connection.
// MassTransit.Transports.RabbitMq.RabbitMqTransportFactory: Failed to close connection
wq.Shutdown();
container.Dispose();
});
});
x.EnableServiceRecovery(src =>
{
src.RestartService(5);
src.RestartService(5);
src.RestartService(5);
src.SetResetPeriod(1);
});
x.RunAsLocalService();
x.SetServiceName("MyServiceName");
x.DependsOn("RabbitMQ");
});
MyHost has dependency to IServiceBus.
I gues that ConstructUsing run imediately without waiting for a RabbitMQ service dependency.
My workaround - maybe not graceful - is to "Sleep" before call ConstructUsing.
I still got error during shutdown:
2015-09-04 08:41:27.1071 WARN MassTransit.Transports.DefaultConnectionPolicy: Invalid Connection when executing callback :
EXCEPTION: System.IO.EndOfStreamException: SharedQueue closed
at RabbitMQ.Util.SharedQueue`1.EnsureIsOpen()
at RabbitMQ.Util.SharedQueue`1.Dequeue(Int32 millisecondsTimeout, T& result)
at MassTransit.Transports.RabbitMq.RabbitMqConsumer.Get(TimeSpan timeout) in z:\Builds\work\aa063b4295dfc097\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqConsumer.cs:line 111
at MassTransit.Transports.RabbitMq.InboundRabbitMqTransport.<>c__DisplayClass1.<Receive>b__0(RabbitMqConnection connection) in z:\Builds\work\aa063b4295dfc097\src\Transports\MassTransit.Transports.RabbitMq\InboundRabbitMqTransport.cs:line 68
2015-09-04 08:41:27.1228 WARN MassTransit.Transports.RabbitMq.RabbitMqTransportFactory: Failed to close connection :
EXCEPTION: RabbitMQ.Client.Exceptions.AlreadyClosedException: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=320, text="CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'", classId=0, methodId=0, cause=
at RabbitMQ.Client.Framing.Impl.Connection.Close(ShutdownEventArgs reason, Boolean abort, Int32 timeout)
at RabbitMQ.Client.Framing.Impl.Connection.Close(UInt16 reasonCode, String reasonText)
at MassTransit.Transports.RabbitMq.RabbitMqExtensions.Cleanup(IConnection connection, UInt16 replyCode, String message) in z:\Builds\work\aa063b4295dfc097\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqExtensions.cs:line 66
2015-09-04 08:41:27.1228 WARN MassTransit.Transports.RabbitMq.RabbitMqTransportFactory: Failed to dispose connection :
EXCEPTION: System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at RabbitMQ.Client.Framing.Impl.Connection.System.IDisposable.Dispose()
at MassTransit.Transports.RabbitMq.RabbitMqExtensions.Cleanup(IConnection connection, UInt16 replyCode, String message) in z:\Builds\work\aa063b4295dfc097\src\Transports\MassTransit.Transports.RabbitMq\RabbitMqExtensions.cs:line 75
Thanks,
Peter
> To view this discussion on the web visit
> https://groups.google.com/d/msg/masstransit-discuss/-/_orW-O2wP_UJ.
>
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages