How to handle "broker unreachable" during bus.Start() as if it were a transient disconnect?

[Alejandro Villarreal]

Is there any way to tell MassTransit that if it can't reach the broker when calling Bus.Start(), it should treat it as it would a transient disconnect, and just keep trying to reach the broker in the background? Here the recommendation is to fail to start the application, but I think being able to choose the behavior in that scenario would be beneficial. In my environment, if RabbitMQ is unreachable it's most probably a transient issue, so I'd rather have my application start normally while the bus keeps trying to reach the broker until it succeeds. If messages are published before the bus can connect to the broker, I'm fine handling the individual exceptions (or if they can be queued somehow, even better), but I'd like the application to start anyway.

With MassTransit 4.0.0.1326-develop, this worked for my purpose:

try
{
  busHandle = bus.StartAsync().GetAwaiter().GetResult();
}
catch (RabbitMqConnectionException e)
{
  if (e.InnerException is BrokerUnreachableException)
  {
    // A second call will succeed, AND we need it to populate busHandle to be able to stop the bus later
    busHandle = bus.StartAsync().GetAwaiter().GetResult();
  }
  else
  {
    throw;
  }
}

// My application code ...

busHandle.Stop(TimeSpan.FromSeconds(30));

But with version 5.1.3 the second call to bus.StartAsync().GetAwaiter().GetResult() fails with MassTransit.MassTransitException: 'The host was already started: guest@localhost:5672/', from here.

I can find no other way of grabbing a busHandle from the IBusControl ('bus' in the code above), and I imagine that if I create a new instance of something that implements IBusControl (which was obtained from DI), the rest of my code will still be using the one from DI that failed to start...?

So to restate the question, is there any way to have MassTransit treat an unreachable broker during bus.Start() as a transient failure instead of leaving us no choice but to fail to start the application?

[Alejandro Villarreal]

Another motivation to be able to do this, is when running a docker-compose file with two containers, one for my application and one for RabbitMQ. Rabbit takes some time to start, so it's pretty much certain that it won't be ready before my application tries to connect to it.

[Chris Patterson]

Just reconfigure the bus each retry attempt, instead of only calling Start additional times.

On Wed, Jul 4, 2018 at 10:24 PM, Alejandro Villarreal <al...@alexvy86.com> wrote:

Another motivation to be able to do this, is when running a docker-compose file with two containers, one for my application and one for RabbitMQ. Rabbit takes some time to start, so it's pretty much certain that it won't be ready before my application tries to connect to it.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.
To post to this group, send email to masstransit-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/a840689e-a421-4f48-a2ca-b1dd4450833f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alejandro Villarreal]

Just reconfigure the bus each retry attempt, instead of only calling Start additional times.

2 questions:

• By "reconfigure the bus" do you mean calling Bus.Factory.CreateUsingRabbitMq() again? Is that possible if I got the bus from DI (see the code below)? I'm registering it as a SingleInstance, so the code that actually uses it to publish events ends up with the same instance as the one I start here...
  
• Does that mean the startup code has to loop until Rabbit is reachable, and only after it is, move on to executing the actual application code? I agree the application doesn't have to fail to start, but I was really hoping that it could just start doing its job while MassTransit tries to reach Rabbit in the background, just like with transient disconnects after the bus is correctly started.
  

Here's where I start the bus:

var bus = container.Resolve<IBusControl>();
BusHandle busHandle = null;

try
{
  busHandle = bus.StartAsync().GetAwaiter().GetResult();
}
catch (RabbitMqConnectionException e)

{
  // Retry logic
}

Here's where I register it in DI ("builder" is an AutoFac ContainerBuilder):

builder.Register(componentContext =>
{
  void ConfigureReceiveEndpoint(IReceiveEndpointConfigurator endpointConfigurator)
  {
    // Some code to configure endpoints
  }

  return Bus.Factory.CreateUsingRabbitMq(busFactoryConfigurator =>
  {
    var host = busFactoryConfigurator.Host(new Uri($"rabbitmq://localhost:5672/"), h =>
    {
      h.Username("guest");
      h.Password("guest");
    });


    busFactoryConfigurator.ReceiveEndpoint(host, ConfigureReceiveEndpoint);


    var loggerFactory = componentContext.Resolve<ILoggerFactory>();
    busFactoryConfigurator.UseExtensionsLogging(loggerFactory);
  });
})
.As<IBusControl>()
.As<IPublishEndpoint>()
.SingleInstance();

[Chris Patterson]

Yeah, typically we prefer the service not start if things are unavailable use the service restart features to restart the service, which gives visibility to operations that the service isn't starting due to RMQ not being available, instead of running in a Zombie mode where it is "running" but doing nothing.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.
To post to this group, send email to masstransit-discuss@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/17275a8b-7504-4dea-af79-4baddc780c77%40googlegroups.com.

[Alejandro Villarreal]

Would you be open to a pull request to let the user choose between the current behavior (throw a RabbitMqConnectionException that can't be recovered from unless you create a new instance of the bus) and treating it as a transient connectivity issue, so the application starts normally while MassTransit keeps trying to connect? If so, I'll try to see if it's possible. I'm really interested in making my services resilient to this kind of scenario.

[Chris Patterson]

Sure, if it's an opt-in thing to attempt to restart the connection, that might be useful.

--

You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.
To post to this group, send email to masstransit-discuss@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/4b293504-de60-4e44-b617-402d324c6b60%40googlegroups.com.

[Alejandro Villarreal]

Great, I'll try to allocate some time to look into this :)