RabbitMQ publish fails with "No route to host"

[Vamsidhar Yeddu]

I have a rabbitmq message publisher. It checks if a connection(com.rabbitmq.client.Connection) is active and creates one(via com.rabbitmq.client.ConnectionFactory) if none exists and then creates a channel(com.rabbitmq.client.Channel) in that connection via which it will publish message(basicPublish). createChannel() function fails intermittently with the following backtrace:

java.io.IOException

        at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106) ~[amqp-client-3.5.2.jar:?]

        at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102) ~[amqp-client-3.5.2.jar:?]

        at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124) ~[amqp-client-3.5.2.jar:?]

        at com.rabbitmq.client.impl.ChannelN.open(ChannelN.java:125) ~[amqp-client-3.5.2.jar:?]

        at com.rabbitmq.client.impl.ChannelManager.createChannel(ChannelManager.java:134) ~[amqp-client-3.5.2.jar:?]

        at com.rabbitmq.client.impl.AMQConnection.createChannel(AMQConnection.java:499) ~[amqp-client-3.5.2.jar:?]

        at com.limeroad.services.rabbitmq.RabbitMQPublisher.publishMessage(RabbitMQPublisher.java:68) ~[utils-1.0.jar:?]

Caused by: com.rabbitmq.client.ShutdownSignalException: connection error

        at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67) ~[amqp-client-3.5.2.jar:?]

        at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33) ~[amqp-client-3.5.2.jar:?]

        at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:348) ~[amqp-client-3.5.2.jar:?]

        at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:221) ~[amqp-client-3.5.2.jar:?]

        at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118) ~[amqp-client-3.5.2.jar:?]

        ... 68 more

Caused by: java.net.SocketException: No route to host

        at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.7.0_75]

        at java.net.SocketInputStream.read(SocketInputStream.java:152) ~[?:1.7.0_75]

        at java.net.SocketInputStream.read(SocketInputStream.java:122) ~[?:1.7.0_75]

        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) ~[?:1.7.0_75]

        at java.io.BufferedInputStream.read(BufferedInputStream.java:254) ~[?:1.7.0_75]

        at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:288) ~[?:1.7.0_75]

        at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95) ~[amqp-client-3.5.2.jar:?]

        at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:139) ~[amqp-client-3.5.2.jar:?]

        at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:534) ~[amqp-client-3.5.2.jar:?]

        ... 1 more

I am not sure why this is happening. Can anybody guide me in the right direction to debug this further and solve this issue.

[Arnaud Cogoluègnes]

There's obviously a network problem somewhere, can you tell us more about your network topology (connections between the client and the server, any load balancer in between, etc)?

You're using Java Client 3.5.2, I guess the system was working fine for a decent period of time and you're now experiencing issues. Can you correlate those issues with some changes in your system?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

> Caused by: com.rabbitmq.client.ShutdownSignalException: connection error

> Caused by: java.net.SocketException: No route to host

are not entirely unambiguous but give a clue: this is a network connectivity problem
and according to the JVM, the host this application was running on couldn't route
IP packets to the server host.

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users"
> group.

> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send an email to rabbitm...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Vamsidhar Yeddu]

I have 3 rabbitmq nodes on 3 ec2 instances running in cluster mode. These 3 instances are under a load balancer(internal). ConnectionFactory connects directly to the loadbalancer. Publishing happens from many other ec2 instances(all internal).

You're using Java Client 3.5.2, I guess the system was working fine for a decent period of time and you're now experiencing issues. Can you correlate those issues with some changes in your system?

This is an intermittent issue. It occurs approximately once every 10-15 days.

 

This is the code that I am using for publishing:

public class RabbitMQPublisher implements ConfChangeCallback {


 private static ConnectionFactory factory = new ConnectionFactory();
 private static Connection connection;
 private static Logger logger = LoggerHandler.INSTANCE.getNewLogger(RabbitMQPublisher.class);
 private static int RETRY_COUNT = 3;
 private static long ACK_TIMEOUT = 60000L;
 private static RabbitMQPublisher rabbit =  new RabbitMQPublisher();
 static {
 initializeConnectionFactory();
 }


 private static void initializeConnectionFactory() {
 if(factory == null){
 factory = new ConnectionFactory();
 }
 System.out.println("initializing connection factory for RabbitMQ");
 factory.setHost(System.getProperty("backendHostName"));//connects to loadbalancer
 factory.setUsername(System.getProperty("backendRabbitmqUsername"));
 factory.setPassword(System.getProperty("backendRabbitmqPassword"));
 
 ConfManager.getInstance().subscribe("backendHostName",rabbit );
 ConfManager.getInstance().subscribe("backendRabbitmqUsername",rabbit );
 ConfManager.getInstance().subscribe("backendRabbitmqPassword",rabbit );
 
 }


 private static void setConnection() throws Exception {
 if(factory == null)
 initializeConnectionFactory();
 
 if(connection == null || !connection.isOpen()) {
 connection = factory.newConnection();
 }
 }
 @SuppressWarnings({ "rawtypes", "unchecked" })
 public static void publishMessage(String message, String contentType, String exchangeName, String exchangeType, String routingKey, String DLExchange) throws Exception {
 int counter = 1;
 boolean status = true;
 Channel channel = null;
 Map arguments = new HashMap();
 if(DLExchange != null) {
 arguments.put("x-dead-letter-exchange", DLExchange);
 }
 BasicProperties messageProperties = new BasicProperties(contentType, null, null, 2, 0, null, null, null, null, null, null, null, null, null);
 while (counter <= RETRY_COUNT) {
 int retriesLeft = RETRY_COUNT - counter;
 try {
 setConnection();
 channel = connection.createChannel();


 //durable true
 channel.exchangeDeclare(exchangeName, exchangeType, true);


 //enables ack
 channel.confirmSelect();
 status = true;
 channel.basicPublish(exchangeName, routingKey, messageProperties, message.getBytes());
 channel.waitForConfirmsOrDie(ACK_TIMEOUT);
 } catch (Exception e2) {
 logger.error("Publishing failed for message: " + message + " in the exchange" + exchangeName + ", Number of retries left: " + retriesLeft,e2);
 counter++;
 status = false;
 } finally {
 if(status) {
 logger.info("Message( " + message +  ") sent to exchange " + exchangeName + ".");
 break;
 }
 }
 }
 try {
 if(channel != null) {
 channel.close();
 }
 } catch (Exception e3) {
 logger.error("Exception in closing channel",e3);
 }
 //throw error if message is not sent
 if(!status) {
 throw new Exception("Unable to publish message " + message);
 }
 }


 @Override
 public void onChange(String key) {
 logger.info("rabbit mq onchange for key= "+key+" called.");
 initializeConnectionFactory();
 logger.info("rabbit mq onchange for key= "+key+" done.");
 }
}

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

[Michael Klishin]

On 7 December 2016 at 14:19:57, Vamsidhar Yeddu (vams...@limeroad.com) wrote:
> This is an intermittent issue. It occurs approximately once
> every 10-15 days.

What do you expect the client library to do? There should be a connection recovery triggered by this
since it is clearly a non-application triggered connection shutdown.

[Vamsidhar Yeddu]

Thanks, I will catch such exceptions and try to recover the message. Is there anything else that I can do to avoid such issues by setting some properties for eg SimpleMessageListenerContainer(org.springframework.amqp.rabbit.listener) or the conf files of rabbitmq nodes?  

Also are there any common known problems which cause one ec2 instance to be unavailable to another instance?(I'm not sure if I should ask this here)

[Michael Klishin]

It's an IP packet routing problem. You can't fix it by configuring Spring-AMQP or RabbitMQ.

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

> To post to this group, send an email to rabbitm...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>

[springusr]

you may want to use latest client library version 4.0, it automatically recovers this type of situation..we do have something similar where our rabbit mq server is down and we can see below in logs, that it is keep trying..

2016-12-07T08:15:28.93-0600 [App/0]      OUT Caused by: com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; protocol method: #method<connection.close>(reply-code=320, reply-text=CONNECTION_FORCED - broker forced connection closure with reason '

 class-id=0, method-id=0)

2016-12-07T08:15:33.69-0600 [App/0]      OUT    at com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:62)

2016-12-07T08:15:33.69-0600 [App/0]      OUT    at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.recoverConnection(AutorecoveringConnection.java:531)