Spring AMQP/RabbitMQ - java.net.SocketException: Connection reset by peer (Write failed)

[Kai Wren]

Spring AMQP/RabbitMQ - java.net.SocketException: Connection reset by peer (Write failed)

I have a RabbitMQ server in a Kubernetes cluster that I am connecting to using Spring AMQP with default settings and RabbitTemplate class.

I have a scheduled method in a sender Spring Boot Microservice that sends a message to RabbitMQ queue using RabbitTemplate.convertAndSend(). The scheduled method is called every 15 seconds and the message is consumed by a receiver Spring Boot microservice and processed soon after it is received by the RabbitMQ server.

Generally, the connection between the RabbitMQ server and the sender Spring Boot Microservice is okay, but rarely (2 times in the last 6 months), the following exception is received and the microservice does not reconnect.

org.springframework.amqp.AmqpIOException: java.net.SocketException: Connection reset by peer (Write failed) at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:70) at org.springframework.amqp.rabbit.connection.RabbitAccessor.convertRabbitAccessException(RabbitAccessor.java:113) at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:2192) at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2138) at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:1065) at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:1130) at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:1123)

After this exception, the sender microservice gets the following broken pipe exception when trying to send messages to the RabbitMQ server.

org.springframework.amqp.AmqpIOException: java.net.SocketException: Broken pipe (Write failed) at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:70) at org.springframework.amqp.rabbit.connection.RabbitAccessor.convertRabbitAccessException(RabbitAccessor.java:113) at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:2192) at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2138) at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:1065) at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:1130) at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:1123)

The sender microservice has the following configured, aside from defaults:

# Don't requeue rejected deliveries # Messages can get requeued due to exceptions thrown on the consumer thread spring.rabbitmq.listener.simple.default-requeue-rejected = false # Change Spring RMQ listener mode to NONE from default AUTO # https://docs.spring.io/spring-amqp/reference/html/#containerAttributes spring.rabbitmq.listener.simple.acknowledge-mode = NONE

The sender microservice needs to be restarted to re-connect to the RabbitMQ server and to send messages successfully.

I haven't been able to replicate the issue. Is this just a network glitch that is not preventable?

Is there a way to force Spring AMQP to reconnect to the RabbitMQ server?

To fix the issue, I restarted the sender microservice and that reconnected to the RabbitMQ server.

I am using all default settings for the Spring AMQP connection, so I am not sure what could be causing the network issue.

I checked the connections between other microservices and the RabbitMQ server and they were not affected by this, even if they are subscribed to the same queue.

I made a StackOverflow post (https://stackoverflow.com/questions/76972339/spring-amqp-rabbitmq-java-net-socketexception-connection-reset-by-peer-write/76972448) and they referred me to here.

Thanks!