rabbitmq server closed with error message {writer,send_failed,{error,timeout}}

[Rui]

Hi, 

I am running an application using rabbitmq (3.6.3)+celery. For large number of tasks the rabbitmq server is always closed during the calculation with the following error message:

=ERROR REPORT==== 30-Jul-2016::23:33:41 ===

closing AMQP connection <0.2130.0> (10.9.17.3:44142 -> 10.9.17.3:5672):

{writer,send_failed,{error,timeout}}

...

=WARNING REPORT==== 30-Jul-2016::23:33:57 ===

Mnesia(rabbit@r1155): ** WARNING ** Mnesia is overloaded: {dump_log,write_threshold}

...

=WARNING REPORT==== 30-Jul-2016::23:35:01 ===

closing AMQP connection <0.27202.1> (10.9.17.3:51214 -> 10.9.17.3:5672):

client unexpectedly closed TCP connection

...

=INFO REPORT==== 30-Jul-2016::23:35:12 ===

Stopping RabbitMQ

=INFO REPORT==== 30-Jul-2016::23:35:12 ===

stopped TCP Listener on [::]:5672

I have changed the heartbeat timeout length but it didn't fix the problem. 

Please let me know if there is any other config. I need to tune to fix it? 

Any suggestions are highly appreciated. Thank you.

Regards,

Rui

[Michael Klishin]

The log says a connection.close frame wasn't delivered due a timeout, then client's TCP connection was lost. Those things happen.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rui]

Hi Michael,

Many thanks for your reply. I see the connection wasn't closed correctly. Is there any configuration in rabbitmq I can tune to address this issue? 

Thanks.

Regards,

Rui

[Gabriele Santomaggio]

Il giorno lunedì 1 agosto 2016 13:52:47 UTC+2, Rui ha scritto:

Hi Michael,

Many thanks for your reply. I see the connection wasn't closed correctly. Is there any configuration in rabbitmq I can tune to address this issue? 

To solve the mnesia warning you can add this:

mnesia, [{dump_log_write_threshold, 1000}]},

to the rabbitmq.config file.

You can tune your RabbitMQ and your OS following these resources:
https://www.rabbitmq.com/production-checklist.html
and
https://www.rabbitmq.com/networking.html

In particular:

1.  ulimit parameters
   
2. Tuning for Throughput
3. Erlang VM I/O Thread Pool

-
Gabriele

[Rad W]

I had was getting a timeout error every few seconds or so. Restarting rabbit didn't work so I restarted the whole server and that did the trick on CentOS7. The server ran out of disk space and some services I guess were crippled. Networking seemed fine though, the client kept reporting that the server closed the connection with code 0 and reason text "Unspecified" (using pika 0.10.0). The server, on the other hand, kept giving me timeout errors.  Thankfully this was in our staging environment.

[Ilya Smirnov]

Hi,

we received such messages({writer,send_failed,{error,timeout}}) when someone published large message and server did not have time to give it to the consumer in 30 seconds (apparently the internal timeout for transmission). Maybe it can help.

[Ilya Smirnov]

Аnd in continuation to the described problem (with large msg):

Maybe somebody know, why the frame size during transmission differs, and is always significantly less than the maximum?  { frame_max, 131072 }

Can we somehow affect this?

Thanks!

[Michael Klishin]

rabbit.frame_max controls AMQP 0-9-1 framing for basic.publish and similar methods that carry a message payload.

What you are looking at in Wireshark is almost always TCP segment size, which is a different thing and is not controlled

by RabbitMQ.

See https://groups.google.com/d/msg/rabbitmq-users/slrqQLbb4r0/Z7EN0SI_AQAJ (which

is a follow-up to https://github.com/rabbitmq/rabbitmq-dotnet-client/pull/364).

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ