client unexpectedly closed TCP connection

[Chris Pretorius]

Good day

Since last night I Rabbitmq started logging the following in the logs.

2020-06-16 12:15:48.578 [info] <0.3040.0> accepting AMQP connection <0.3040.0> (172.16.20.:53346 -> 172.16.20.:5672)

2020-06-16 12:15:54.582 [warning] <0.3040.0> closing AMQP connection <0.3040.0> (172.16.20.:53346 -> 172.16.20.:5672):

client unexpectedly closed TCP connection

The apps connect to rabbitmq all fail to start up and throws rabbitmq errors in the stacktraces.

I tried to add a new user for the apps to connect with but still receive the error in the logs.

I also tried to restart rabbitmq-server and even did a reboot of the server

Kind regards

[Wesley Peng]

This is most likely a network issue. Sever IO busy, VM in snapshot, network storage issue, or network filter can cause the problem.

Thanks 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/2af3220d-744f-4b56-a82d-3338bb361410o%40googlegroups.com.

[Chris Pretorius]

Hi

Thank you for responding.

Server has no IO, its a stage server and it a public holiday by us, no one is using the server.

I couldnt find snapshots for the VM

Telnet tests from app servers are ok, what other test can I run?

Regards

On Tuesday, June 16, 2020 at 1:01:59 PM UTC+2, Wesley Peng wrote:

This is most likely a network issue. Sever IO busy, VM in snapshot, network storage issue, or network filter can cause the problem.

Thanks 

On Tue, Jun 16, 2020 at 6:30 PM Chris Pretorius <pret...@gmail.com> wrote:

Good day

Since last night I Rabbitmq started logging the following in the logs.

2020-06-16 12:15:48.578 [info] <0.3040.0> accepting AMQP connection <0.3040.0> (172.16.20.:53346 -> 172.16.20.:5672)

2020-06-16 12:15:54.582 [warning] <0.3040.0> closing AMQP connection <0.3040.0> (172.16.20.:53346 -> 172.16.20.:5672):

client unexpectedly closed TCP connection

The apps connect to rabbitmq all fail to start up and throws rabbitmq errors in the stacktraces.

I tried to add a new user for the apps to connect with but still receive the error in the logs.

I also tried to restart rabbitmq-server and even did a reboot of the server

Kind regards

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

[Michael Klishin]

See [1][2][3] and connection recovery docs for the client library used.

If a proxy or load balancer is involved, also [4].

1. https://www.rabbitmq.com/logging.html#logged-events

2. https://www.rabbitmq.com/troubleshooting-networking.html

3. https://www.rabbitmq.com/heartbeats.html

4. https://www.rabbitmq.com/heartbeats.html#tcp-proxies

[Chris Pretorius]

I found the following in the java app logs

2020-06-16 16:45:37,030  WARN [localhost-startStop-1] ConnectionConfigurationImpl  - connection to RabbitMQ host 'rabbitmq-stage.:5672' on virtual host '/' had no name assigned; assigning name '0a28b

b19-ed61-412f-9005-a20660f36b38'

2020-06-16 16:45:37,110  INFO [localhost-startStop-1] ConnectionContextImpl  - connecting to RabbitMQ server '0a28bb19-ed61-412f-9005-a20660f36b38' at 'rabbitmq-stage.:5672' on virtual host '/'

2020-06-16 16:45:42,127 ERROR [localhost-startStop-1] GrailsContextLoader  - Error initializing the application: java.util.concurrent.TimeoutException

org.codehaus.groovy.runtime.InvokerInvocationException: java.util.concurrent.TimeoutException

I recrated the user, this is from the rabbt logs:

The rabbitmq web show an active connection.

2020-06-16 16:43:05.828 [info] <0.3595.1> Deleting user 'sservices'

2020-06-16 16:43:05.833 [info] <0.403.0> Closing all connections from user 'sservices' because it's being deleted

2020-06-16 16:44:15.951 [info] <0.3757.1> Creating user 'sservices'

2020-06-16 16:44:15.960 [info] <0.3757.1> Setting user tags for user 'sservices' to [management]

2020-06-16 16:44:22.367 [info] <0.3780.1> Setting permissions for 'sservices' in '/' to '.*', '.*', '.*'

2020-06-16 16:45:29.789 [info] <0.3929.1> accepting AMQP connection <0.3929.1> (172.16.20.:50600 -> 172.16.20.:5672)

2020-06-16 16:45:34.792 [info] <0.3939.1> accepting AMQP connection <0.3939.1> (172.16.20.:50602 -> 172.16.20.:5672)

2020-06-16 16:45:35.788 [warning] <0.3929.1> closing AMQP connection <0.3929.1> (172.16.20.:50600 -> 172.16.20.:5672):

client unexpectedly closed TCP connection

2020-06-16 16:45:35.815 [info] <0.3939.1> connection <0.3939.1> (172.16.20.58:50602 -> 172.16.20.:5672): user 'sservices' authenticated and granted access to vhost '/'

2020-06-16 16:45:37.119 [info] <0.5417.1> accepting AMQP connection <0.5417.1> (172.16.20.:50604 -> 172.16.20.:5672)

2020-06-16 16:45:43.117 [warning] <0.5417.1> closing AMQP connection <0.5417.1> (172.16.20.:50604 -> 172.16.20.:5672):

Regards

[Michael Klishin]

It's not clear where does the the timeout exception comes from.

However, an unhandled exception in the main JVM thread will terminate it, which naturally closes its TCP connection abruptly (as far as the server is concerned).

The root cause is very likely in your app's code.

[Wesley Peng]

Does your rabbitmq server have correct name resolution?

Regards 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/dee7970e-00e7-4b61-a1bc-cff51c2b0f12o%40googlegroups.com.

[Chris Pretorius]

Hi

Did a couple tcpdumps and found that the rabbit server does DNS lookups on the servers configured in /etc/resolve for the hostname on the rabbit server.

The lookup fails since the DNS server is not configured to give an answer for the local server hostnames. The hostname is configured in /etc/hosts but rabbit seems to have its own dns resolver setup.  

http://erlang.org/doc/apps/erts/inet_cfg.html

Installed and configured dnsmasq and the problem was resolved. All connection requests to rabbit is successful now.

I guess adding ERL_INET_ETC_DIR to the rabbit-env.conf may help as well.

What should the syntax be?

Regards

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

[Wesley Peng]

Hi

Both the long hostname and short hostname should be configured correctly, either in /etc/hosts or via DNS resolution. For me I never need a local dnsmask server to keep rabbitmq running.

Thanks 

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/c6146f4a-fb48-418d-8c58-9821c5ec3353o%40googlegroups.com.