Occasional connection reset by peer / rabbitmq crash when running docker based integration tests

[Bruce Adams]

Hi,

     I am working on a golang application which communicates with another application (in C++) using AMQP.

I have some integration tests which run the application in a docker container with rabbitmq:3-management (https://hub.docker.com/_/rabbitmq)  in another docker container.

As part of the test setup, I use the management API http:/server:15672/api/aliveness-test/%2F to check that the rabbitmq server is up

before starting either application.

Occasionally one or more of the tests fail with "connection reset by peer" after several successful communications between the two apps.

The applications logs show the apps losing connection at the same time.

The rabbitmq log ends abruptly suggesting something killed it.

Example logs:

go app:

            2020-03-04T16:51:44Z: info: AMQP connection  closed: Exception (501) Reason: "read tcp 172.20.0.5:53008->172.20.0.2:5672: read: connection reset by peer"
            2020-03-04T16:51:44Z: info: Retrying AMQP connection (time since first failure 0.614410s): dial tcp 172.20.0.2:5672: connect: connection refused
            2020-03-04T16:51:44Z: info: Retrying AMQP connection (time since first failure 0.846827s): dial tcp 172.20.0.2:5672: connect: connection refused

C++ app:
            2020-03-04T16:51:44Z: warning: Connection reset by peer
            2020-03-04T16:51:44Z: warning: I/O failure while communicating with amqp://guest:XX...@172.20.0.2:5672
            2020-03-04T16:51:44Z: warning: server closed connection to amqp://guest:XX...@172.20.0.2:5672
            2020-03-04T16:51:44Z: warning: Connection reset by peer
            2020-03-04T16:51:44Z: warning: I/O failure while communicating with amqp://guest:XX...@172.20.0.2:5672
            2020-03-04T16:51:44Z: warning: server closed connection to amqp://guest:XX...@172.20.0.2:5672

rabbitmq log:
            2020-03-04 16:51:30.136 [info] <0.8.0> Server startup complete; 3 plugins started.
             * rabbitmq_management
             * rabbitmq_management_agent
             * rabbitmq_web_dispatch
             completed with 3 plugins.
            2020-03-04 16:51:33.443 [info] <0.699.0> accepting AMQP connection <0.699.0> (172.20.0.1:59950 -> 172.20.0.2:5672)
            2020-03-04 16:51:33.445 [info] <0.699.0> connection <0.699.0> (172.20.0.1:59950 -> 172.20.0.2:5672): user 'guest' authenticated and granted access to vhost '/'
            2020-03-04 16:51:35.312 [info] <0.718.0> accepting AMQP connection <0.718.0> (172.20.0.4:35936 -> 172.20.0.2:5672)
            2020-03-04 16:51:35.315 [info] <0.718.0> connection <0.718.0> (172.20.0.4:35936 -> 172.20.0.2:5672): user 'guest' authenticated and granted access to vhost '/'
            2020-03-04 16:51:36.590 [info] <0.737.0> accepting AMQP connection <0.737.0> (172.20.0.5:53008 -> 172.20.0.2:5672)
            2020-03-04 16:51:36.593 [info] <0.737.0> connection <0.737.0> (172.20.0.5:53008 -> 172.20.0.2:5672): user 'guest' authenticated and granted access to vhost '/'
            2020-03-04 16:51:37.138 [info] <0.699.0> closing AMQP connection <0.699.0> (172.20.0.1:59950 -> 172.20.0.2:5672, vhost: '/', user: 'guest')
            2020-03-04 16:51:43.802 [info] <0.768.0> accepting AMQP connection <0.768.0> (172.20.0.4:35974 -> 172.20.0.2:5672)
            2020-03-04 16:51:43.804 [info] <0.768.0> connection <0.768.0> (172.20.0.4:35974 -> 172.20.0.2:5672): user 'guest' authenticated and granted access to vhost '/'

Do you have any suggestions on how to diagnose, avoid and fix this?

Diagnosis is made hard by the fact that this typically happens when the test is run under jenkins.

Regards,

Bruce.

[Wesley Peng]

Hi

user guest only works for localhost by default.

Regards 

5 березня 2020 р., 01:30:57,від "Bruce Adams" <torto...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/661b875d-59ac-4b23-a710-57a5d622e31d%40googlegroups.com.

[Bruce Adams]

Hi,

    This is an intermittant issue. If it was a permissions issue it should either always work or never work.

The docker container must allow remote access to be useful and some messages are passed so I remote guest access is enabled.

It is possible to change the default user and password. I will do this because it is good practice but I don't think it will help.

If you wish to change the default username and password of guest / guest, you can do so with the RABBITMQ_DEFAULT_USER and RABBITMQ_DEFAULT_PASS environmental variables:

$ docker run -d --hostname my-rabbit --name some-rabbit -e RABBITMQ_DEFAULT_USER=user -e RABBITMQ_DEFAULT_PASS=password rabbitmq:3-management

See https://hub.docker.com/_/rabbitmq

Regards,

Bruce.

On Wednesday, 4 March 2020 20:39:34 UTC, Wesley Peng wrote:

Hi

user guest only works for localhost by default.

Regards 

5 березня 2020 р., 01:30:57,від "Bruce Adams" <torto...@gmail.com>:

Hi,

     I am working on a golang application which communicates with another application (in C++) using AMQP.

I have some integration tests which run the application in a docker container with rabbitmq:3-management (https://hub.docker.com/_/rabbitmq)  in another docker container.

As part of the test setup, I use the management API http:/server:15672/api/aliveness-test/%2F to check that the rabbitmq server is up

before starting either application.

Occasionally one or more of the tests fail with "connection reset by peer" after several successful communications between the two apps.

The applications logs show the apps losing connection at the same time.

The rabbitmq log ends abruptly suggesting something killed it.

Example logs:

go app:

            2020-03-04T16:51:44Z: info: AMQP connection  closed: Exception (501) Reason: "read tcp 172.20.0.5:53008->172.20.0.2:5672: read: connection reset by peer"
            2020-03-04T16:51:44Z: info: Retrying AMQP connection (time since first failure 0.614410s): dial tcp 172.20.0.2:5672: connect: connection refused
            2020-03-04T16:51:44Z: info: Retrying AMQP connection (time since first failure 0.846827s): dial tcp 172.20.0.2:5672: connect: connection refused

C++ app:
            2020-03-04T16:51:44Z: warning: Connection reset by peer

            2020-03-04T16:51:44Z: warning: I/O failure while communicating with amqp://guest...@172.20.0.2:5672
            2020-03-04T16:51:44Z: warning: server closed connection to amqp://guest...@172.20.0.2:5672

            2020-03-04T16:51:44Z: warning: Connection reset by peer

            2020-03-04T16:51:44Z: warning: I/O failure while communicating with amqp://guest...@172.20.0.2:5672
            2020-03-04T16:51:44Z: warning: server closed connection to amqp://guest...@172.20.0.2:5672

rabbitmq log:
            2020-03-04 16:51:30.136 [info] <0.8.0> Server startup complete; 3 plugins started.
             * rabbitmq_management
             * rabbitmq_management_agent
             * rabbitmq_web_dispatch
             completed with 3 plugins.
            2020-03-04 16:51:33.443 [info] <0.699.0> accepting AMQP connection <0.699.0> (172.20.0.1:59950 -> 172.20.0.2:5672)
            2020-03-04 16:51:33.445 [info] <0.699.0> connection <0.699.0> (172.20.0.1:59950 -> 172.20.0.2:5672): user 'guest' authenticated and granted access to vhost '/'
            2020-03-04 16:51:35.312 [info] <0.718.0> accepting AMQP connection <0.718.0> (172.20.0.4:35936 -> 172.20.0.2:5672)
            2020-03-04 16:51:35.315 [info] <0.718.0> connection <0.718.0> (172.20.0.4:35936 -> 172.20.0.2:5672): user 'guest' authenticated and granted access to vhost '/'
            2020-03-04 16:51:36.590 [info] <0.737.0> accepting AMQP connection <0.737.0> (172.20.0.5:53008 -> 172.20.0.2:5672)
            2020-03-04 16:51:36.593 [info] <0.737.0> connection <0.737.0> (172.20.0.5:53008 -> 172.20.0.2:5672): user 'guest' authenticated and granted access to vhost '/'
            2020-03-04 16:51:37.138 [info] <0.699.0> closing AMQP connection <0.699.0> (172.20.0.1:59950 -> 172.20.0.2:5672, vhost: '/', user: 'guest')
            2020-03-04 16:51:43.802 [info] <0.768.0> accepting AMQP connection <0.768.0> (172.20.0.4:35974 -> 172.20.0.2:5672)
            2020-03-04 16:51:43.804 [info] <0.768.0> connection <0.768.0> (172.20.0.4:35974 -> 172.20.0.2:5672): user 'guest' authenticated and granted access to vhost '/'

Do you have any suggestions on how to diagnose, avoid and fix this?

Diagnosis is made hard by the fact that this typically happens when the test is run under jenkins.

Regards,

Bruce.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

[Wesley Peng]

Have you checked systems log? either of network issue, filter problem, IO issue, CPU limitation will affect the connection success, thanks.

6 березня 2020 р., 02:32:43,від "Bruce Adams" <torto...@gmail.com>:

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/614c27a1-8d02-40db-815e-e48d72342db9%40googlegroups.com.