oslo-messaging report a lot of error: [Errno 32] Broken pipe，why and what it means?

[cche...@gmail.com]

hello all， I run a queen version openstack with python2-oslo-messaging-5.35.5, rabbitmq 3.8.7, centos7. After a ceph cluster connection error, I find a lot of [Errno 32] Broken pipe show below:

2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager     with RADOSClient(self) as client:

2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 102, in __init__

2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager     self.cluster, self.ioctx = driver._connect_to_rados(pool)

2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 133, in _connect_to_rados

2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager     client.connect()

2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager   File "rados.pyx", line 893, in rados.Rados.connect (/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8/rpm/el7/BUILD/ceph-14.2.8/build/src/pybind/rados/pyrex/rados.c:9834)

2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager PermissionError: [errno 1] error connecting to the cluster

2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager

2021-08-10 20:43:56.007 19626 ERROR oslo.messaging._drivers.impl_rabbit [-] [cef69dca-f065-4b5e-8129-b42032d23717] AMQP server on 10.39.176.17:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe

2021-08-10 20:43:56.008 19626 ERROR oslo.messaging._drivers.impl_rabbit [-] [cf287c2d-0135-405a-9d83-784fc3cc1729] AMQP server on 10.39.176.9:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe

2021-08-10 20:43:56.008 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe

2021-08-10 20:43:56.010 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe

2021-08-10 20:43:56.010 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe

2021-08-10 20:43:56.011 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe

2021-08-10 20:43:56.011 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe

2021-08-10 20:43:56.011 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe

2021-08-10 20:43:56.012 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe

2021-08-10 20:43:56.012 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe

2021-08-10 20:43:56.012 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe

2021-08-10 20:43:56.017 19626 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8a95348-8155-49d8-bc77-9d7c466cbea9] AMQP server on 10.39.176.2:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe

2021-08-10 20:43:56.018 19626 ERROR oslo.messaging._drivers.impl_rabbit [req-27e11dbc-6beb-4543-8768-df7a35fb7ed2 - - - - -] [d4886cfb-5645-409a-86b0-cf8702bedcd1] AMQP server on 10.39.176.9:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe

2021-08-10 20:43:57.041 19626 INFO oslo.messaging._drivers.impl_rabbit [-] [b8a95348-8155-49d8-bc77-9d7c466cbea9] Reconnected to AMQP server on 10.39.176.2:5672 via [amqp] client with port 42738.

2021-08-10 20:43:57.045 19626 INFO oslo.messaging._drivers.impl_rabbit [req-27e11dbc-6beb-4543-8768-df7a35fb7ed2 - - - - -] [d4886cfb-5645-409a-86b0-cf8702bedcd1] Reconnected to AMQP server on 10.39.176.9:5672 via [amqp] client with port 49842.

2021-08-10 20:43:57.047 19626 INFO oslo.messaging._drivers.impl_rabbit [-] [cf287c2d-0135-405a-9d83-784fc3cc1729] Reconnected to AMQP server on 10.39.176.9:5672 via [amqp] client with port 49838.

2021-08-10 20:43:57.084 19626 INFO oslo.messaging._drivers.impl_rabbit [-] [cef69dca-f065-4b5e-8129-b42032d23717] Reconnected to AMQP server on 10.39.176.17:5672 via [amqp] client with port 40620.

2021-08-10 20:44:18.665 19626 ERROR oslo.messaging._drivers.impl_rabbit [-] [29e26e71-50d9-4049-a22f-082b0a9e4b30] AMQP server on 10.39.176.2:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe

2021-08-10 20:44:18.676 19626 ERROR oslo.messaging._drivers.impl_rabbit [-] [b74ce5ce-1864-46b1-b4eb-a509e64666bc] AMQP server on 10.39.176.2:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe

2021-08-10 20:44:18.679 19626 ERROR oslo.messaging._drivers.impl_rabbit [-] [20efaa3c-79ef-4e45-b83c-91bda815f549] AMQP server on 10.39.176.17:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe

2021-08-10 20:44:18.691 19626 ERROR oslo.messaging._drivers.impl_rabbit [-] [204ff30c-cd50-4224-905d-e251e0d0c5cf] AMQP server on 10.39.176.9:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe

2021-08-10 20:44:20.900 19626 INFO oslo.messaging._drivers.impl_rabbit [-] [204ff30c-cd50-4224-905d-e251e0d0c5cf] Reconnected to AMQP server on 10.39.176.9:5672 via [amqp] client with port 50416.

2021-08-10 20:44:20.900 19626 INFO oslo.messaging._drivers.impl_rabbit [-] [b74ce5ce-1864-46b1-b4eb-a509e64666bc] Reconnected to AMQP server on 10.39.176.2:5672 via [amqp] client with port 43308.

2021-08-10 20:44:20.901 19626 INFO oslo.messaging._drivers.impl_rabbit [-] [29e26e71-50d9-4049-a22f-082b0a9e4b30] Reconnected to AMQP server on 10.39.176.2:5672 via [amqp] client with port 43306.

2021-08-10 20:44:20.902 19626 INFO oslo.messaging._drivers.impl_rabbit [-] [20efaa3c-79ef-4e45-b83c-91bda815f549] Reconnected to AMQP server on 10.39.176.17:5672 via [amqp] client with port 41196.

2021-08-10 20:44:22.952 19626 INFO nova.compute.resource_tracker [req-27e11dbc-6beb-4543-8768-df7a35fb7ed2 - - - - -] Compute node realtime free ram 224326 mb, realtime cpu idle 0.694195710231

At the same time, the rabbitmq's log hasn't any error info.

What does Broken pipe mean and why there is so many Broken pipe error?

[cche...@gmail.com]

I check the log again and find when oslo.messaging report Broken pipe, at the same time , mq report client unexpectedly closed TCP connection, the log time is exactly the same, I can't find out which log is first and which log is last:

nova-compute.log:

2021-08-10 20:43:57.047 19626 INFO oslo.messaging._drivers.impl_rabbit [-] [cf287c2d-0135-405a-9d83-784fc3cc1729] Reconnected to AMQP server on 10.39.176.9:5672 via [amqp] client with port 49838.

2021-08-10 20:43:57.084 19626 INFO oslo.messaging._drivers.impl_rabbit [-] [cef69dca-f065-4b5e-8129-b42032d23717] Reconnected to AMQP server on 10.39.176.17:5672 via [amqp] client with port 40620.

2021-08-10 20:44:18.665 19626 ERROR oslo.messaging._drivers.impl_rabbit [-] [29e26e71-50d9-4049-a22f-082b0a9e4b30] AMQP server on 10.39.176.2:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe

2021-08-10 20:44:18.676 19626 ERROR oslo.messaging._drivers.impl_rabbit [-] [b74ce5ce-1864-46b1-b4eb-a509e64666bc] AMQP server on 10.39.176.2:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe

2021-08-10 20:44:18.679 19626 ERROR oslo.messaging._drivers.impl_rabbit [-] [20efaa3c-79ef-4e45-b83c-91bda815f549] AMQP server on 10.39.176.17:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe

the mq node 10.39.176.2's log:

2021-08-10 20:44:11.118 [info] <0.19291.3925> accepting AMQP connection <0.19291.3925> (10.39.176.252:47724 -> 10.39.176.2:5672)

--

2021-08-10 20:44:18.531 [info] <0.19797.4111> Connection <0.19797.4111> (10.39.176.32:42654 -> 10.39.176.2:5672) has a client-provided name: nova-compute:41162:7fc76855-a3b4-4579-ba94-cf60df829f85

2021-08-10 20:44:18.536 [info] <0.19797.4111> connection <0.19797.4111> (10.39.176.32:42654 -> 10.39.176.2:5672 - nova-compute:41162:7fc76855-a3b4-4579-ba94-cf60df829f85): user 'openstack' authenticated and granted access to vhost '/'

2021-08-10 20:44:18.665 [warning] <0.28284.4169> closing AMQP connection <0.28284.4169> (10.39.176.22:42948 -> 10.39.176.2:5672 - nova-compute:19626:29e26e71-50d9-4049-a22f-082b0a9e4b30, vhost: '/', user: 'openstack'):

client unexpectedly closed TCP connection

2021-08-10 20:44:18.676 [warning] <0.11227.3703> closing AMQP connection <0.11227.3703> (10.39.176.22:42942 -> 10.39.176.2:5672 - nova-compute:19626:b74ce5ce-1864-46b1-b4eb-a509e64666bc, vhost: '/', user: 'openstack'):

client unexpectedly closed TCP connection

2021-08-10 20:44:18.758 [info] <0.2478.3652> accepting AMQP connection <0.2478.3652> (10.39.176.22:43216 -> 10.39.176.2:5672)

at the exactly same time 2021-08-10 20:44:18.665  and 2021-08-10 20:44:18.676, client report Broken pipe, mq report client unexpectedly closed TCP connection, why?