I'm new to openstack, I'm testing the product with no luck, the instance can not find hosts, I have issued a command:
#########################################################################################
Message from syslogd@amer at Jan 27 19:26:19 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 26s! [swapper/5:0]
Message from syslogd@amer at Jan 27 19:26:19 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#1 stuck for 27s! [dmeventd:71548]
Message from syslogd@amer at Jan 27 19:27:30 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [6_scheduler:64928]
Message from syslogd@amer at Jan 27 19:31:25 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [ksoftirqd/5:34]
Message from syslogd@amer at Jan 27 19:32:42 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 33s! [swift-object-up:11358]
Message from syslogd@amer at Jan 27 19:33:55 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [dmeventd:71548]
Message from syslogd@amer at Jan 27 19:34:25 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 65s! [kworker/2:0:59993]
Message from syslogd@amer at Jan 27 19:37:50 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [kworker/u256:3:8447]
Message from syslogd@amer at Jan 27 19:37:50 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [ksoftirqd/5:34]
Message from syslogd@amer at Jan 27 19:37:51 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [systemd:11968]
The CPU has been disabled by the guest operating system. Power off or reset the virtual machine.
########################################################################################
well ... RH people opened a case for me before that error happen for a neutron and swift bug, Bugzilla decided that there is no bug, but I'm sure that each time on a clean installation I instantiate an instance the swift-object-server crashes and causes the nova to time out refusing to create the instance.
anyway I have a new VM on 12GB RAM, 4 cores CPU, it's a bit faster VM, and same scenario, no instance no swift, nothing , the product is locked..
thanks for your understanding, I have the nova-api.log if you want to trace, also I have issued the following error log :
[root@amer ~(keystone_admin)]# Jan 28 20:25:46
amer.example.com swift-object-server[34320]: error_msg = strategy.do_bind_ports()
-bash: syntax error near unexpected token `('
[root@amer ~(keystone_admin)]# Jan 28 20:25:46
amer.example.com swift-object-server[34320]: File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 615, i..._ports
bash: Jan: command not found...
[root@amer ~(keystone_admin)]# Jan 28 20:25:46
amer.example.com swift-object-server[34320]: self.sock = get_socket(self.conf)
-bash: syntax error near unexpected token `('
[root@amer ~(keystone_admin)]# Jan 28 20:25:46
amer.example.com swift-object-server[34320]: File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 212, i...socket
bash: Jan: command not found...
[root@amer ~(keystone_admin)]# Jan 28 20:25:46
amer.example.com swift-object-server[34320]: 'timeout': bind_timeout})
-bash: syntax error near unexpected token `)'
[root@amer ~(keystone_admin)]# Jan 28 20:25:46
amer.example.com swift-object-server[34320]: Exception: Could not bind to
192.168.43.110:6000 after trying for 30 seconds
bash: Jan: command not found...
[root@amer ~(keystone_admin)]# Jan 28 20:25:46
amer.example.com systemd[1]: openstack-swift-object.service: main process exited, code=exited, status=1/FAILURE
bash: Jan: command not found...
[root@amer ~(keystone_admin)]# Jan 28 20:25:46
amer.example.com systemd[1]: Unit openstack-swift-object.service entered failed state.
bash: Jan: command not found...
[root@amer ~(keystone_admin)]# Jan 28 20:25:46
amer.example.com systemd[1]: openstack-swift-object.service failed.
bash: Jan: command not found...
[root@amer ~(keystone_admin)]# clear
[root@amer ~(keystone_admin)]# cat /var/log/nova/nova*.log|grep -i error
2019-01-28 17:17:18.998 90010 ERROR nova.compute.manager [req-77575a66-e1c5-40fd-9610-f0d5413e1959 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 17:18:18.903 90010 ERROR nova.compute.manager [req-77575a66-e1c5-40fd-9610-f0d5413e1959 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 17:19:19.902 90010 ERROR nova.compute.manager [req-77575a66-e1c5-40fd-9610-f0d5413e1959 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 17:20:20.827 90010 ERROR nova.compute.manager [req-77575a66-e1c5-40fd-9610-f0d5413e1959 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 17:21:22.825 90010 ERROR nova.compute.manager [req-77575a66-e1c5-40fd-9610-f0d5413e1959 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 19:50:42.338 14335 ERROR oslo.messaging._drivers.impl_rabbit [req-ddf72a59-bf78-4186-80ee-507227a1f603 - - - - -] [3bf4693c-2bec-4f5a-bcd6-c7dd7bdb0390] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:43.385 14335 ERROR oslo.messaging._drivers.impl_rabbit [req-ddf72a59-bf78-4186-80ee-507227a1f603 - - - - -] [3bf4693c-2bec-4f5a-bcd6-c7dd7bdb0390] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:45.433 14335 ERROR oslo.messaging._drivers.impl_rabbit [req-ddf72a59-bf78-4186-80ee-507227a1f603 - - - - -] [3bf4693c-2bec-4f5a-bcd6-c7dd7bdb0390] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:49.460 14335 ERROR oslo.messaging._drivers.impl_rabbit [req-ddf72a59-bf78-4186-80ee-507227a1f603 - - - - -] [3bf4693c-2bec-4f5a-bcd6-c7dd7bdb0390] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:51:26.909 14335 WARNING keystoneauth.identity.generic.base [req-8e0e9e0e-5960-41c9-a252-6dea20d970cd - - - - -] Failed to discover available identity versions when contacting
http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3166a38690>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:51:26.910 14335 ERROR nova.virt.ironic.driver [req-8e0e9e0e-5960-41c9-a252-6dea20d970cd - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3166a38690>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3166a38690>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:52:26.723 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 19:52:26.775 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting
http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0e90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:52:26.777 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0e90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0e90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:52:26.840 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting
http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697ee10>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:52:26.841 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697ee10>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697ee10>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:52:42.360 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 19:52:42.442 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting
http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697edd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:52:42.444 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697edd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697edd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:53:42.313 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 19:53:42.365 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting
http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697ea90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:53:42.366 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697ea90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697ea90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:54:43.318 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 19:54:43.377 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting
http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0450>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:54:43.379 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0450>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0450>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:55:47.648 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 19:55:48.350 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting
http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3166991dd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:55:48.351 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3166991dd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3166991dd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:56:46.333 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 19:56:46.404 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting
http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0fd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:56:46.406 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0fd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0fd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:57:46.356 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 19:57:46.427 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting
http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0110>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:57:46.428 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0110>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0110>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:58:46.350 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 19:58:46.419 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting
http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0990>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:58:46.422 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0990>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0990>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:59:46.345 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 19:59:46.418 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting
http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0f90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:59:46.419 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0f90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to
http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0f90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 20:00:47.376 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:01:48.331 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:02:49.412 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:03:49.319 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:04:51.315 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:05:51.359 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:06:51.381 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:07:51.320 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:08:52.312 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:09:53.307 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:10:53.323 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:11:55.345 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:12:56.523 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:13:56.311 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:14:58.313 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:15:58.348 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:16:59.367 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:17:59.333 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:19:00.335 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:20:02.310 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:21:03.328 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:22:05.369 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:23:07.405 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:24:08.323 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:25:09.328 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:26:11.322 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:27:13.311 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:28:15.335 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:29:16.318 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:30:18.329 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:31:18.325 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:32:19.327 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:33:20.511 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:34:21.387 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:35:21.393 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:36:23.364 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:37:25.358 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:38:25.386 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:39:27.358 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:40:27.352 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:41:28.353 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:42:29.393 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:43:30.390 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:44:30.383 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:45:31.382 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:46:33.351 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:47:35.380 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:48:37.384 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:49:37.386 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:50:38.359 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:51:38.376 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:52:38.383 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:53:38.306 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:54:39.370 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:55:41.382 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:56:43.380 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:57:44.387 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:58:44.353 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 20:59:46.364 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 21:00:47.374 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 21:01:47.372 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 21:02:49.387 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 21:03:49.365 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 21:04:51.381 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 21:05:53.415 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host
amer.example.com: ComputeHostNotFound_Remote: Compute host
amer.example.com could not be found.
2019-01-28 19:50:28.785 15079 ERROR oslo.messaging._drivers.impl_rabbit [req-4bd8ffb9-adc5-4986-9458-b06cace9de09 - - - - -] [c3eb3c33-9084-47f5-bbaf-f1627ce6863e] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:28.889 15078 ERROR oslo.messaging._drivers.impl_rabbit [req-442bb7a9-f715-4af8-8d40-67387b3325e8 - - - - -] [2d5de7bf-b63c-4ac7-816d-530b7cf1457b] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:29.599 15080 ERROR oslo.messaging._drivers.impl_rabbit [req-04a181fe-45da-47ef-9496-e59f38516a68 - - - - -] [c4fbf101-f9d5-43ae-a8f2-cccbeee60651] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:29.630 15077 ERROR oslo.messaging._drivers.impl_rabbit [req-87b37edb-b721-4812-9a7d-5b5a084639e8 - - - - -] [4a5084b7-8eaa-4192-b5d4-3823b3d8dd04] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:29.839 15079 ERROR oslo.messaging._drivers.impl_rabbit [req-4bd8ffb9-adc5-4986-9458-b06cace9de09 - - - - -] [c3eb3c33-9084-47f5-bbaf-f1627ce6863e] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:29.937 15078 ERROR oslo.messaging._drivers.impl_rabbit [req-442bb7a9-f715-4af8-8d40-67387b3325e8 - - - - -] [2d5de7bf-b63c-4ac7-816d-530b7cf1457b] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:30.644 15080 ERROR oslo.messaging._drivers.impl_rabbit [req-04a181fe-45da-47ef-9496-e59f38516a68 - - - - -] [c4fbf101-f9d5-43ae-a8f2-cccbeee60651] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:30.678 15077 ERROR oslo.messaging._drivers.impl_rabbit [req-87b37edb-b721-4812-9a7d-5b5a084639e8 - - - - -] [4a5084b7-8eaa-4192-b5d4-3823b3d8dd04] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:31.884 15079 ERROR oslo.messaging._drivers.impl_rabbit [req-4bd8ffb9-adc5-4986-9458-b06cace9de09 - - - - -] [c3eb3c33-9084-47f5-bbaf-f1627ce6863e] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:31.983 15078 ERROR oslo.messaging._drivers.impl_rabbit [req-442bb7a9-f715-4af8-8d40-67387b3325e8 - - - - -] [2d5de7bf-b63c-4ac7-816d-530b7cf1457b] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:32.674 15080 ERROR oslo.messaging._drivers.impl_rabbit [req-04a181fe-45da-47ef-9496-e59f38516a68 - - - - -] [c4fbf101-f9d5-43ae-a8f2-cccbeee60651] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:32.724 15077 ERROR oslo.messaging._drivers.impl_rabbit [req-87b37edb-b721-4812-9a7d-5b5a084639e8 - - - - -] [4a5084b7-8eaa-4192-b5d4-3823b3d8dd04] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:35.930 15079 ERROR oslo.messaging._drivers.impl_rabbit [req-4bd8ffb9-adc5-4986-9458-b06cace9de09 - - - - -] [c3eb3c33-9084-47f5-bbaf-f1627ce6863e] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:36.033 15078 ERROR oslo.messaging._drivers.impl_rabbit [req-442bb7a9-f715-4af8-8d40-67387b3325e8 - - - - -] [2d5de7bf-b63c-4ac7-816d-530b7cf1457b] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:36.724 15080 ERROR oslo.messaging._drivers.impl_rabbit [req-04a181fe-45da-47ef-9496-e59f38516a68 - - - - -] [c4fbf101-f9d5-43ae-a8f2-cccbeee60651] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:36.789 15077 ERROR oslo.messaging._drivers.impl_rabbit [req-87b37edb-b721-4812-9a7d-5b5a084639e8 - - - - -] [4a5084b7-8eaa-4192-b5d4-3823b3d8dd04] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:41.977 15079 ERROR oslo.messaging._drivers.impl_rabbit [req-4bd8ffb9-adc5-4986-9458-b06cace9de09 - - - - -] [c3eb3c33-9084-47f5-bbaf-f1627ce6863e] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:42.077 15078 ERROR oslo.messaging._drivers.impl_rabbit [req-442bb7a9-f715-4af8-8d40-67387b3325e8 - - - - -] [2d5de7bf-b63c-4ac7-816d-530b7cf1457b] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:42.769 15080 ERROR oslo.messaging._drivers.impl_rabbit [req-04a181fe-45da-47ef-9496-e59f38516a68 - - - - -] [c4fbf101-f9d5-43ae-a8f2-cccbeee60651] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:42.842 15077 ERROR oslo.messaging._drivers.impl_rabbit [req-87b37edb-b721-4812-9a7d-5b5a084639e8 - - - - -] [4a5084b7-8eaa-4192-b5d4-3823b3d8dd04] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:50.007 15079 ERROR oslo.messaging._drivers.impl_rabbit [req-4bd8ffb9-adc5-4986-9458-b06cace9de09 - - - - -] [c3eb3c33-9084-47f5-bbaf-f1627ce6863e] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:50.124 15078 ERROR oslo.messaging._drivers.impl_rabbit [req-442bb7a9-f715-4af8-8d40-67387b3325e8 - - - - -] [2d5de7bf-b63c-4ac7-816d-530b7cf1457b] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:50.791 15080 ERROR oslo.messaging._drivers.impl_rabbit [req-04a181fe-45da-47ef-9496-e59f38516a68 - - - - -] [c4fbf101-f9d5-43ae-a8f2-cccbeee60651] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:50.895 15077 ERROR oslo.messaging._drivers.impl_rabbit [req-87b37edb-b721-4812-9a7d-5b5a084639e8 - - - - -] [4a5084b7-8eaa-4192-b5d4-3823b3d8dd04] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager [req-b5bcd4c5-a59e-4a9d-abea-89f140f6d47b 086d1437fd694fdfb826df6934e87775 5843f5028ae04d0d9db3471dd75b6eb0 - default default] Failed to schedule instances: NoValidHost_Remote: No valid host was found.
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager Traceback (most recent call last):
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 1231, in schedule_and_build_instances
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager instance_uuids, return_alternates=True)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 748, in _schedule_instances
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager return_alternates=return_alternates)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 907, in wrapped
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager return func(*args, **kwargs)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 53, in select_destinations
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager instance_uuids, return_objects, return_alternates)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager instance_uuids, return_objects, return_alternates)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 158, in select_destinations
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager retry=self.retry)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager retry=retry)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager call_monitor_timeout, retry=retry)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager raise result
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found.
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager Traceback (most recent call last):
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager return func(*args, **kwargs)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 154, in select_destinations
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager raise exception.NoValidHost(reason="")
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager NoValidHost: No valid host was found.
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager
2019-01-28 20:16:18.027 15078 WARNING nova.scheduler.utils [req-b5bcd4c5-a59e-4a9d-abea-89f140f6d47b 086d1437fd694fdfb826df6934e87775 5843f5028ae04d0d9db3471dd75b6eb0 - default default] [instance: 8844cca1-65ce-47b7-815d-edd9d5c13389] Setting instance to ERROR state.: NoValidHost_Remote: No valid host was found.
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager [req-2bbd6011-46a8-4333-a4f8-89450c19a9a2 086d1437fd694fdfb826df6934e87775 5843f5028ae04d0d9db3471dd75b6eb0 - default default] Failed to schedule instances: NoValidHost_Remote: No valid host was found.
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager Traceback (most recent call last):
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 1231, in schedule_and_build_instances
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager instance_uuids, return_alternates=True)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 748, in _schedule_instances
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager return_alternates=return_alternates)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 907, in wrapped
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager return func(*args, **kwargs)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 53, in select_destinations
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager instance_uuids, return_objects, return_alternates)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager instance_uuids, return_objects, return_alternates)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 158, in select_destinations
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager retry=self.retry)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager retry=retry)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager call_monitor_timeout, retry=retry)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager raise result
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found.
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager Traceback (most recent call last):
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager return func(*args, **kwargs)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 154, in select_destinations
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager raise exception.NoValidHost(reason="")
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager NoValidHost: No valid host was found.
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager
2019-01-28 20:27:10.886 15077 WARNING nova.scheduler.utils [req-2bbd6011-46a8-4333-a4f8-89450c19a9a2 086d1437fd694fdfb826df6934e87775 5843f5028ae04d0d9db3471dd75b6eb0 - default default] [instance: 30267bb9-e52e-4b71-8a8f-0800f1e81b4a] Setting instance to ERROR state.: NoValidHost_Remote: No valid host was found.
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines [-] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at:
http://sqlalche.me/e/e3q8)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines connection.scalar(select([1]))
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines return self.execute(object, *multiparams, **params).scalar()
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines return connection._execute_clauseelement(self, multiparams, params)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines compiled_sql, distilled_params
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines context)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines util.raise_from_cause(newraise, exc_info)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines context)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines result = self._query(query)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines conn.query(q)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines result.read()
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet()
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 998, in _read_packet
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines "Lost connection to MySQL server during query")
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at:
http://sqlalche.me/e/e3q8)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db [-] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.43.110' ([Errno 111] ECONNREFUSED)") [SQL: u'SELECT 1'] (Background on this error at:
http://sqlalche.me/e/e3q8)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db Traceback (most recent call last):
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db service.service_ref.save()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 396, in save
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db self._check_minimum_version()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 371, in _check_minimum_version
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 475, in get_minimum_version
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db use_slave=use_slave)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 452, in get_minimum_version_multi
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 210, in wrapper
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 438, in _db_service_get_minimum_version
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 117, in service_get_minimum_version
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 441, in service_get_minimum_version
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db return dict(min_versions)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, in __iter__
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2946, in _execute_and_instances
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db close_with_result=True)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2955, in _get_bind_args
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db **kw
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2937, in _connection_from_session
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db execution_options=execution_options)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db engine, execution_options)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2125, in contextual_connect
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db **kwargs)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 110, in __init__
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db self.dispatch.engine_connect(self, self.__branch)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 284, in __call__
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db fn(*args, **kw)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 83, in _connect_ping_listener
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db connection.scalar(select([1]))
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db return self.execute(object, *multiparams, **params).scalar()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db return meth(self, multiparams, params)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db return connection._execute_clauseelement(self, multiparams, params)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db compiled_sql, distilled_params
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1132, in _execute_context
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db None, None)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1125, in _execute_context
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db conn = self._revalidate_connection()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 429, in _revalidate_connection
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db self.__connection = self.engine.raw_connection(_connection=self)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2188, in raw_connection
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db self.pool.unique_connection, _connection)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2164, in _wrap_pool_connect
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db util.reraise(*sys.exc_info())
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db return fn()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 345, in unique_connection
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 788, in _checkout
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db rec._checkin_failed(err)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 627, in get_connection
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db self.__connect()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 671, in __connect
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 106, in connect
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db self.connect()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db raise exc
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.43.110' ([Errno 111] ECONNREFUSED)") [SQL: u'SELECT 1'] (Background on this error at:
http://sqlalche.me/e/e3q8)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db
2019-01-28 19:50:22.956 10777 ERROR oslo.messaging._drivers.impl_rabbit [req-504e234e-1af3-4471-9893-30480786f439 - - - - -] [e21c5911-6553-47a8-bb5e-44fc20bd4c6d] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:24.019 10777 ERROR oslo.messaging._drivers.impl_rabbit [req-504e234e-1af3-4471-9893-30480786f439 - - - - -] [e21c5911-6553-47a8-bb5e-44fc20bd4c6d] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:26.067 10777 ERROR oslo.messaging._drivers.impl_rabbit [req-504e234e-1af3-4471-9893-30480786f439 - - - - -] [e21c5911-6553-47a8-bb5e-44fc20bd4c6d] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:30.129 10777 ERROR oslo.messaging._drivers.impl_rabbit [req-504e234e-1af3-4471-9893-30480786f439 - - - - -] [e21c5911-6553-47a8-bb5e-44fc20bd4c6d] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:36.181 10777 ERROR oslo.messaging._drivers.impl_rabbit [req-504e234e-1af3-4471-9893-30480786f439 - - - - -] [e21c5911-6553-47a8-bb5e-44fc20bd4c6d] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:44.242 10777 ERROR oslo.messaging._drivers.impl_rabbit [req-504e234e-1af3-4471-9893-30480786f439 - - - - -] [e21c5911-6553-47a8-bb5e-44fc20bd4c6d] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:57:39.426 10777 ERROR oslo.messaging._drivers.impl_rabbit [-] [e21c5911-6553-47a8-bb5e-44fc20bd4c6d] AMQP server
192.168.43.110:5672 closed the connection. Check login credentials: Socket closed: IOError: Socket closed
2019-01-28 19:50:36.887 15222 ERROR oslo.messaging._drivers.impl_rabbit [req-9094253e-624b-4654-9494-45da68c4e08e - - - - -] [b860dcb3-6a51-458a-92c5-55bdd08613bb] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:36.943 15221 ERROR oslo.messaging._drivers.impl_rabbit [req-dc40ab3d-f7e4-4ea7-bce1-fcd830d17f9a - - - - -] [386112ab-9467-4fa1-9e52-6ddd6a62a00a] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:37.933 15222 ERROR oslo.messaging._drivers.impl_rabbit [req-9094253e-624b-4654-9494-45da68c4e08e - - - - -] [b860dcb3-6a51-458a-92c5-55bdd08613bb] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:37.992 15221 ERROR oslo.messaging._drivers.impl_rabbit [req-dc40ab3d-f7e4-4ea7-bce1-fcd830d17f9a - - - - -] [386112ab-9467-4fa1-9e52-6ddd6a62a00a] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:39.983 15222 ERROR oslo.messaging._drivers.impl_rabbit [req-9094253e-624b-4654-9494-45da68c4e08e - - - - -] [b860dcb3-6a51-458a-92c5-55bdd08613bb] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:40.036 15221 ERROR oslo.messaging._drivers.impl_rabbit [req-dc40ab3d-f7e4-4ea7-bce1-fcd830d17f9a - - - - -] [386112ab-9467-4fa1-9e52-6ddd6a62a00a] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:44.034 15222 ERROR oslo.messaging._drivers.impl_rabbit [req-9094253e-624b-4654-9494-45da68c4e08e - - - - -] [b860dcb3-6a51-458a-92c5-55bdd08613bb] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:44.083 15221 ERROR oslo.messaging._drivers.impl_rabbit [req-dc40ab3d-f7e4-4ea7-bce1-fcd830d17f9a - - - - -] [386112ab-9467-4fa1-9e52-6ddd6a62a00a] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:50.060 15222 ERROR oslo.messaging._drivers.impl_rabbit [req-9094253e-624b-4654-9494-45da68c4e08e - - - - -] [b860dcb3-6a51-458a-92c5-55bdd08613bb] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:50.131 15221 ERROR oslo.messaging._drivers.impl_rabbit [req-dc40ab3d-f7e4-4ea7-bce1-fcd830d17f9a - - - - -] [386112ab-9467-4fa1-9e52-6ddd6a62a00a] AMQP server on
192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines [-] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at:
http://sqlalche.me/e/e3q8)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines connection.scalar(select([1]))
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines return self.execute(object, *multiparams, **params).scalar()
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines return connection._execute_clauseelement(self, multiparams, params)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines compiled_sql, distilled_params
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines context)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines util.raise_from_cause(newraise, exc_info)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines context)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines result = self._query(query)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines conn.query(q)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines result.read()
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet()
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines packet_header = self._read_bytes(4)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1033, in _read_bytes
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at:
http://sqlalche.me/e/e3q8)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines [-] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at:
http://sqlalche.me/e/e3q8)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines connection.scalar(select([1]))
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines return self.execute(object, *multiparams, **params).scalar()
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines return connection._execute_clauseelement(self, multiparams, params)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines compiled_sql, distilled_params
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines context)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines util.raise_from_cause(newraise, exc_info)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines context)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines result = self._query(query)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines conn.query(q)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines result.read()
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet()
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines packet_header = self._read_bytes(4)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1033, in _read_bytes
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at:
http://sqlalche.me/e/e3q8)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines
2019-01-28 19:57:58.181 15222 ERROR oslo.messaging._drivers.impl_rabbit [-] [b860dcb3-6a51-458a-92c5-55bdd08613bb] AMQP server
192.168.43.110:5672 closed the connection. Check login credentials: Socket closed: IOError: Socket closed
2019-01-28 19:57:58.247 15221 ERROR oslo.messaging._drivers.impl_rabbit [-] [386112ab-9467-4fa1-9e52-6ddd6a62a00a] AMQP server
192.168.43.110:5672 closed the connection. Check login credentials: Socket closed: IOError: Socket closed
[root@amer ~(keystone_admin)]#
any one can interpret it to me, mysql is gone some times and rabbit timedout, swift is down, and nova not creating instances...
Thanks in advance for you patience.