Multi-node Nova

69 views

Skip to first unread message

Soheil Eizadi

unread,

Mar 19, 2015, 1:53:44 PM3/19/15

to opscode-che...@googlegroups.com

I tried to run the multi_nova use case, but had issues with the way the vagrant-multi-nova was checked in. I changed the endpoints and got it to work, but was trying to figure out how it worked the way it was checked in, I am using NAT for the first NIC and Bridged for the second NIC. To get it to work I had to change the cluster to use the second NIC for the inter-cluster communication. I also had to patch rabbitmq since the default was to use guest and it is not allowed on anything other than local port.
https://bugs.launchpad.net/packstack/+bug/1393715

-Soheil

JJ Asghar

unread,

Mar 19, 2015, 3:14:06 PM3/19/15

to opscode-che...@googlegroups.com

Yep, we’ve noticed this problem.

I’m attempting to get this review through so we can ideally resolve this.

https://review.openstack.org/#/c/165506/

I think this is the issue you may have ran into. Can you pull down these changes and see with a clean clone of the repo if it resolves it?

--
You received this message because you are subscribed to the Google Groups "opscode-chef-openstack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opscode-chef-open...@googlegroups.com.
To post to this group, send email to opscode-che...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/opscode-chef-openstack/CALFsGHm7V427o%2BXhaEPRtb3ZVe71-FNfFgWc4GgdTFdnQ%3DB2gA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Best Regards,
JJ Asghar
e: j...@chef.io | c: 512.619.0722

Soheil Eizadi

unread,

Mar 19, 2015, 4:17:34 PM3/19/15

to opscode-che...@googlegroups.com

Thanks JJ, I added some comments on the patch. -Soheil

To view this discussion on the web visit https://groups.google.com/d/msgid/opscode-chef-openstack/8E1315EF-399F-43DB-94CF-B613383C54B4%40chef.io.

Mark Vanderwiel

unread,

Mar 23, 2015, 2:49:43 PM3/23/15

to opscode-che...@googlegroups.com

would like to see your environment for multi-nova, what you changed (in both the env and recipe files).

I also think we should tweak that a bit we now have global attributes in common to help with binding to eth0 or 1 and not have to worry about specific ip's for at least the inter-cloud communications.

Soheil Eizadi

unread,

Mar 23, 2015, 3:52:53 PM3/23/15

to opscode-che...@googlegroups.com

Hi Mark,

Here are the change I have made below. I cloned the master yesterday.

-Soheil

$ git status
On branch master
....
    modified:   environments/vagrant-multi-nova.json
    modified:   multi-nova.rb

Untracked files:
    teststack.rc

$ git diff environments/vagrant-multi-nova.json
diff --git a/environments/vagrant-multi-nova.json b/environments/vagrant-multi-nova.json
index 975580c..f672337 100644
--- a/environments/vagrant-multi-nova.json
+++ b/environments/vagrant-multi-nova.json
@@ -21,7 +21,14 @@
             "allow_remote_root": true,
             "root_network_acl": ["%"]
         },
-        "openstack": {
+        "rabbitmq":{
+            "default_user": "admin",
+            "default_pass": "mypass"
+        },
+         "openstack": {
+            "mq": {
+                       "user": "admin"
+            },
             "yum": {
                 "uri": "https://repos.fedorapeople.org/repos/openstack/openstack-kilo/epel-7"
             },

$ git diff multi-nova.rb
diff --git a/multi-nova.rb b/multi-nova.rb
index d450138..fb310f3 100644
--- a/multi-nova.rb
+++ b/multi-nova.rb
@@ -16,7 +16,7 @@ controller_config = <<-ENDCONFIG
     v.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
   end
   config.vm.network "public_network", ip: "172.16.100.60",
-    bridge: '<put your interface device name here>'
+    bridge: 'en0: Wi-Fi (AirPort)'
ENDCONFIG

machine 'controller' do
@@ -47,7 +47,7 @@ config.vm.provider "virtualbox" do |v|
   v.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
end
config.vm.network "public_network", ip: "172.16.100.#{ip_suff}",
- bridge: '<put your interface device name here>'
+ bridge: 'en0: Wi-Fi (AirPort)'
ENDCONFIG
       role 'os-compute-worker'
       # if you would like to use centos7 you'll need to

On Mon, Mar 23, 2015 at 11:49 AM, Mark Vanderwiel <vand...@us.ibm.com> wrote:

would like to see your environment for multi-nova, what you changed (in both the env and recipe files).
I also think we should tweak that a bit we now have global attributes in common to help with binding to eth0 or 1 and not have to worry about specific ip's for at least the inter-cloud communications.

--

You received this message because you are subscribed to the Google Groups "opscode-chef-openstack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opscode-chef-open...@googlegroups.com.
To post to this group, send email to opscode-che...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/opscode-chef-openstack/a3903025-9136-492f-9844-14c5f638a706%40googlegroups.com.

Mark Vanderwiel

unread,

Mar 23, 2015, 4:54:46 PM3/23/15

to opscode-che...@googlegroups.com

Soheil, for that rabbit override, you don't need the default_user/default_pass, as the openstack->mq->user will handle setting up the mq admin user (and will auto delete guest user), see ops-msg cookbook.

I would like to reduce the multi_nova env a bit more but using the common high-level bind-interface, https://github.com/stackforge/cookbook-openstack-common/blob/master/attributes/default.rb#L182 attribute to 'eth1', that should reduce the amount of ip addresses we need to specify in the env file.

Soheil Eizadi

unread,

Mar 23, 2015, 5:55:48 PM3/23/15

to opscode-che...@googlegroups.com

I took your suggestions and cleaned it up a bit more, see below. I ran this also and it converged.

-Soheil

$ git diff environments/vagrant-multi-nova.json
diff --git a/environments/vagrant-multi-nova.json b/environments/vagrant-multi-nova.json

index 975580c..936001e 100644
--- a/environments/vagrant-multi-nova.json
+++ b/environments/vagrant-multi-nova.json
@@ -21,26 +21,16 @@

             "allow_remote_root": true,
             "root_network_acl": ["%"]
         },
-        "openstack": {

+         "openstack": {
+            "mq": {
+                       "user": "admin"
+            },
             "yum": {
                 "uri": "https://repos.fedorapeople.org/repos/openstack/openstack-kilo/epel-7"
             },

             "identity": {
                 "bind_interface": "eth1"
             },
-            "endpoints": {
-                "bind-host": "172.16.100.60",
-                "host": "172.16.100.60",
-                "mq": {
-                    "host": "172.16.100.60"
-                },
-                "db": {
-                    "host": "172.16.100.60"
-                },
-                "compute-vnc-bind":{
-                    "host": "0.0.0.0"
-                }
-            },
             "image": {
                 "api": {
                     "bind_interface": "eth1"

On Mon, Mar 23, 2015 at 1:54 PM, Mark Vanderwiel <vand...@us.ibm.com> wrote:

Soheil, for that rabbit override, you don't need the default_user/default_pass, as the openstack->mq->user will handle setting up the mq admin user (and will auto delete guest user), see ops-msg cookbook.

I would like to reduce the multi_nova env a bit more but using the common high-level bind-interface, https://github.com/stackforge/cookbook-openstack-common/blob/master/attributes/default.rb#L182 attribute to 'eth1', that should reduce the amount of ip addresses we need to specify in the env file.

--

You received this message because you are subscribed to the Google Groups "opscode-chef-openstack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opscode-chef-open...@googlegroups.com.
To post to this group, send email to opscode-che...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/opscode-chef-openstack/e8f33f44-2d95-4ba2-a30f-6d721b9fb2e1%40googlegroups.com.

JJ Asghar

unread,

Mar 23, 2015, 6:28:17 PM3/23/15

to opscode-che...@googlegroups.com

Good news everyone,

I think with this: https://review.openstack.org/#/c/167021/ we should be back on track.

To view this discussion on the web visit https://groups.google.com/d/msgid/opscode-chef-openstack/CALFsGHmWZBYt%3DfnLGC%2BzaEBm_803H54NV1gubBRTh-nJs5gjFw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Amer Hwitat

unread,

Jan 28, 2019, 9:11:31 PM1/28/19

to opscode-chef-openstack

Hi guys,

I'm new to openstack, I'm testing the product with no luck, the instance can not find hosts, I have issued a command:

the services are up, I'm using all in one on a VM

I traced the log of RabbitMQ

it shows connection inet timedout

did some configurations then it showed this

#########################################################################################

Message from syslogd@amer at Jan 27 19:26:19 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 26s! [swapper/5:0]

Message from syslogd@amer at Jan 27 19:26:19 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#1 stuck for 27s! [dmeventd:71548]

Message from syslogd@amer at Jan 27 19:27:30 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [6_scheduler:64928]

Message from syslogd@amer at Jan 27 19:31:25 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [ksoftirqd/5:34]

Message from syslogd@amer at Jan 27 19:32:42 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 33s! [swift-object-up:11358]

Message from syslogd@amer at Jan 27 19:33:55 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [dmeventd:71548]

Message from syslogd@amer at Jan 27 19:34:25 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 65s! [kworker/2:0:59993]

Message from syslogd@amer at Jan 27 19:37:50 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [kworker/u256:3:8447]

Message from syslogd@amer at Jan 27 19:37:50 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [ksoftirqd/5:34]

Message from syslogd@amer at Jan 27 19:37:51 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [systemd:11968]

The CPU has been disabled by the guest operating system. Power off or reset the virtual machine.

########################################################################################

then the Machine crashed and, I've attached the scroonshot:

Red Hat Enterprise Linux 7 64-bit (2)-2019-01-28-03-57-27.png

and I filed a bug report with no answer from Bugzilla, I think it's not all my mis-configuration, I was on 8GB RAM 8 cores CPU, and it was very slow.., also launchpad didn't reply well, ask.openstack.org and RH support saying I have to subscribe to support to get the service , for a kernel Bug ...

well ... RH people opened a case for me before that error happen for a neutron and swift bug, Bugzilla decided that there is no bug, but I'm sure that each time on a clean installation I instantiate an instance the swift-object-server crashes and causes the nova to time out refusing to create the instance.

● openstack-swift-object.service - OpenStack Object Storage (swift) - Object Server
Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2019-01-28 20:25:46 EST; 38min ago
Process: 34320 ExecStart=/usr/bin/swift-object-server /etc/swift/object-server.conf (code=exited, status=1/FAILURE)
Main PID: 34320 (code=exited, status=1/FAILURE)

Jan 28 20:25:46 amer.example.com swift-object-server[34320]: File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 1036, ...n_wsgi
Jan 28 20:25:46 amer.example.com swift-object-server[34320]: error_msg = strategy.do_bind_ports()
Jan 28 20:25:46 amer.example.com swift-object-server[34320]: File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 615, i..._ports
Jan 28 20:25:46 amer.example.com swift-object-server[34320]: self.sock = get_socket(self.conf)
Jan 28 20:25:46 amer.example.com swift-object-server[34320]: File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 212, i...socket
Jan 28 20:25:46 amer.example.com swift-object-server[34320]: 'timeout': bind_timeout})
Jan 28 20:25:46 amer.example.com swift-object-server[34320]: Exception: Could not bind to 192.168.43.110:6000 after trying for 30 seconds
Jan 28 20:25:46 amer.example.com systemd[1]: openstack-swift-object.service: main process exited, code=exited, status=1/FAILURE
Jan 28 20:25:46 amer.example.com systemd[1]: Unit openstack-swift-object.service entered failed state.
Jan 28 20:25:46 amer.example.com systemd[1]: openstack-swift-object.service failed.

anyway I have a new VM on 12GB RAM, 4 cores CPU, it's a bit faster VM, and same scenario, no instance no swift, nothing , the product is locked..

thanks for your understanding, I have the nova-api.log if you want to trace, also I have issued the following error log :

[root@amer ~(keystone_admin)]# Jan 28 20:25:46 amer.example.com swift-object-server[34320]: error_msg = strategy.do_bind_ports()
-bash: syntax error near unexpected token `('
[root@amer ~(keystone_admin)]# Jan 28 20:25:46 amer.example.com swift-object-server[34320]: File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 615, i..._ports
bash: Jan: command not found...
[root@amer ~(keystone_admin)]# Jan 28 20:25:46 amer.example.com swift-object-server[34320]: self.sock = get_socket(self.conf)
-bash: syntax error near unexpected token `('
[root@amer ~(keystone_admin)]# Jan 28 20:25:46 amer.example.com swift-object-server[34320]: File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 212, i...socket
bash: Jan: command not found...
[root@amer ~(keystone_admin)]# Jan 28 20:25:46 amer.example.com swift-object-server[34320]: 'timeout': bind_timeout})
-bash: syntax error near unexpected token `)'
[root@amer ~(keystone_admin)]# Jan 28 20:25:46 amer.example.com swift-object-server[34320]: Exception: Could not bind to 192.168.43.110:6000 after trying for 30 seconds
bash: Jan: command not found...
[root@amer ~(keystone_admin)]# Jan 28 20:25:46 amer.example.com systemd[1]: openstack-swift-object.service: main process exited, code=exited, status=1/FAILURE
bash: Jan: command not found...
[root@amer ~(keystone_admin)]# Jan 28 20:25:46 amer.example.com systemd[1]: Unit openstack-swift-object.service entered failed state.
bash: Jan: command not found...
[root@amer ~(keystone_admin)]# Jan 28 20:25:46 amer.example.com systemd[1]: openstack-swift-object.service failed.
bash: Jan: command not found...
[root@amer ~(keystone_admin)]# clear
[root@amer ~(keystone_admin)]# cat /var/log/nova/nova*.log|grep -i error
2019-01-28 17:17:18.998 90010 ERROR nova.compute.manager [req-77575a66-e1c5-40fd-9610-f0d5413e1959 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 17:18:18.903 90010 ERROR nova.compute.manager [req-77575a66-e1c5-40fd-9610-f0d5413e1959 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 17:19:19.902 90010 ERROR nova.compute.manager [req-77575a66-e1c5-40fd-9610-f0d5413e1959 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 17:20:20.827 90010 ERROR nova.compute.manager [req-77575a66-e1c5-40fd-9610-f0d5413e1959 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 17:21:22.825 90010 ERROR nova.compute.manager [req-77575a66-e1c5-40fd-9610-f0d5413e1959 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 19:50:42.338 14335 ERROR oslo.messaging._drivers.impl_rabbit [req-ddf72a59-bf78-4186-80ee-507227a1f603 - - - - -] [3bf4693c-2bec-4f5a-bcd6-c7dd7bdb0390] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:43.385 14335 ERROR oslo.messaging._drivers.impl_rabbit [req-ddf72a59-bf78-4186-80ee-507227a1f603 - - - - -] [3bf4693c-2bec-4f5a-bcd6-c7dd7bdb0390] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:45.433 14335 ERROR oslo.messaging._drivers.impl_rabbit [req-ddf72a59-bf78-4186-80ee-507227a1f603 - - - - -] [3bf4693c-2bec-4f5a-bcd6-c7dd7bdb0390] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:49.460 14335 ERROR oslo.messaging._drivers.impl_rabbit [req-ddf72a59-bf78-4186-80ee-507227a1f603 - - - - -] [3bf4693c-2bec-4f5a-bcd6-c7dd7bdb0390] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:51:26.909 14335 WARNING keystoneauth.identity.generic.base [req-8e0e9e0e-5960-41c9-a252-6dea20d970cd - - - - -] Failed to discover available identity versions when contacting http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3166a38690>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:51:26.910 14335 ERROR nova.virt.ironic.driver [req-8e0e9e0e-5960-41c9-a252-6dea20d970cd - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3166a38690>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3166a38690>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:52:26.723 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 19:52:26.775 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0e90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:52:26.777 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0e90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0e90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:52:26.840 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697ee10>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:52:26.841 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697ee10>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697ee10>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:52:42.360 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 19:52:42.442 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697edd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:52:42.444 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697edd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697edd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:53:42.313 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 19:53:42.365 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697ea90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:53:42.366 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697ea90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f316697ea90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:54:43.318 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 19:54:43.377 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0450>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:54:43.379 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0450>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0450>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:55:47.648 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 19:55:48.350 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3166991dd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:55:48.351 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3166991dd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3166991dd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:56:46.333 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 19:56:46.404 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0fd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:56:46.406 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0fd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0fd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:57:46.356 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 19:57:46.427 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0110>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:57:46.428 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0110>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0110>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:58:46.350 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 19:58:46.419 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0990>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:58:46.422 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0990>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0990>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:59:46.345 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 19:59:46.418 14335 WARNING keystoneauth.identity.generic.base [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] Failed to discover available identity versions when contacting http://192.168.43.110:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0f90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 19:59:46.419 14335 ERROR nova.virt.ironic.driver [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0f90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.43.110:35357: HTTPConnectionPool(host='192.168.43.110', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31669f0f90>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2019-01-28 20:00:47.376 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:01:48.331 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:02:49.412 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:03:49.319 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:04:51.315 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:05:51.359 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:06:51.381 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:07:51.320 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:08:52.312 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:09:53.307 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:10:53.323 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:11:55.345 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:12:56.523 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:13:56.311 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:14:58.313 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:15:58.348 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:16:59.367 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:17:59.333 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:19:00.335 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:20:02.310 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:21:03.328 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:22:05.369 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:23:07.405 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:24:08.323 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:25:09.328 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:26:11.322 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:27:13.311 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:28:15.335 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:29:16.318 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:30:18.329 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:31:18.325 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:32:19.327 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:33:20.511 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:34:21.387 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:35:21.393 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:36:23.364 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:37:25.358 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:38:25.386 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:39:27.358 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:40:27.352 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:41:28.353 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:42:29.393 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:43:30.390 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:44:30.383 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:45:31.382 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:46:33.351 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:47:35.380 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:48:37.384 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:49:37.386 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:50:38.359 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:51:38.376 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:52:38.383 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:53:38.306 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:54:39.370 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:55:41.382 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:56:43.380 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:57:44.387 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:58:44.353 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 20:59:46.364 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 21:00:47.374 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 21:01:47.372 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 21:02:49.387 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 21:03:49.365 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 21:04:51.381 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 21:05:53.415 14335 ERROR nova.compute.manager [req-688f0b6d-6300-40f2-b030-da0e2b768b68 - - - - -] No compute node record for host amer.example.com: ComputeHostNotFound_Remote: Compute host amer.example.com could not be found.
2019-01-28 19:50:28.785 15079 ERROR oslo.messaging._drivers.impl_rabbit [req-4bd8ffb9-adc5-4986-9458-b06cace9de09 - - - - -] [c3eb3c33-9084-47f5-bbaf-f1627ce6863e] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:28.889 15078 ERROR oslo.messaging._drivers.impl_rabbit [req-442bb7a9-f715-4af8-8d40-67387b3325e8 - - - - -] [2d5de7bf-b63c-4ac7-816d-530b7cf1457b] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:29.599 15080 ERROR oslo.messaging._drivers.impl_rabbit [req-04a181fe-45da-47ef-9496-e59f38516a68 - - - - -] [c4fbf101-f9d5-43ae-a8f2-cccbeee60651] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:29.630 15077 ERROR oslo.messaging._drivers.impl_rabbit [req-87b37edb-b721-4812-9a7d-5b5a084639e8 - - - - -] [4a5084b7-8eaa-4192-b5d4-3823b3d8dd04] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:29.839 15079 ERROR oslo.messaging._drivers.impl_rabbit [req-4bd8ffb9-adc5-4986-9458-b06cace9de09 - - - - -] [c3eb3c33-9084-47f5-bbaf-f1627ce6863e] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:29.937 15078 ERROR oslo.messaging._drivers.impl_rabbit [req-442bb7a9-f715-4af8-8d40-67387b3325e8 - - - - -] [2d5de7bf-b63c-4ac7-816d-530b7cf1457b] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:30.644 15080 ERROR oslo.messaging._drivers.impl_rabbit [req-04a181fe-45da-47ef-9496-e59f38516a68 - - - - -] [c4fbf101-f9d5-43ae-a8f2-cccbeee60651] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:30.678 15077 ERROR oslo.messaging._drivers.impl_rabbit [req-87b37edb-b721-4812-9a7d-5b5a084639e8 - - - - -] [4a5084b7-8eaa-4192-b5d4-3823b3d8dd04] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:31.884 15079 ERROR oslo.messaging._drivers.impl_rabbit [req-4bd8ffb9-adc5-4986-9458-b06cace9de09 - - - - -] [c3eb3c33-9084-47f5-bbaf-f1627ce6863e] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:31.983 15078 ERROR oslo.messaging._drivers.impl_rabbit [req-442bb7a9-f715-4af8-8d40-67387b3325e8 - - - - -] [2d5de7bf-b63c-4ac7-816d-530b7cf1457b] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:32.674 15080 ERROR oslo.messaging._drivers.impl_rabbit [req-04a181fe-45da-47ef-9496-e59f38516a68 - - - - -] [c4fbf101-f9d5-43ae-a8f2-cccbeee60651] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:32.724 15077 ERROR oslo.messaging._drivers.impl_rabbit [req-87b37edb-b721-4812-9a7d-5b5a084639e8 - - - - -] [4a5084b7-8eaa-4192-b5d4-3823b3d8dd04] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:35.930 15079 ERROR oslo.messaging._drivers.impl_rabbit [req-4bd8ffb9-adc5-4986-9458-b06cace9de09 - - - - -] [c3eb3c33-9084-47f5-bbaf-f1627ce6863e] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:36.033 15078 ERROR oslo.messaging._drivers.impl_rabbit [req-442bb7a9-f715-4af8-8d40-67387b3325e8 - - - - -] [2d5de7bf-b63c-4ac7-816d-530b7cf1457b] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:36.724 15080 ERROR oslo.messaging._drivers.impl_rabbit [req-04a181fe-45da-47ef-9496-e59f38516a68 - - - - -] [c4fbf101-f9d5-43ae-a8f2-cccbeee60651] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:36.789 15077 ERROR oslo.messaging._drivers.impl_rabbit [req-87b37edb-b721-4812-9a7d-5b5a084639e8 - - - - -] [4a5084b7-8eaa-4192-b5d4-3823b3d8dd04] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:41.977 15079 ERROR oslo.messaging._drivers.impl_rabbit [req-4bd8ffb9-adc5-4986-9458-b06cace9de09 - - - - -] [c3eb3c33-9084-47f5-bbaf-f1627ce6863e] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:42.077 15078 ERROR oslo.messaging._drivers.impl_rabbit [req-442bb7a9-f715-4af8-8d40-67387b3325e8 - - - - -] [2d5de7bf-b63c-4ac7-816d-530b7cf1457b] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:42.769 15080 ERROR oslo.messaging._drivers.impl_rabbit [req-04a181fe-45da-47ef-9496-e59f38516a68 - - - - -] [c4fbf101-f9d5-43ae-a8f2-cccbeee60651] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:42.842 15077 ERROR oslo.messaging._drivers.impl_rabbit [req-87b37edb-b721-4812-9a7d-5b5a084639e8 - - - - -] [4a5084b7-8eaa-4192-b5d4-3823b3d8dd04] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:50.007 15079 ERROR oslo.messaging._drivers.impl_rabbit [req-4bd8ffb9-adc5-4986-9458-b06cace9de09 - - - - -] [c3eb3c33-9084-47f5-bbaf-f1627ce6863e] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:50.124 15078 ERROR oslo.messaging._drivers.impl_rabbit [req-442bb7a9-f715-4af8-8d40-67387b3325e8 - - - - -] [2d5de7bf-b63c-4ac7-816d-530b7cf1457b] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:50.791 15080 ERROR oslo.messaging._drivers.impl_rabbit [req-04a181fe-45da-47ef-9496-e59f38516a68 - - - - -] [c4fbf101-f9d5-43ae-a8f2-cccbeee60651] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:50.895 15077 ERROR oslo.messaging._drivers.impl_rabbit [req-87b37edb-b721-4812-9a7d-5b5a084639e8 - - - - -] [4a5084b7-8eaa-4192-b5d4-3823b3d8dd04] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager [req-b5bcd4c5-a59e-4a9d-abea-89f140f6d47b 086d1437fd694fdfb826df6934e87775 5843f5028ae04d0d9db3471dd75b6eb0 - default default] Failed to schedule instances: NoValidHost_Remote: No valid host was found.
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager Traceback (most recent call last):
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 1231, in schedule_and_build_instances
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager     instance_uuids, return_alternates=True)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 748, in _schedule_instances
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager     return_alternates=return_alternates)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 907, in wrapped
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager     return func(*args, **kwargs)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 53, in select_destinations
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager     instance_uuids, return_objects, return_alternates)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager     return getattr(self.instance, __name)(*args, **kwargs)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager     instance_uuids, return_objects, return_alternates)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 158, in select_destinations
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager     return cctxt.call(ctxt, 'select_destinations', **msg_args)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager     retry=self.retry)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager     retry=retry)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager     call_monitor_timeout, retry=retry)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager     raise result
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found.
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager Traceback (most recent call last):
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager     return func(*args, **kwargs)
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 154, in select_destinations
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager     raise exception.NoValidHost(reason="")
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager NoValidHost: No valid host was found.
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager
2019-01-28 20:16:17.537 15078 ERROR nova.conductor.manager
2019-01-28 20:16:18.027 15078 WARNING nova.scheduler.utils [req-b5bcd4c5-a59e-4a9d-abea-89f140f6d47b 086d1437fd694fdfb826df6934e87775 5843f5028ae04d0d9db3471dd75b6eb0 - default default] [instance: 8844cca1-65ce-47b7-815d-edd9d5c13389] Setting instance to ERROR state.: NoValidHost_Remote: No valid host was found.
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager [req-2bbd6011-46a8-4333-a4f8-89450c19a9a2 086d1437fd694fdfb826df6934e87775 5843f5028ae04d0d9db3471dd75b6eb0 - default default] Failed to schedule instances: NoValidHost_Remote: No valid host was found.
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager Traceback (most recent call last):
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 1231, in schedule_and_build_instances
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager     instance_uuids, return_alternates=True)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 748, in _schedule_instances
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager     return_alternates=return_alternates)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 907, in wrapped
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager     return func(*args, **kwargs)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 53, in select_destinations
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager     instance_uuids, return_objects, return_alternates)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager     return getattr(self.instance, __name)(*args, **kwargs)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager     instance_uuids, return_objects, return_alternates)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 158, in select_destinations
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager     return cctxt.call(ctxt, 'select_destinations', **msg_args)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager     retry=self.retry)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager     retry=retry)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager     call_monitor_timeout, retry=retry)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager     raise result
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found.
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager Traceback (most recent call last):
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager     return func(*args, **kwargs)
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 154, in select_destinations
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager     raise exception.NoValidHost(reason="")
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager NoValidHost: No valid host was found.
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager
2019-01-28 20:27:10.662 15077 ERROR nova.conductor.manager
2019-01-28 20:27:10.886 15077 WARNING nova.scheduler.utils [req-2bbd6011-46a8-4333-a4f8-89450c19a9a2 086d1437fd694fdfb826df6934e87775 5843f5028ae04d0d9db3471dd75b6eb0 - default default] [instance: 30267bb9-e52e-4b71-8a8f-0800f1e81b4a] Setting instance to ERROR state.: NoValidHost_Remote: No valid host was found.
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines [-] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines     connection.scalar(select([1]))
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines     return self.execute(object, *multiparams, **params).scalar()
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines     return meth(self, multiparams, params)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines     return connection._execute_clauseelement(self, multiparams, params)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines     compiled_sql, distilled_params
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines     context)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines     util.raise_from_cause(newraise, exc_info)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines     context)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines     cursor.execute(statement, parameters)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines     result = self._query(query)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines     conn.query(q)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines     result.read()
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines     first_packet = self.connection._read_packet()
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 998, in _read_packet
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines     "Lost connection to MySQL server during query")
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)
2019-01-28 17:21:39.125 65362 ERROR oslo_db.sqlalchemy.engines
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db [-] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.43.110' ([Errno 111] ECONNREFUSED)") [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db Traceback (most recent call last):
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     service.service_ref.save()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     return fn(self, *args, **kwargs)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 396, in save
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     self._check_minimum_version()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 371, in _check_minimum_version
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     minver = self.get_minimum_version(self._context, self.binary)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     result = fn(cls, context, *args, **kwargs)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 475, in get_minimum_version
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     use_slave=use_slave)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     result = fn(cls, context, *args, **kwargs)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 452, in get_minimum_version_multi
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     context, binaries, use_slave=use_slave)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 210, in wrapper
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     return f(*args, **kwargs)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 438, in _db_service_get_minimum_version
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     return db.service_get_minimum_version(context, binaries)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 117, in service_get_minimum_version
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     return IMPL.service_get_minimum_version(context, binary)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     return f(context, *args, **kwargs)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 441, in service_get_minimum_version
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     return dict(min_versions)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, in __iter__
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     return self._execute_and_instances(context)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2946, in _execute_and_instances
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     close_with_result=True)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2955, in _get_bind_args
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     **kw
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2937, in _connection_from_session
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     conn = self.session.connection(**kw)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     execution_options=execution_options)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     engine, execution_options)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     conn = bind.contextual_connect()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2125, in contextual_connect
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     **kwargs)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 110, in __init__
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     self.dispatch.engine_connect(self, self.__branch)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 284, in __call__
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     fn(*args, **kw)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 83, in _connect_ping_listener
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     connection.scalar(select([1]))
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     return self.execute(object, *multiparams, **params).scalar()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     return meth(self, multiparams, params)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     return connection._execute_clauseelement(self, multiparams, params)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     compiled_sql, distilled_params
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1132, in _execute_context
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     None, None)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     util.raise_from_cause(newraise, exc_info)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1125, in _execute_context
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     conn = self._revalidate_connection()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 429, in _revalidate_connection
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     self.__connection = self.engine.raw_connection(_connection=self)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2188, in raw_connection
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     self.pool.unique_connection, _connection)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2164, in _wrap_pool_connect
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     util.reraise(*sys.exc_info())
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     return fn()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 345, in unique_connection
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     return _ConnectionFairy._checkout(self)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 788, in _checkout
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     fairy = _ConnectionRecord.checkout(pool)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     rec._checkin_failed(err)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     compat.reraise(exc_type, exc_value, exc_tb)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     dbapi_connection = rec.get_connection()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 627, in get_connection
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     self.__connect()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 671, in __connect
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     connection = pool._invoke_creator(self)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 106, in connect
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     return dialect.connect(*cargs, **cparams)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     return self.dbapi.connect(*cargs, **cparams)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     return Connection(*args, **kwargs)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     self.connect()
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db     raise exc
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.43.110' ([Errno 111] ECONNREFUSED)") [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)
2019-01-28 17:21:39.147 65362 ERROR nova.servicegroup.drivers.db
2019-01-28 19:50:22.956 10777 ERROR oslo.messaging._drivers.impl_rabbit [req-504e234e-1af3-4471-9893-30480786f439 - - - - -] [e21c5911-6553-47a8-bb5e-44fc20bd4c6d] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:24.019 10777 ERROR oslo.messaging._drivers.impl_rabbit [req-504e234e-1af3-4471-9893-30480786f439 - - - - -] [e21c5911-6553-47a8-bb5e-44fc20bd4c6d] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:26.067 10777 ERROR oslo.messaging._drivers.impl_rabbit [req-504e234e-1af3-4471-9893-30480786f439 - - - - -] [e21c5911-6553-47a8-bb5e-44fc20bd4c6d] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:30.129 10777 ERROR oslo.messaging._drivers.impl_rabbit [req-504e234e-1af3-4471-9893-30480786f439 - - - - -] [e21c5911-6553-47a8-bb5e-44fc20bd4c6d] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:36.181 10777 ERROR oslo.messaging._drivers.impl_rabbit [req-504e234e-1af3-4471-9893-30480786f439 - - - - -] [e21c5911-6553-47a8-bb5e-44fc20bd4c6d] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:44.242 10777 ERROR oslo.messaging._drivers.impl_rabbit [req-504e234e-1af3-4471-9893-30480786f439 - - - - -] [e21c5911-6553-47a8-bb5e-44fc20bd4c6d] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:57:39.426 10777 ERROR oslo.messaging._drivers.impl_rabbit [-] [e21c5911-6553-47a8-bb5e-44fc20bd4c6d] AMQP server 192.168.43.110:5672 closed the connection. Check login credentials: Socket closed: IOError: Socket closed
2019-01-28 19:50:36.887 15222 ERROR oslo.messaging._drivers.impl_rabbit [req-9094253e-624b-4654-9494-45da68c4e08e - - - - -] [b860dcb3-6a51-458a-92c5-55bdd08613bb] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:36.943 15221 ERROR oslo.messaging._drivers.impl_rabbit [req-dc40ab3d-f7e4-4ea7-bce1-fcd830d17f9a - - - - -] [386112ab-9467-4fa1-9e52-6ddd6a62a00a] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:37.933 15222 ERROR oslo.messaging._drivers.impl_rabbit [req-9094253e-624b-4654-9494-45da68c4e08e - - - - -] [b860dcb3-6a51-458a-92c5-55bdd08613bb] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:37.992 15221 ERROR oslo.messaging._drivers.impl_rabbit [req-dc40ab3d-f7e4-4ea7-bce1-fcd830d17f9a - - - - -] [386112ab-9467-4fa1-9e52-6ddd6a62a00a] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:39.983 15222 ERROR oslo.messaging._drivers.impl_rabbit [req-9094253e-624b-4654-9494-45da68c4e08e - - - - -] [b860dcb3-6a51-458a-92c5-55bdd08613bb] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:40.036 15221 ERROR oslo.messaging._drivers.impl_rabbit [req-dc40ab3d-f7e4-4ea7-bce1-fcd830d17f9a - - - - -] [386112ab-9467-4fa1-9e52-6ddd6a62a00a] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:44.034 15222 ERROR oslo.messaging._drivers.impl_rabbit [req-9094253e-624b-4654-9494-45da68c4e08e - - - - -] [b860dcb3-6a51-458a-92c5-55bdd08613bb] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:44.083 15221 ERROR oslo.messaging._drivers.impl_rabbit [req-dc40ab3d-f7e4-4ea7-bce1-fcd830d17f9a - - - - -] [386112ab-9467-4fa1-9e52-6ddd6a62a00a] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:50.060 15222 ERROR oslo.messaging._drivers.impl_rabbit [req-9094253e-624b-4654-9494-45da68c4e08e - - - - -] [b860dcb3-6a51-458a-92c5-55bdd08613bb] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:50:50.131 15221 ERROR oslo.messaging._drivers.impl_rabbit [req-dc40ab3d-f7e4-4ea7-bce1-fcd830d17f9a - - - - -] [386112ab-9467-4fa1-9e52-6ddd6a62a00a] AMQP server on 192.168.43.110:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds.: error: [Errno 111] ECONNREFUSED
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines [-] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines     connection.scalar(select([1]))
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines     return self.execute(object, *multiparams, **params).scalar()
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines     return meth(self, multiparams, params)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines     return connection._execute_clauseelement(self, multiparams, params)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines     compiled_sql, distilled_params
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines     context)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines     util.raise_from_cause(newraise, exc_info)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines     context)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines     cursor.execute(statement, parameters)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines     result = self._query(query)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines     conn.query(q)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines     result.read()
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines     first_packet = self.connection._read_packet()
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines     packet_header = self._read_bytes(4)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1033, in _read_bytes
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines     CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)
2019-01-28 19:57:38.688 15222 ERROR oslo_db.sqlalchemy.engines
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines [-] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines     connection.scalar(select([1]))
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines     return self.execute(object, *multiparams, **params).scalar()
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines     return meth(self, multiparams, params)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines     return connection._execute_clauseelement(self, multiparams, params)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines     compiled_sql, distilled_params
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines     context)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines     util.raise_from_cause(newraise, exc_info)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines     context)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines     cursor.execute(statement, parameters)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines     result = self._query(query)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines     conn.query(q)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines     result.read()
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines     first_packet = self.connection._read_packet()
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines     packet_header = self._read_bytes(4)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1033, in _read_bytes
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines     CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)
2019-01-28 19:57:39.807 15221 ERROR oslo_db.sqlalchemy.engines
2019-01-28 19:57:58.181 15222 ERROR oslo.messaging._drivers.impl_rabbit [-] [b860dcb3-6a51-458a-92c5-55bdd08613bb] AMQP server 192.168.43.110:5672 closed the connection. Check login credentials: Socket closed: IOError: Socket closed
2019-01-28 19:57:58.247 15221 ERROR oslo.messaging._drivers.impl_rabbit [-] [386112ab-9467-4fa1-9e52-6ddd6a62a00a] AMQP server 192.168.43.110:5672 closed the connection. Check login credentials: Socket closed: IOError: Socket closed
[root@amer ~(keystone_admin)]#

any one can interpret it to me, mysql is gone some times and rabbit timedout, swift is down, and nova not creating instances...