KeyError: 'compute02' on global-manager-service.log

[Albert Vonpupp]

Hello Anton,

I've successfully deployed DevStack, and almost, OpenStack Neat too but I'm having some problems [1]. I have basically 2 compute nodes (compute01 and compute02) and a controller, all of the using Fedora 17. I'm using Python 2.7.3.

[root@controller ~]# cat /var/log/neat/global-manager-service.log [1]
Traceback (most recent call last):
  File "/usr/bin/neat-global-manager", line 9, in <module>
    load_entry_point('openstack-neat==0.1', 'console_scripts', 'neat-global-manager')()
  File "/usr/lib/python2.7/site-packages/openstack_neat-0.1-py2.7.egg/neat/globals/manager.py", line 169, in start
    state['compute_hosts'])
  File "<string>", line 2, in switch_hosts_on
  File "/usr/lib/python2.7/site-packages/contracts/main.py", line 296, in contracts_checker
    result = function_(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/openstack_neat-0.1-py2.7.egg/neat/globals/manager.py", line 868, in switch_hosts_on
    db.insert_host_states(dict((x, 1) for x in hosts))
  File "<string>", line 2, in insert_host_states
  File "/usr/lib/python2.7/site-packages/contracts/main.py", line 296, in contracts_checker
    result = function_(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/openstack_neat-0.1-py2.7.egg/neat/db.py", line 320, in insert_host_states
    for k, v in hosts.items()]
KeyError: 'compute02'

More logs on the controller:

[root@controller ~]# cat /var/log/neat/global-manager.log
2013-09-24 15:32:16,460 DEBUG    neat.db Instantiated a Database object
2013-09-24 15:32:16,460 DEBUG    neat.db_utils Initialized a DB connection to mysql://root:badpa...@marte.eclipse.ime.usp.br/neat
2013-09-24 15:32:16,475 DEBUG    neat.globals.manager Calling: ether-wake -i em1 00:1c:c0:c3:f3:1f
2013-09-24 15:32:16,495 DEBUG    neat.globals.manager Calling: ether-wake -i em1 00:27:0e:23:06:e9
2013-09-24 15:32:16,506 INFO     neat.globals.manager Switched on hosts: ['compute01', 'compute02']

[root@controller ~]# cat /var/log/neat/db-cleaner.log
2013-09-24 15:32:16,380 INFO     neat.globals.db_cleaner Starting the database cleaner, iterations every 7200 seconds
2013-09-24 15:32:16,410 DEBUG    neat.db Instantiated a Database object
2013-09-24 15:32:16,410 DEBUG    neat.db_utils Initialized a DB connection to mysql://root:badpa...@marte.eclipse.ime.usp.br/neat
2013-09-24 15:32:16,412 INFO     neat.globals.db_cleaner Cleaned up data older than 2013-09-24 13:32:16

[root@controller ~]# cat /var/log/neat/db-cleaner-service.log
(empty)

More logs on the compute02:

[root@compute02 ~]# cat /var/log/neat/local-manager.log
2013-09-24 15:32:18,596 INFO     neat.locals.manager Starting the local manager, iterations every 300 seconds
2013-09-24 15:32:18,684 DEBUG    neat.db Instantiated a Database object
2013-09-24 15:32:18,684 DEBUG    neat.db_utils Initialized a DB connection to mysql://root:badpa...@marte.eclipse.ime.usp.br/neat
2013-09-24 15:32:18,686 INFO     neat.locals.manager Started an iteration
2013-09-24 15:32:18,687 INFO     neat.locals.manager The host is idle
2013-09-24 15:32:18,687 INFO     neat.locals.manager Skipped an iteration
2013-09-24 15:37:18,689 INFO     neat.locals.manager Started an iteration
2013-09-24 15:37:18,690 INFO     neat.locals.manager The host is idle
2013-09-24 15:37:18,690 INFO     neat.locals.manager Skipped an iteration

[root@compute02 ~]# cat /var/log/neat/local-manager-service.log
(empty)

[root@compute02 ~]# cat /var/log/neat/data-collector.log
2013-09-24 15:32:15,409 INFO     neat.locals.collector Creaned up the local data directory: /var/lib/neat
2013-09-24 15:32:15,409 INFO     neat.locals.collector Starting the data collector, iterations every 300 seconds
2013-09-24 15:32:15,488 DEBUG    neat.db Instantiated a Database object
2013-09-24 15:32:15,488 DEBUG    neat.db_utils Initialized a DB connection to mysql://root:badpa...@marte.eclipse.ime.usp.br/neat

[root@compute02 ~]# cat /var/log/neat/data-collector-service.log
Traceback (most recent call last):
  File "/usr/bin/neat-data-collector", line 9, in <module>
    load_entry_point('openstack-neat==0.1', 'console_scripts', 'neat-data-collector')()
  File "<string>", line 2, in start
  File "/usr/lib/python2.7/site-packages/contracts/main.py", line 296, in contracts_checker
    result = function_(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/openstack_neat-0.1-py2.7.egg/neat/locals/collector.py", line 140, in start
    int(interval))
  File "<string>", line 2, in start
  File "/usr/lib/python2.7/site-packages/contracts/main.py", line 296, in contracts_checker
    result = function_(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/openstack_neat-0.1-py2.7.egg/neat/common.py", line 58, in start
    state = init_state(config)
  File "<string>", line 2, in init_state
  File "/usr/lib/python2.7/site-packages/contracts/main.py", line 296, in contracts_checker
    result = function_(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/openstack_neat-0.1-py2.7.egg/neat/locals/collector.py", line 168, in init_state
    host_ram)
  File "<string>", line 2, in update_host
  File "/usr/lib/python2.7/site-packages/contracts/main.py", line 305, in contracts_checker
    raise e
contracts.interface.ContractNotRespected: Breach for return value of Database:update_host().
Could not satisfy any of the 3 clauses in float|int|uint.
 ---- Clause #1:   float
 | Could not satisfy any of the 3 clauses in Float|np_scalar_float|np_scalar,array(float).
 |  ---- Clause #1:   Float
 |  | Expected type 'float', got 'long'.
 |  | checking: Float   for value: Instance of long: 3L  
 |  | Variables bound in inner context:
 |  | - self: Instance of Database: <neat.db.Database object at 0x2712a90>
 |  ---- Clause #2:   np_scalar_float
 |  | Could not satisfy any of the 2 clauses in np_float32|np_float64.
 |  |  ---- Clause #1:   np_float32
 |  |  | Expected type 'float32', got 'long'.
 |  |  | checking: np_float32   for value: Instance of long: 3L  
 |  |  | Variables bound in inner context:
... so on ...

I couldn't find a solution on my own for this issue. Could you please help me out?

I will also take the opportunity to ask some questions:
- If I start Neat, how much time it would last until it will pm-suspend the compute nodes due to inactivity?
- Is it possible to change the algorithm that chooses the next underloaded host where to migrate? For testing purposes I would like to change that algorithm to "random" to see what happens.
- Do you have any script that customizes OpenStack? i.e: creates the flavor, image, etc. I did everything by hand but since I'm using DevStack changes are not persistent, I'm working on that already but if you do have something it would be helpful.

Many thanks.

Regards,
Albert.

[Albert Vonpupp]

I forgot to include my /etc/hosts, which could be important too:

[root@controller ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
143.107.45.200  marte.eclipse.ime.usp.br marte controller
143.107.45.201  jupiter.eclipse.ime.usp.br jupiter compute01
143.107.45.202  saturno.eclipse.ime.usp.br saturno compute02

Thanks.

Regards,
Albert.

On Tuesday, September 24, 2013 3:58:46 PM UTC-3, Albert Vonpupp wrote:

Hello Anton,

I've successfully deployed DevStack, and almost, OpenStack Neat too but I'm having some problems [1]. I have basically 2 compute nodes (compute01 and compute02) and a controller, all of the using Fedora 17. I'm using Python 2.7.3.

[root@controller ~]# cat /var/log/neat/global-manager-service.log [1]
Traceback (most recent call last):
  File "/usr/bin/neat-global-manager", line 9, in <module>
    load_entry_point('openstack-neat==0.1', 'console_scripts', 'neat-global-manager')()
  File "/usr/lib/python2.7/site-packages/openstack_neat-0.1-py2.7.egg/neat/globals/manager.py", line 169, in start
    state['compute_hosts'])
  File "<string>", line 2, in switch_hosts_on
  File "/usr/lib/python2.7/site-packages/contracts/main.py", line 296, in contracts_checker
    result = function_(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/openstack_neat-0.1-py2.7.egg/neat/globals/manager.py", line 868, in switch_hosts_on
    db.insert_host_states(dict((x, 1) for x in hosts))
  File "<string>", line 2, in insert_host_states
  File "/usr/lib/python2.7/site-packages/contracts/main.py", line 296, in contracts_checker
    result = function_(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/openstack_neat-0.1-py2.7.egg/neat/db.py", line 320, in insert_host_states
    for k, v in hosts.items()]
KeyError: 'compute02'

More logs on the controller:

[root@controller ~]# cat /var/log/neat/global-manager.log
2013-09-24 15:32:16,460 DEBUG    neat.db Instantiated a Database object

2013-09-24 15:32:16,460 DEBUG    neat.db_utils Initialized a DB connection to mysql://root:badpassword@marte.eclipse.ime.usp.br/neat

2013-09-24 15:32:16,475 DEBUG    neat.globals.manager Calling: ether-wake -i em1 00:1c:c0:c3:f3:1f
2013-09-24 15:32:16,495 DEBUG    neat.globals.manager Calling: ether-wake -i em1 00:27:0e:23:06:e9
2013-09-24 15:32:16,506 INFO     neat.globals.manager Switched on hosts: ['compute01', 'compute02']

[root@controller ~]# cat /var/log/neat/db-cleaner.log
2013-09-24 15:32:16,380 INFO     neat.globals.db_cleaner Starting the database cleaner, iterations every 7200 seconds
2013-09-24 15:32:16,410 DEBUG    neat.db Instantiated a Database object

2013-09-24 15:32:16,410 DEBUG    neat.db_utils Initialized a DB connection to mysql://root:badpassword@marte.eclipse.ime.usp.br/neat

2013-09-24 15:32:16,412 INFO     neat.globals.db_cleaner Cleaned up data older than 2013-09-24 13:32:16

[root@controller ~]# cat /var/log/neat/db-cleaner-service.log
(empty)

More logs on the compute02:

[root@compute02 ~]# cat /var/log/neat/local-manager.log
2013-09-24 15:32:18,596 INFO     neat.locals.manager Starting the local manager, iterations every 300 seconds
2013-09-24 15:32:18,684 DEBUG    neat.db Instantiated a Database object

2013-09-24 15:32:18,684 DEBUG    neat.db_utils Initialized a DB connection to mysql://root:badpassword@marte.eclipse.ime.usp.br/neat

2013-09-24 15:32:18,686 INFO     neat.locals.manager Started an iteration
2013-09-24 15:32:18,687 INFO     neat.locals.manager The host is idle
2013-09-24 15:32:18,687 INFO     neat.locals.manager Skipped an iteration
2013-09-24 15:37:18,689 INFO     neat.locals.manager Started an iteration
2013-09-24 15:37:18,690 INFO     neat.locals.manager The host is idle
2013-09-24 15:37:18,690 INFO     neat.locals.manager Skipped an iteration

[root@compute02 ~]# cat /var/log/neat/local-manager-service.log
(empty)

[root@compute02 ~]# cat /var/log/neat/data-collector.log
2013-09-24 15:32:15,409 INFO     neat.locals.collector Creaned up the local data directory: /var/lib/neat
2013-09-24 15:32:15,409 INFO     neat.locals.collector Starting the data collector, iterations every 300 seconds
2013-09-24 15:32:15,488 DEBUG    neat.db Instantiated a Database object

2013-09-24 15:32:15,488 DEBUG    neat.db_utils Initialized a DB connection to mysql://root:badpassword@marte.eclipse.ime.usp.br/neat

[Albert Vonpupp]

I think I manage to solve that issue,

Neat needs hosts without FQDN, now this is the output:
[stack@marte ~]$ cat /var/log/neat/*
2013-09-26 15:57:49,083 INFO     neat.globals.db_cleaner Starting the database cleaner, iterations every 7200 seconds
2013-09-26 15:59:18,272 INFO     neat.globals.db_cleaner Starting the database cleaner, iterations every 7200 seconds
2013-09-26 15:59:18,460 DEBUG    neat.db Instantiated a Database object
2013-09-26 15:59:18,461 DEBUG    neat.db_utils Initialized a DB connection to mysql://root:badpa...@marte.eclipse.ime.usp.br/neat
2013-09-26 15:59:18,464 INFO     neat.globals.db_cleaner Cleaned up data older than 2013-09-26 13:59:18
2013-09-26 16:03:44,955 INFO     neat.globals.db_cleaner Starting the database cleaner, iterations every 7200 seconds
2013-09-26 16:03:45,142 DEBUG    neat.db Instantiated a Database object
2013-09-26 16:03:45,143 DEBUG    neat.db_utils Initialized a DB connection to mysql://root:badpa...@marte.eclipse.ime.usp.br/neat
2013-09-26 16:03:45,146 INFO     neat.globals.db_cleaner Cleaned up data older than 2013-09-26 14:03:45
2013-09-26 15:59:18,521 DEBUG    neat.db Instantiated a Database object
2013-09-26 15:59:18,521 DEBUG    neat.db_utils Initialized a DB connection to mysql://root:badpa...@marte.eclipse.ime.usp.br/neat
2013-09-26 15:59:18,538 DEBUG    neat.globals.manager Calling: ether-wake -i em1 00:1c:c0:c3:f3:1f
2013-09-26 15:59:18,563 DEBUG    neat.globals.manager Calling: ether-wake -i em1 00:27:0e:23:06:e9
2013-09-26 15:59:18,583 DEBUG    neat.globals.manager Calling: ether-wake -i em1 70:71:bc:08:55:eb
2013-09-26 15:59:18,597 INFO     neat.globals.manager Switched on hosts: ['jupiter', 'saturno', 'venus']
2013-09-26 16:03:45,187 DEBUG    neat.db Instantiated a Database object
2013-09-26 16:03:45,187 DEBUG    neat.db_utils Initialized a DB connection to mysql://root:badpa...@marte.eclipse.ime.usp.br/neat
2013-09-26 16:03:45,208 DEBUG    neat.globals.manager Calling: ether-wake -i em1 00:1c:c0:c3:f3:1f
2013-09-26 16:03:45,232 DEBUG    neat.globals.manager Calling: ether-wake -i em1 00:27:0e:23:06:e9
2013-09-26 16:03:45,256 DEBUG    neat.globals.manager Calling: ether-wake -i em1 70:71:bc:08:55:eb
2013-09-26 16:03:45,271 INFO     neat.globals.manager Switched on hosts: ['jupiter', 'saturno', 'venus']
2013-09-26 16:03:45,454 INFO     neat.globals.manager Starting the global manager listening to marte.eclipse.ime.usp.br:60080
Bottle v0.11.6 server starting up (using WSGIRefServer())...
Listening on http://marte.eclipse.ime.usp.br:60080/
Hit Ctrl-C to quit.

I started 2 VMs (pico - 128Mb). By having 3 compute nodes I expect that they should be migrated to a single compute node, is that right? If so:
- How can I check if that really happened? The logs will contain when the compute nodes will be suspended?
- How much time do I have to wait until the first suspend?

Thanks a lot!

[Anton Beloglazov]

Hi Albert,

Sorry again for not replying to you quickly: had a really busy week. I'm glad you've solved the host name issue. Regarding the other questions:

1. There is a script called vm-placement.py, which based on the config and information from the database, displays all the hosts with VMs allocated to them together with the current resource usage information. You can get real-time updates on the current VM placement and resource usage by running this script with watch (that's what I did) like so: watch ./vm-placement.py

2. By default the local_manager_interval parameter in neat.conf is set to 5 minutes, which means that local managers will run overload/underload detection algorithms every 5 minutes. Once underload conditions are detected on a host, the global manager will migrate the VM from it to other hosts (if needed) and then suspend the host. This means that if a host was idle from the beginning, it should be switched to the sleep mode in around 5 minutes.

Please let me know if you have other questions.

Cheers,

Anton

On Friday, 27 September 2013 05:16:50 UTC+10, Albert Vonpupp wrote:

I think I manage to solve that issue,

Neat needs hosts without FQDN, now this is the output:
[stack@marte ~]$ cat /var/log/neat/*
2013-09-26 15:57:49,083 INFO     neat.globals.db_cleaner Starting the database cleaner, iterations every 7200 seconds
2013-09-26 15:59:18,272 INFO     neat.globals.db_cleaner Starting the database cleaner, iterations every 7200 seconds
2013-09-26 15:59:18,460 DEBUG    neat.db Instantiated a Database object

2013-09-26 15:59:18,461 DEBUG    neat.db_utils Initialized a DB connection to mysql://root:badpassword@marte.eclipse.ime.usp.br/neat

2013-09-26 15:59:18,464 INFO     neat.globals.db_cleaner Cleaned up data older than 2013-09-26 13:59:18
2013-09-26 16:03:44,955 INFO     neat.globals.db_cleaner Starting the database cleaner, iterations every 7200 seconds
2013-09-26 16:03:45,142 DEBUG    neat.db Instantiated a Database object

2013-09-26 16:03:45,143 DEBUG    neat.db_utils Initialized a DB connection to mysql://root:badpassword@marte.eclipse.ime.usp.br/neat

2013-09-26 16:03:45,146 INFO     neat.globals.db_cleaner Cleaned up data older than 2013-09-26 14:03:45
2013-09-26 15:59:18,521 DEBUG    neat.db Instantiated a Database object

2013-09-26 15:59:18,521 DEBUG    neat.db_utils Initialized a DB connection to mysql://root:badpassword@marte.eclipse.ime.usp.br/neat

2013-09-26 15:59:18,538 DEBUG    neat.globals.manager Calling: ether-wake -i em1 00:1c:c0:c3:f3:1f
2013-09-26 15:59:18,563 DEBUG    neat.globals.manager Calling: ether-wake -i em1 00:27:0e:23:06:e9
2013-09-26 15:59:18,583 DEBUG    neat.globals.manager Calling: ether-wake -i em1 70:71:bc:08:55:eb
2013-09-26 15:59:18,597 INFO     neat.globals.manager Switched on hosts: ['jupiter', 'saturno', 'venus']
2013-09-26 16:03:45,187 DEBUG    neat.db Instantiated a Database object

2013-09-26 16:03:45,187 DEBUG    neat.db_utils Initialized a DB connection to mysql://root:badpassword@marte.eclipse.ime.usp.br/neat

[Anton Beloglazov]

You can change the VM placement algorithm by setting the algorithm_vm_placement_factory configuration option. However, currently it's not possible to set different algorithm for VM placement from underloaded and overloaded hosts. This is not too hard to implement, just requires an extra option in the config file and changing the global manager to call the appropriate function in these two situations.

Unfortunately, I don't have any scripts for creating flavors, images, etc. There are scripts for selecting, distributing, and starting workloads on VMs in this repository: https://github.com/beloglazov/spe-2013-experiments The workload-starter.py script should be deployed, configured with the server IP, and automatically started in the VM image on boot. This way you can create a single image, and then create multiple instances which will automatically request the server for the workload. Then, with the workload-distributor.py script you can distribute workload traces to all the VMs at the same time.

Cheers,

Anton

[Kashyap Raiyani]

Hi Albert,

I am also facing same problems and having same log files as yours can you please tell me that how did you solve those errors https://groups.google.com/forum/#!topic/openstack-neat/2CmdlFIVggc.

Thanking you,
Kashyap