Starting Openstack VMs in parallel

[Kurt Yoder]

Hello list,

I anticipate provisioning 10-20 VMs using Ansible, then assigning floating IPs to each, then waiting for SSH to become available for each VM. I would like to do this in parallel instead of serially. Specifically:

• Start the VMs, but don't block
• Assign the IPs, but don't block
• Wait on SSH until all VMs respond

I saw the nova_compute "wait: 'no'" option, but when I use it I get a stack trace:

failed: [localhost] => (item=1) => {"failed": true, "item": 1, "parsed": false}
invalid output was: Traceback (most recent call last):
  File "/home/ubuntu/.ansible/tmp/ansible-tmp-1405028178.0-234314980043958/nova_compute", line 1490, in <module>
    main()
  File "/home/ubuntu/.ansible/tmp/ansible-tmp-1405028178.0-234314980043958/nova_compute", line 266, in main
    _create_server(module, nova)
  File "/home/ubuntu/.ansible/tmp/ansible-tmp-1405028178.0-234314980043958/nova_compute", line 194, in _create_server
    private = [ x['addr'] for x in getattr(server, 'addresses').itervalues().next() if x['OS-EXT-IPS:type'] == 'fixed']
StopIteration

Perhaps I'm using it incorrectly:

- name: Launch cluster VM on Openstack
  nova_compute:
    name: "{{ os_username }}_cluster1"
    state: present
    login_username: "{{ os_username }}"
    login_tenant_name: "{{ os_tenant }}"
    login_password: "{{ os_password }}"
    image_id: "{{ os_image_id }}"
    key_name: "{{ os_username }}_controller_key"
    wait: "no"
    flavor_id: "{{ os_flavor_id }}"
    auth_url: "{{ os_url }}"
    user_data: "#cloud-config\nmanage_etc_hosts: true"

So, two questions:

1. Am I using "wait" correctly?
2. Should I use "wait" to get to my desired parallel VM launch, as described above, or should I use something else, e.g. "async"?

Thanks,

-Kurt

[Kurt Yoder]

Well, "async" is totally a bust. I got a message:

fatal: [localhost] => lookup plugins (with_*) cannot be used with async tasks

[Kurt Yoder]

I guess that error is because I put a "with_items" in there.

How does everyone else do this? I don't understand how to loop asynchronously. See pseudo-code:

< start all 5 at once: >

   < start up openstack host >

   < assign it a floating ip >

   < capture the floating ip >

< end when all 5 have a floating ip >

< wait for all 5 floating IPs to have an open SSH port >

[Michael DeHaan]

So some of the provisioning modules, like AWS in particular, support spinning up "N" modules at a time by passing "count" or "exact_count".

Rackspace I believe does this with manual looping (for now), but I could be wrong and that might have just been historical truth.

If the OpenStack API says we can launch 10 at once, it could be made to do similar things.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/32b58cdf-17cd-42c9-8ef6-dc90327e989a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Kurt Yoder]

So should I make a custom module which loops over nova_compute asynchronously, and also assigns floating IPs?

Would such a module be useful to the wider community, or is it too specialized to contribute back?

[Michael DeHaan]

"So should I make a custom module which loops over nova_compute asynchronously, and also assigns floating IPs?"

I'd first rather know where the openstack API allows simultaneous creation of N virtual machines of the same image type.

I expect the floating IP stuff is fast and a usual with_items loop isn't a problem there, once those guests exist.

(Using neutron, I assume?)

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/2326c06a-dec5-43b2-bd22-bb6856d14227%40googlegroups.com.

[Kurt Yoder]

There's an undocumented min_count and max_count option in python-novaclient ServerManager.create() which looks like what you want [1].

-Kurt

[1] https://github.com/openstack/python-novaclient/blob/cc7364067f995bd952149cd172cebf083fe4dc99/novaclient/v3/servers.py#L427

[Kurt Yoder]

To answer your other question about neutron: I use "quantum_floating_ip" to assign IPs on Openstack.

It would be very useful if these API calls could also be parallelized. I imagine a lot more time would be spent on 100 serial API calls than doing them in parallel.

-Kurt

[Michael DeHaan]

Yep, I think we'd be open to that.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/a1fc1c1c-a6ca-42ab-8296-a83bf447d2c2%40googlegroups.com.

[Kurt Yoder]

Is this a "pull requests welcome" situation? If so, I can try my best...

:D

[Kurt Yoder]

Opened two github issues:

• https://github.com/ansible/ansible/issues/8131
  
• https://github.com/ansible/ansible/issues/8132
  

[Michael DeHaan]

Minor process note:

If you're going to submit the code to try, don't open the feature requests, just clutters up the queue.

If you aren't, opening an RFE usually doesn't mean the code fairies implement it anyway, since we're likely going to attend to existing bug reports/PRs and feature PRs first

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/21bb9dba-6dc6-4ad6-95ed-8a99898ac953%40googlegroups.com.

[Kurt Yoder]

I dug a bit further. The API does allow min_count and max_count, much the way boto does for AWS.

When you submit a request with min_count, your instances are named <ansible-provide name>-<instance UUID>. That's acceptable, though not ideal.

I'm taking a look at the Ansible ec2 module, and the code for boto instance-launching code looks very different than the nova_compute instance-launching code. I haven't run it yet; I need to dig around to find my ec2 creds so I can run a test.

The Ansible ec2 module also allows one to assign public IPs while launching multiple instances. The Ansible nova_compute module does not permit this ATM.

[Michael DeHaan]

Additions of new params to add IP spawning behavior would be reasonable.

(assign_public_ip, True/False, etc)

What might you prefer on names?

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/2949ac2a-e83e-4074-bc26-02652b90e605%40googlegroups.com.

[Kurt Yoder]

I found a different approach to this problem. 

First create a cluster configuration file.

$ cat cluster.yml
---
cluster:
  - cluster1
  - cluster2

Define the first task: setting up one connection to localhost for each API call. 

- include_vars: cluster.yml
 
- add_host:
    name: "os_api_{{ item }}"
    ansible_ssh_host: 127.0.0.1
    groups: os_api
    ansible_connection: local
    oshost: "{{ item }}"
  with_items: cluster

Define a follow-on task: tell Openstack to start up the VMs.

- name: Show host name

  debug:

    msg: "API connection: os_api_{{ oshost }}; Openstack host: {{ oshost }}"

- name: Launch cluster VM on Openstack

  nova_compute:

    name: "{{ os_username }}_{{ oshost }}"

    state: present

    login_username: "{{ os_username }}"

    login_tenant_name: "{{ os_tenant }}"

    login_password: "{{ os_password }}"

    image_id: "{{ os_image_id }}"

    key_name: "{{ os_username }}_controller_key"

    wait_for: 200

    flavor_id: "{{ os_flavor_id }}"

    auth_url: "{{ os_url }}"

        user_data: "#cloud-config\nmanage_etc_hosts: true"

- name: Assign IP address to cluster VM

  quantum_floating_ip:

    state: present

    login_username: "{{ os_username }}"

    login_password: "{{ os_password }}"

    login_tenant_name: "{{ os_tenant }}"

    network_name: "{{ os_network_name }}"

    instance_name: "{{ os_username }}_{{ oshost }}"

    internal_network_name: "{{ os_internal_network_name }}"

    auth_url: "{{ os_url }}"

  register: quantum_info

- name: Wait for cluster SSH to become available

  wait_for:

    port: 22

    host: "{{ quantum_info.public_ip }}"

    timeout: 180

    state: started

This method gives the following benefits:

• I can define lots of flavors and images in my cluster.yml definition.
• I can launch them all in parallel using Ansible's built-in, robust parallel execution. 
• I have access to all of Ansible's primitives while doing so, so I can build in *any* custom logic.
  
• No need to add options to nova_compute.

Overall, I'm extremely happy with this solution. To reiterate: *no code changes are required*!

[Kurt Yoder]

I forgot to include a critical piece of this solution. Here's how you invoke the task to tell Openstack to start the VMs:

- name: Create cluster
  hosts: os_api
  gather_facts: no
  roles: [instantiate] # "instantiate" is a role that includes the Openstack startup task

Perhaps this kind of example should be included in docs for parallel execution in Ansible. Let me know if you want me to help with that.

-Kurt

[Tony Kinsley]

So I currently have an inventory script that creates me "fake" devices which I run the nova_compute task against. This allows me to provision all the vms in parallel using the native parallelization that ansible provides which is nice cause I can set it up to only provision 10 - 20 vms at a time using the serial flag. 

So this method provisions the VMs great, but since the "fake" devices do not have connection information I cannot use any other modules later in the playbook. I am trying to add these newly created VMs to a new group using the add_host module but I cannot get it to add more than one device to the running inventory.

- name: spawn vms
  local_action:
    module: nova_compute
    name: {{ vm_name }}
    ...
  register: nova

- name: add new hosts
  local_action:
    module: add_hosts
    name: {{ nova.private_ip }}

I saw the fake device approach on another thread and liked the ability to use ansible to throttle my provisioning as well as choose the names for the vms. I was wondering though if it would be possible to update host with the connection information ( specifically the ansible_ssh_host, ansible_ssh_private_key_file, and whatever variable specifies how to connect to the host ) rather than creating new hosts. 

I also am stuck trying to get add_hosts to get all of my hosts. I assumed that add_hosts would also run in parallel but after searching around all of the examples I have seen show it using a loop. If I need to use a loop is there a good way to get the registered output from nova into a list rather than per host? I could not figure out any way to use set_fact to append the per host values of the registered nova variable into a single variable.

I felt like this approach was pretty clean, but I am definitely open to other ideas. My goal is to be able to provision a scalability test environment so I am expecting to be provisioning 500 VMs in a private Openstack network. 

Tony

[Michael DeHaan]

I'm not sure why you think add_host needs to run in parallel, it completes in almost zero time since it's just doing some internal inventory "math".

That all being said, what we really want is OpenStack's module having the equivalent of exact_count in AWS/Rax other (though the Rackspace implementation is a bit more low level).

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/13cc7a2e-bbb5-49b6-9ab3-ac1893c36d93%40googlegroups.com.

[Bob Tiernay]

Hey Kurt,

Do you happen to have a full example in a repo somewhere? I would be very interested to check it out.

Thanks in advance,

Bob

[Bob Tiernay]

FYI - http://docs.openstack.org/api/openstack-compute/2/content/POST_multiple-create-v2_createServer__v2__tenant_id__servers_ext-os-multi-server-create.html

To get exact_count semantics, set min_count and max_count to the same value. This is part of the public OpenStack API.

Cheers,

Bob

[Bob Tiernay]

Hi Tony,

Does this example do what you want? 

https://github.com/ostack/ansible-example/blob/d7849bf464399e5078a1fe7d79002a8710293fa8/playbooks/deploy.yaml

It basically starts with the add_hosts and then uses that as the basis for executing nova_compute, etc. It is set to be serial in this example, but I don't know the rationale there.

I'm just getting started with both OpenStack and Ansible and I'm finding there aren't many high quality examples in this space.  

[Kurt Yoder]

At the moment my code is bound up in a playbook that I can't share. Sorry.