Dynamic/complex inventory - Specific reboot order

[Patrick Black]

So this is a pretty specific case but I am pretty lost on the best way to approach it. Hopefully the hive mind can give me some guidance here.

I work for an MSP and each customer has specific reboot order requirements due to the specific applications we host for them.

For instance;

I have the above .yml variable file that defines the groups for each customer. It then applies those groups as tags on each server in vCenter.

This allows servers to be placed in keyed groups using the VMware dynamic inventory plugin. So that's all working fantastic.

My problem now is the best way to go about making sure the tasks that need to be run the on the servers (for example, a reboot), are executed precisely in the order of that group. Of course, I could manually specify that, but at the scale (think thousands of servers) with dozens of groups, is not practical.

Does anyone have recommendations on the best way to approach this? I'd like it to be scalable to multiple different customers with different grouping requirements without having to manually specify this that would be most ideal.

Thanks in advance and let me know if you need any additional information.

[Todd Lewis]

First, let's get some text instead of pixels. You've got

---

# example.yml

customer_name:
  database_server:
    - down_group: 3
    - up_group: 1
    - vcenter: vcenter.local
    - exclude: false
  file_server:
    - down_group: 2
    - up_group: 3
    - vcenter: vcenter.local
    - exclude: false
  print_server:
    - down_group: 1
    - up_group: 2
    - vcenter: vcenter.local
    - exclude: false

Is "customer_name" a placeholder for things like "AcmeCo", and are there more sections similar to this for other customers in the same file, or are they in different files?

I take it for this example you want to down the servers in ("print_server", "file_server", "database_ server") order, then bring them back up in ("database_server", "print_server", "file_server") order?

I'm curious why you've got each of your "server types" composed of a list of single entry dicts rather than simply dicts. (It would look exactly the same as above, but without the dashes.) Is that data design attractive for other reasons, and would you be willing to rework it if some other structure proved more practical?

[Patrick Black]

Hey Todd,

Thanks for the reply. To answer your questions:

1. I went with that particular structure mostly due to some previous attempts at this setup, the plan would be 1 customer per file. The `customer_name` portion could certainly be omitted.
2. Each *_server entry was intended to represent a single server in this example, not a group.
3. The other reason I included the customer_name portion at the top was as a simple way to differentiate which server the customer belongs to (since I am using dynamic inventory). I also have a tag and matching keyed group, so I could certainly use that instead.

So based on your recommendations and to avoid confusion:

# example.yml
database_server_01:
  down_group: 3
  up_group: 1
  vcenter: vcenter.local
  exclude: false
database_server_02:
  down_group: 3
  up_group: 2
  vcenter: vcenter.local
  exclude: false
file_server_01:
  down_group: 2
  up_group: 3
  vcenter: vcenter.local
  exclude: false
print_server_01:
  down_group: 1
  up_group: 2
  vcenter: vcenter.local
  exclude: false

The order is unfortunately not as simple as groups of servers, but rather each server individually has a specific “power up” or “power down” group. An unfortunate holdover from an archaic software that also prevents me from simply using a reboot or win_reboot command to handle this more cleanly.

[Todd Lewis]

That clarifies things a bit, for me at least. Before we get back to the original question, where's the Source of Truth? Are these YAML files generated from the tags in vcenter, or are the vcenter tags set based on these files? Or [gulp] are they kept in sync manually?

Help me understand your original question, which was, "My problem now is the best way to go about making sure the tasks that need to be run the on the servers (for example, a reboot), are executed precisely in the order of that group." Can you expand on that scenario? For example, if instead of having Ansible you had an admin sitting by a phone. The phone rings and customer "example" requests a reboot. What exactly does that request include? Based on that request and access to the file(s) like the one above, how does your admin decide what steps to take and in what order? Don't hesitate to include obvious details; they're only obvious to you! :) Explain it like you would to a new intern who's only here because his Play-Doh dried out. I can't automate what I can't explain, and right now I can't explain your process. Feel free to throw in other use cases / scenarios. I've got scroll bars and I'm not afraid to use them.

[Walter Rowe]

I would create one vars file per customer and use a variable that sources the customer's vars file at run time. You also could have a folder per customer that holds all customer specific items.

% ansible-playbook -e customer_name='customer_A'

In the playbook source customer_A's vars file.

vars:

  server_groups: "{{ lookup('files', 'path/to/vars/' + {{ customer_name }} + '.yml') | from_yaml }}"

OR

vars:

  server_groups: "{{ lookup('files', 'path/to/vars/' + {{ customer_name }} + '/server_groups.yml') | from_yaml }}"

This loads your customer-specific YAML file as a dictionary into server_groups. It gives you enormous flexibility to add/remove customers over time without changing your playbook.

---
- name: read yaml vars into dictionary
  hosts: localhost
  become: no
  gather_facts: no
  vars:
    server_groups: "{{ lookup('file','./' + customer_name + '.yml') | from_yaml }}"
  tasks:
    - debug: var=server_groups

Then use filters to massage the dictionary ...

+++

---
- name: read yaml vars into dictionary
  hosts: localhost
  become: no
  gather_facts: no
  vars:
    server_groups: "{{ lookup('file','./' + customer_name + '.yml') | from_yaml }}"
    server_order: []
  tasks:

    # build parallel lists of name, down_group, up_group
    - set_fact:
        name_list: "{{ server_groups.keys() | list }}"
        halt_list: "{{ server_groups | dict2items | map(attribute='value') | map(attribute='down_group') }}"
        boot_list: "{{ server_groups | dict2items | map(attribute='value') | map(attribute='up_group') }}"

    # merge parallel lists into JSON list with server, down_group, up_group
    - set_fact:
        server_order: "{{ server_order + [ { 'name': item.0, 'halt': item.1, 'boot': item.2 } ] }}"
      with_together:
        - "{{ name_list }}"
        - "{{ halt_list }}"
        - "{{ boot_list }}"

    - debug: var=server_order

+++

... and you get a simple JSON list with server name, down_group, up_group ... I bet others know slicker ways get to this in a single task of filter transformations ...

+++

% ansible-playbook foo.yml -e customer_name=customer
PLAY [read yaml vars into dictionary] **********************************************************************************

TASK [set_fact] ********************************************************************************************************
ok: [localhost]

TASK [set_fact] ********************************************************************************************************
ok: [localhost] => (item=['database_server_01', 3, 1])
ok: [localhost] => (item=['database_server_02', 3, 2])
ok: [localhost] => (item=['file_server_01', 2, 3])
ok: [localhost] => (item=['print_server_01', 1, 2])

TASK [debug] ***********************************************************************************************************
ok: [localhost] => {
    "server_order": [
        {
            "boot": 1,
            "halt": 3,
            "name": "database_server_01"
        },
        {
            "boot": 2,
            "halt": 3,
            "name": "database_server_02"
        },
        {
            "boot": 3,
            "halt": 2,
            "name": "file_server_01"
        },
        {
            "boot": 2,
            "halt": 1,
            "name": "print_server_01"
        }
    ]
}

PLAY RECAP *************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

+++

Hopefully from here you can craft the rest.

--

Walter Rowe, Chief

Infrastructure Services

Office of Information Systems Management

National Institute of Standards and Technology

United States Department of Commerce

[Walter Rowe]

adding to the playbook I already showed ... you now can sort this list by the 'down_group' (halt) or 'up_group' (boot) attribute to get your ordered lists.

    - debug: msg="{{ server_order | sort(attribute='halt') }}"
    - debug: msg="{{ server_order | sort(attribute='boot') }}"

I think that gets you to where you want.

--

Walter Rowe, Chief

Infrastructure Services

Office of Information Systems Management

National Institute of Standards and Technology

United States Department of Commerce

[Walter Rowe]

Just to re-enforce what I already provided ...

+++

---
- name: read yaml vars into dictionary
  hosts: localhost
  become: no
  gather_facts: no
  vars:
    server_groups: "{{ lookup('file','./' + customer_name + '.yml') | from_yaml }}" 
    server_order: []
  tasks:

    - set_fact:
        name_list: "{{ server_groups.keys() | list }}"
        halt_list: "{{ server_groups | dict2items | map(attribute='value') | map(attribute='down_group') }}"
        boot_list: "{{ server_groups | dict2items | map(attribute='value') | map(attribute='up_group') }}"

    - set_fact:
        server_order: "{{ server_order + [ { 'name': item.0, 'halt': item.1, 'boot': item.2 } ] }}"
      with_together:
        - "{{ name_list }}"
        - "{{ halt_list }}"
        - "{{ boot_list }}"

    - name: Shutting down servers
      debug: msg="Shutting down server {{ item.name }}."
      loop: "{{ server_order | sort(attribute='halt') }}"

    - name: Starting up servers
      debug: msg="Starting up server {{ item.name }}."
      loop: "{{ server_order | sort(attribute='boot') }}"

+++

+++
% ansible-playbook foo.yml -e customer_name=customer

PLAY [read yaml vars into dictionary] **********************************************************************************

TASK [set_fact] ********************************************************************************************************
ok: [localhost]

TASK [set_fact] ********************************************************************************************************
ok: [localhost] => (item=['database_server_01', 3, 1])
ok: [localhost] => (item=['database_server_02', 3, 2])
ok: [localhost] => (item=['file_server_01', 2, 3])
ok: [localhost] => (item=['print_server_01', 1, 2])

TASK [Shutting down servers] *******************************************************************************************
ok: [localhost] => (item={'name': 'print_server_01', 'halt': 1, 'boot': 2}) => {
    "msg": "Shutting down server print_server_01."
}
ok: [localhost] => (item={'name': 'file_server_01', 'halt': 2, 'boot': 3}) => {
    "msg": "Shutting down server file_server_01."
}
ok: [localhost] => (item={'name': 'database_server_01', 'halt': 3, 'boot': 1}) => {
    "msg": "Shutting down server database_server_01."
}
ok: [localhost] => (item={'name': 'database_server_02', 'halt': 3, 'boot': 2}) => {
    "msg": "Shutting down server database_server_02."
}

TASK [Starting up servers] *********************************************************************************************
ok: [localhost] => (item={'name': 'database_server_01', 'halt': 3, 'boot': 1}) => {
    "msg": "Starting up server database_server_01."
}
ok: [localhost] => (item={'name': 'database_server_02', 'halt': 3, 'boot': 2}) => {
    "msg": "Starting up server database_server_02."
}
ok: [localhost] => (item={'name': 'print_server_01', 'halt': 1, 'boot': 2}) => {
    "msg": "Starting up server print_server_01."
}
ok: [localhost] => (item={'name': 'file_server_01', 'halt': 2, 'boot': 3}) => {
    "msg": "Starting up server file_server_01."
}

PLAY RECAP *************************************************************************************************************

localhost                  : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

+++

--

Walter Rowe, Chief

Infrastructure Services

Office of Information Systems Management

National Institute of Standards and Technology

United States Department of Commerce

[Walter Rowe]

... and you can add the exclude attribute and filter out items where exclude is false ...

+++

---
- name: read yaml vars into dictionary
  hosts: localhost
  become: no
  gather_facts: no
  vars:
    server_groups: "{{ lookup('file','./' + customer_name + '.yml') | from_yaml }}"
    server_order: []
  tasks:
    - set_fact:
        name_list: "{{ server_groups.keys() | list }}"
        halt_list: "{{ server_groups | dict2items | map(attribute='value') | map(attribute='down_group') }}"
        boot_list: "{{ server_groups | dict2items | map(attribute='value') | map(attribute='up_group') }}"

        excl_list: "{{ server_groups | dict2items | map(attribute='value') | map(attribute='exclude') }}"

    - set_fact:
        server_order: "{{ server_order + [ { 'name': item.0, 'halt': item.1, 'boot': item.2, 'excl': item.3 } ] }}"

      with_together:
        - "{{ name_list }}"
        - "{{ halt_list }}"
        - "{{ boot_list }}"

        - "{{ excl_list }}"

    - name: Shutting down servers
      debug: msg="Shutting down server {{ item.name }}."

      loop: "{{ server_order | json_query('[?excl==`false`]') | sort(attribute='halt') }}"

    - name: Starting up servers
      debug: msg="Starting up server {{ item.name }}."

      loop: "{{ server_order | json_query('[?excl==`false`]') | sort(attribute='boot') }}"

+++

Thanks for humoring me .. nice challenge.

--

Walter Rowe, Chief

Infrastructure Services

Office of Information Systems Management

National Institute of Standards and Technology

United States Department of Commerce

[Rowe, Walter P. (Fed)]

I managed to generate the compressed list in a single task ...

+++

---
- name: read yaml vars into dictionary
  hosts: localhost
  become: no
  gather_facts: no
  vars:
    server_groups: "{{ lookup('file','./' + customer_name + '.yml') | from_yaml }}"
    server_order: []
  tasks:

    - set_fact:
        server_order: "{{ server_order + this_server }}"
      with_together:
        - "{{ server_groups.keys() | list }}"
        - "{{ server_groups | dict2items | map(attribute='value') }}"
      vars:
        this_server: [ { name: "{{ item.0 }}", halt: "{{ item.1.down_group }}", boot: "{{ item.1.up_group }}", excl: "{{ item.1.exclude }}" } ]

    - name: Shutting down servers
      debug: msg="Shutting down server {{ item.name }}."
      loop: "{{ server_order | json_query('[?excl==`false`]') | sort(attribute='halt') }}"

    - name: Starting up servers
      debug: msg="Starting up server {{ item.name }}."
      loop: "{{ server_order | json_query('[?excl==`false`]') | sort(attribute='boot') }}"

+++

Walter
--
Walter Rowe, Division Chief
Infrastructure Services, OISM
Mobile: 202.355.4123

[Black Patrick - Nashville]

You are a lifesaver. Thank you so much! This gives me a fantastic place to start.

 

From: 'Rowe, Walter P. (Fed)' via Ansible Project <ansible...@googlegroups.com>
Date: Thursday, August 18, 2022 at 10:51 AM
To: ansible...@googlegroups.com <ansible...@googlegroups.com>
Subject: {EXTERNAL} Re: [ansible-project] Dynamic/complex inventory - Specific reboot order

CAUTION! This email originated from outside of our organization. DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe.

--
You received this message because you are subscribed to a topic in the Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-project/YPr7kdsA1es/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ansible-proje...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/237D07ED-81AF-4AD4-A3E7-90D7958BF0B9%40nist.gov.

[Patrick Black]

So right now the yaml files are serving as the source of truth.

1. Create YAML file with information per customer.
2. Run Terraform to create the categories/tags in vCenter.
3. Run Ansible community.vmware.vmware_tag_manager to apply the tags to the appropriate servers.

- name: Create tags

  community.general.terraform:

  project_path: ../../scripts/terraform

  register: tag_output

- name: Terraform output

  ansible.builtin.debug:

  msg: "{{ tag_output }}"

- name: Terraform output

  ansible.builtin.debug:

  msg: "{{ tag_output.stderr_lines }}"

  when: tag_output.failed

- name: Add tags

  community.vmware.vmware_tag_manager:

  # omitted details, but applies tags based on requirements

So that's what that looks like as far as tag management. If you think there's a better way I'm absolutely open to suggestions.

The reboot order of the servers is dictated by the vendor and their unique combination of third-party applications they run in combination. So for instance, one customer may need a shutdown order of file servers, database servers, batch servers, management servers, web servers; other may need a completely different order. 

That's why I chose the YAML as the source of truth as this can change as the vendor makes updates to the application so it does need to be fairly flexible. As far as the customer is concerned, this is considered just a generic "downtime" for patching, application updates, whatever is needed.

So to answer use your phone analogy:

****riiiiiinnggggg****

Customer: Hi, application needs to be updated. Can we schedule a downtime for this date and time?
Admin: Why certainly.

--Date and Time Arrives--

Vendor will perform work needed while users are locked out of the system.

Admin will then shutdown servers (perhaps applying patches at the same time) and then bring the servers back up.

Vendor will then unlock the application to end users and go on their way.

Right now, the reboot order is managed via a CSV spreadsheet and a nightmare-ish PowerShell script. The CSV "source of truth" holds the same information I have above in the YAML file: server name, shutdown group, powerup group, and vcenter host. Effective but very difficult to maintain or add new functionality without a complete rewrite.

So the steps and what order are predetermined by whatever source of truth, CSV/YAML/etc.

I hope that gives some more context and happy to answer any additional questions.