Rescue all hosts if one host fails

[Aaron Falk]

There is no possibility to run the rescue part for all hosts if one host fails. I'm working on my deployment for a website. I want to deploy the site to multiple hosts. If something fails for one host than all hosts should not be deployed. Therefore, a rollback on all hosts should happen. The configuration "any_errors_fatal" is not working for it.

I am using ansible 2.4.1.0

Here the steps to reproduce:

- block:
    - name: Update database schema
      shell: cd '{{ ansible_web_deployment_webroot }}/releases/{{ ansible_web_deployment_release }}' && php7.1 console.php orm:schema-tool:update --force
      when: inventory_hostname in groups['schema']
    - name: Deployment
      debug:
        msg: 'Deploying'
  rescue:
    - name: Rollback
      debug:
        msg: 'Rollback'
  any_errors_fatal: true

Expected results:

PLAY [test] ********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [x1.example.com]
ok: [x2.example.com]
ok: [x3.example.com]
ok: [x4.example.com]

TASK [ansible_web_deployment : Update database schema] *************************
skipping: [x2.example.com]
skipping: [x3.example.com]
skipping: [x4.example.com]
fatal: [x1.example.com]: FAILED! => {"changed": true, "cmd": "cd '/var/www/x.example.com/releases/20171113081452' && php7.1 console.php orm:schema-tool:update --force", "delta": "0:00:00.007578", "end": "2017-11-13 08:15:05.709148", "failed": true, "msg": "non-zero return code", "rc": 2, "start": "2017-11-13 08:15:05.701570", "stderr": "/bin/sh: 1: cd: can't cd to /var/www/x.example.com/releases/20171113081452", "stderr_lines": ["/bin/sh: 1: cd: can't cd to /var/www/x.example.com/releases/20171113081452"], "stdout": "", "stdout_lines": []}

TASK [ansible_web_deployment : Rollback] *****************************
ok: [x1.example.com] => {
    "msg": "Rollback"
}
ok: [x2.example.com] => {
    "msg": "Rollback"
}
ok: [x2.example.com] => {
    "msg": "Rollback"
}
ok: [x4.example.com] => {
    "msg": "Rollback"
}

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=2    changed=0    unreachable=0    failed=0
x1.example.com : ok=2    changed=1    unreachable=0    failed=1
x2.example.com : ok=2    changed=1    unreachable=0    failed=1
x3.example.com : ok=2    changed=1    unreachable=0    failed=1
x4.example.com : ok=2    changed=1    unreachable=0    failed=1

Actual results:

PLAY [test] ********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [x1.example.com]
ok: [x2.example.com]
ok: [x3.example.com]
ok: [x4.example.com]

TASK [ansible_web_deployment : Update database schema] *************************
skipping: [x2.example.com]
skipping: [x3.example.com]
skipping: [x4.example.com]
fatal: [x1.example.com]: FAILED! => {"changed": true, "cmd": "cd '/var/www/x.example.com/releases/20171113081452' && php7.1 console.php orm:schema-tool:update --force", "delta": "0:00:00.007578", "end": "2017-11-13 08:15:05.709148", "failed": true, "msg": "non-zero return code", "rc": 2, "start": "2017-11-13 08:15:05.701570", "stderr": "/bin/sh: 1: cd: can't cd to /var/www/x.example.com/releases/20171113081452", "stderr_lines": ["/bin/sh: 1: cd: can't cd to /var/www/x.example.com/releases/20171113081452"], "stdout": "", "stdout_lines": []}

TASK [ansible_web_deployment : Restart services] *****************************
ok: [x2.example.com] => {
    "msg": "Deploying"
}
ok: [x2.example.com] => {
    "msg": "Deploying"
}
ok: [x4.example.com] => {
    "msg": "Deploying"
}

TASK [ansible_web_deployment : Rollback] *****************************
ok: [x1.example.com] => {
    "msg": "Rollback"
})

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=2    changed=0    unreachable=0    failed=0
x1.example.com : ok=2    changed=1    unreachable=0    failed=1
x2.example.com : ok=2    changed=1    unreachable=0    failed=0
x3.example.com : ok=2    changed=1    unreachable=0    failed=0
x4.example.com : ok=2    changed=1    unreachable=0    failed=0

How can I achieve that all hosts go in rescue mode when a single host throws an exception?

If you have a look at the Github issue 14024, you'll see that this should work.

Unfortunately I was not able to get it to work.

Thank you for your input.

[Pshem Kowalczyk]

Hi

You can use the following method (sorry I don't have the playbook ready).

1. Run the upgrade on each host, ignore errors, but register the results and store them in a file on the controller machine (you can also manually - fail: once the error is registered for a given host)

2. Once all upgrades are done on all hosts - check the content of the 'results' files on the controller. If any of them show issues - run another role to revert the changes.

kind regards

Pshem

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/661b16c0-7a35-45ed-9b8d-84eaaf836140%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[amit ahuja]

Hi Aaron,

did you managed to have ansible rollback deployment on all hosts when anyone failed in any task ? 

[Brian Coca]

I can think of a couple of ways.

You can force a fail at the end of block if any hosts failed:

- block:
- name: Update database schema
shell: cd '{{ ansible_web_deployment_webroot }}/releases/{{
ansible_web_deployment_release }}' && php7.1 console.php
orm:schema-tool:update --f
orce
when: inventory_hostname in groups['schema']
- name: Deployment
debug:
msg: 'Deploying'

- fail:
when: ansible_play_hosts|length != ansible_play_hosts_all|length

rescue:
- name: Rollback
debug:
msg: 'Rollback'
any_errors_fatal: true

Or delegate rescue to all hosts:

- block:
- name: Update database schema
shell: cd '{{ ansible_web_deployment_webroot }}/releases/{{
ansible_web_deployment_release }}' && php7.1 console.php
orm:schema-tool:update --f
orce
when: inventory_hostname in groups['schema']
- name: Deployment
debug:
msg: 'Deploying'
rescue:
- name: Rollback
debug:
msg: 'Rollback'

delegate_to: '{{item}}'
loop: '{{ ansible_play_hosts_all }}'
any_errors_fatal: true

--
----------
Brian Coca