How to detect why Ansible playbook hangs during execution

[9dev]

Some of tasks I wrote start and never end. Ansible does not provide any errors or logs that would explain this, even with -vvvv option. Playbook just hangs and passing hours doesn't change anything.

When I try to run my tasks manually (by entering commands via SSH) everything is fine.

Example task that hangs:

- name: apt upgrade
  shell: apt-get upgrade

Is there any way to see stdout and stderr ? I tried:

- name: apt upgrade
  shell: apt-get upgrade
  register: hello
- debug: msg="{{ hello.stdout }}"
- debug: msg="{{ hello.stderr }}"
 

but nothing changed.

I do have required permissions and I pass correct sudo password - other tasks that require sudo execute correctly.

[Michael DeHaan]

Apt by default needs eleventy zillion flags to not go interactive.

There's a parameter you should use on the 'apt' module that does upgrades that will take care of this for you.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Michael DeHaan <mic...@ansibleworks.com>
CTO, AnsibleWorks, Inc.
http://www.ansibleworks.com/

[9dev...@gmail.com]

Thanks for answer.

However, apt is not the only task that hangs without ANY information why.

Another example:

shell: apt-get install --no-install-recommends virtualbox-guest-utils && sudo apt-get install virtualbox-guest-dkms

Should I always use your apt module? Why can't I just execute simple shell command ?

Note that I tried to execute all these tasks manually and there was no errors and no need for any user interaction. Why Ansible has problem with them?

[Brian Coca]

you can execute shell commands, just not ones that expect interactivity as they hold the tty (this happens with any background ssh execution).

for apt commands, yes use the apt module, it takes care of 99% f these issues.

[Michael DeHaan]

You should absolutely use the apt module.

You can install multiple packages in a single step by using "with_items" and there's a flag for controlling whether recommended packages are installed (see module docs).

I suspect as for the commands not being interactive the second time it was because you've already executed them and already bypassed the shell prompts.

On Sun, Dec 29, 2013 at 11:52 AM, Brian Coca <bria...@gmail.com> wrote:

you can execute shell commands, just not ones that expect interactivity as they hold the tty (this happens with any background ssh execution).

for apt commands, yes use the apt module, it takes care of 99% f these issues.

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[9dev...@gmail.com]

Thank you for your answers. I will use your apt module.

But just for the record - I always rollback to fresh virtual machine, so the reason for hanging is not double task execution for sure.

[Michael Palumbo]

What is the parameter of the apt module to use to avoid that?

I run the following and it hangs...

- name: update APT package cache
sudo: yes
apt: update_cache=yes

- name: upgrade all safe packages
sudo: yes
apt: upgrade=safe


Thanks

[Michael DeHaan]

apt should never need a parameter to not hang, however some *specific* packages can sometimes be poorly coded.

Not saying that's the case here.

I'd first check your ansible version and then file a bug with steps to reproduce.

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/d0595406-cff0-441d-98b3-a79b8f19b5de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Palumbo]

So there is no solution to avoid being stuck on the following? It happens to me often when I am upgrading the system!

- name: upgrade all safe packages
  sudo: yes
  apt: upgrade=safe

Is there at least a way to see the progress? Looking at some logs on the machine?

The state of my machine is messy when this task is stuck because I have to stop ansible-playbook with CTRL+C and all the handlers are not notified... :-/

[Michael DeHaan]

When you kill a playbook you can use --force-handlers to make sure handlers on a repeated runs.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/e8099617-0b3f-4ba5-8698-235976cf7194%40googlegroups.com.

[Michael Palumbo]

Does that also work when one task fails and the playbook stops?

Does it work with the tasks of the roles as well?

--
You received this message because you are subscribed to a topic in the Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-project/mm99yAPVrfc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ansible-proje...@googlegroups.com.

To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgwFr%3DTTP8Kw6CR0Dph_SyTZbLVdX6GserpfodmZ%3Dnf%2BOw%40mail.gmail.com.

[Michael Palumbo]

I just tried to run my playbook with --force-handlers and did CTRL+C: it stopped it right away and the handlers were not notified.

Does it work for you?

I use ansible 1.6.7

Thanks.

[Michael DeHaan]

This is a misunderstanding of force-handler behavior.

You would run-it again with --force-handlers, and any handlers would run, whether notified or not.

Don't control-C the second run :)

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CALa-rByeCF-8-eO%2BKeLe%3DS75ZJ%3Dx%2B73nL1xT%2BQ7kri%3D%2BzWW7%2Bw%40mail.gmail.com.

[Tony Marin]

I use a screen trick but mostly useful when using commands modules. Anyway, if it hangs doing apt upgrade, the behavior should be the same if your run it as a command.

- debug: msg="Please, connect to {{ inventory_hostname }}, screen -r, and respond the possible questions."
   when: ansible_hostname == drbd_master.name
- name: Enable HA
   command: screen -dm -S halizard bash -c 'ha-cfg status'
   when: ansible_hostname == drbd_master.name
   async: 60
   poll: 60

It's also useful when you cannot avoid interactive mode on some actions, but you miss the unattended possibilities.

Br,
Tony

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/e8099617-0b3f-4ba5-8698-235976cf7194%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Tony Marin

DevOps Manager

Swiss Mobility Solutions
a Gemalto Company

+34 637402568

+34 966370055

antonymcs

Av. Locutor Vicente Hipólito 39
03540 Alicante Spain

 

[Karl E. Jorgensen]

Hi

On Tue, Jul 22, 2014 at 02:41:53PM -0700, Michael Palumbo wrote:
> So there is no solution to avoid being stuck on the following? It happens to me
> often when I am upgrading the system!
> - name: upgrade all safe packages
> sudo: yes
> apt: upgrade=safe
>
> Is there at least a way to see the progress? Looking at some logs on the
> machine?

In the case of apt (which uses dpkg), it may be useful to:

tail -f /var/log/dpkg.log

on the server being upgraded.

If you interrupt the task, I'd expect ansible to give info about what
was output by the upgrade so far - that should contain clues as to
what it was doing at the time of the upgrade.

Also, for debugging purposes, it may be useful to find out which tty
is being used by ansible on the target machine (ps -ef|grep ansible or
similar), and see what is running on that tty:

ps -ft${basename-of-the-tty}

Hope this helps

--
Karl E. Jorgensen

[Rubén Ortiz]

Hi

sorry for reply in old post but just want to comment something about...

What about asynchronous polling? 

https://docs.ansible.com/playbooks_async.html

What if we run a playbook like:

- hosts: vm-ubuntu

  sudo: true

  user: root

  serial: 4

  vars_files:

    - external_vars.yml

  tasks:

  - name: update cache

    apt: update_cache=yes cache_valid_time=86400

  - name: update packages full into each server

    apt: name={{ item }}  state=present

    with_items: ubuntu_packages

    ignore_errors: True

    async: 45

    pool: 5

  tags:

    - install

Will be useful to avoid SSH timeouts in a long playbooks tasks?

What do you think?

Thanks