Introducing Accelerated Mode

[James Cammarata]

I've just merged the latest feature for 1.3: Accelerated Mode. 

Accelerated mode is essentially fireball mode, with a few improvements:

• No bootstrapping required.
• Support for running commands via sudo.
• Fewer requirements! 0mq is no longer required, and only python-keyczar is required.

To use accelerated mode, all you have to do is add "accelerate: true" to your play:

    ---

    - hosts: all

      accelerate: true

      tasks:

      - name: test command

        shell: echo "hello world"

And that's pretty much it! The first time Ansible connects to the target hosts it will start the accelerated daemon, after which time it will connect to it directly. We have seen speed improvements of anywhere from 2-6x over SSH with ControlPersist enabled, so the performance over paramiko or SSH without ControlPersist is extremely good.

There are a few more details (and caveats regarding sudoing comands) in the docs: 

http://www.ansibleworks.com/docs/playbooks2.html#accelerated-mode

Enjoy!

--

James Cammarata <jcamm...@ansibleworks.com>
Sr. Software Engineer, AnsibleWorks, Inc.
http://www.ansibleworks.com/

[Serge van Ginderachter]

​This is great!​

On 3 September 2013 06:59, James Cammarata <jcamm...@ansibleworks.com> wrote:

and only python-keyczar is required.

On local command server and remote node that is: when missing on the command server, a warning is issued; 

the warning generated on the remote node by the accelerate module is however not given back to the user, it might be nice to be more verbose about that?

[Michael DeHaan]

Shouldn't be a warning, should be an error.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Michael DeHaan <mic...@ansibleworks.com>
CTO, AnsibleWorks, Inc.
http://www.ansibleworks.com/

[James Cammarata]

Yes, local mode should ignore the accelerate keyword, in my opinion. Can you open a github issue for this? Thanks!

[Serge van Ginderachter]

On 3 September 2013 16:58, James Cammarata <jcamm...@ansibleworks.com> wrote:

Yes, local mode should ignore the accelerate keyword, in my opinion. Can you open a github issue for this? Thanks!

No local mode involved, I just mean that both ansible-playbook​ (running locally) and the remote node running the accelerate module both need python-keyczar, but a warning is only got from ansible-playbook locally, whilst it doesn't report that from the remote moduie

[James Cammarata]

Ahh gotcha. I saw the issue was opened, thanks!

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Jürgen Haas]

Very interesting. Raises three questions for me:

1) What is the scope of such a temporary daemon? Is it limited to a play within a playbook? Or is it limited to a playbook across all included plays? Or is it limited to a session of ansible-playbook? Or is it limited to the workstation and lives there for those 30minutes that are mentioned in the documentation even across several instances of ansible-playbook?

2) What about any security implications of that? Anything we should be careful with? E.g. the temporary daemon does have to store the keys somewhere and I wonder if this is potentially opening some potential holes.

I assume the authentication for the initial daemon on each host is still going through the original ssh authentication process using either username/password or certificates, right?

3) Is there a fallback included so that if the requirements for the accelerated mode are not met that they always fall back to ssh connection type?

[James Cammarata]

1. It is the same as the previous fireball, it will persist for 30 minutes even across other plays, so if two plays have access to the shared secret key they can execute tasks via the accelerated mode without having to relaunch the daemon.

2. Again, this is the same in regards to the previous fireball mode. The key is stored in a directory located in the $HOME of the user launching the daemon. The original auth and transfer of the keys all occurs via SSH.

3. That was proposed as an RFE, and we will look to add that in 1.4. For now, it's pretty easy to get the systems into alignment since the only requirement is python-keyczar (which has 2 dependencies of its own: python-crypto and python-asn1).

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[James Cammarata]

Just pushed up a fix for this, if you'd like to give it a try.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Jürgen Haas]

Thanks a lot for the detailed answers, this is extremely helpful.

I've just tried my common role on one remote host with gathering_fact on 3 hosts and 12 tasks (all without changes) and compare the "connection: ssh" versus the "acceleration: true" with a time saving of 5 seconds only, down from 33 to 28s. This is less than expected. But still 15%. Is there anything I could do to debug this any further?

[James Cammarata]

The time savings generally grow as the number of tasks and the number of hosts grow. Your example is a single task on 3 hosts, so the fact that you're seeing a 15% increase is actually pretty good :) My usual test playbook (run on a couple of different public clouds) was a single task with 10 with_items, followed by 40 individual tasks (all were essentially command: echo "something"). All together, that means 50 tasks were run on each target, and I was typically testing with 5-10 target nodes with an equal number of forks.

Another user ran a more simple playbook (create 5 directories, sleep 10 seconds, delete the 5 directories) and was seeing a 3x increase in speed (18 seconds vs 46 when using SSH). That in particular was a very nice result, as it was 10 operations in 8 seconds (after factoring out the sleep time).

These are all compared to SSH using ControlPersist, which is itself 2-3x faster than paramiko or SSH without CP, thus the note that it can be up to 6x faster than straight SSH management connections.

[Jürgen Haas]

OK, I'll do another benchmark when I've finished rewriting my playbook and roles.

However, one more question:

Can you see a way to pass a parameter to a playbook whether acceleration should be used or not? In my playbook I'm having this:

- name: "Gather Facts"
  hosts: all
  connection: ssh
  accelerate: "{{accelerate}}"
  gather_facts: true
  sudo: yes

and I'm calling this with "ansible-playbook -i HOSTS PLAYBOOK --extra-vars "accelerate=false" --ask-sudo-pass"

It still tries to use acceleration mode. Any idea?

[James Cammarata]

That I did not try. I just pushed a commit up to fix this (the code was simply checking for the presence of the accelerate: keyword in the YAML and not checking the actual value assigned to the key).

If you re-pull via git, this should be working as you expect.

[Jürgen Haas]

Great. Thanks you!!!

[Jürgen Haas]

Found another problem with the accelerated mode: timeouts.

The task where this happens:

- name: "MySQL | Install required packages"
  apt: pkg={{ item }} state=installed
  with_items:
    - mysql-server
    - mysql-client

When I run this in accelerated mode and my host is really (!!!) slow, meaning that this takes significantly longer than 10 seconds, then the playbook aborts with the following error message:

fatal: [vm1] => timed out while waiting to receive data

Using -vvv to get more detailed debug data, I get this output:

TASK: [MySQL | Install required packages] *************************************
EXEC COMMAND /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=kkyyizrfwfsovplsimpsfxvpmqojahdj] password: " -u root /bin/sh -c '"'"'mkdir -p $HOME/.ansible/tmp/ansible-1378283391.67-257013052215365 && chmod a+rx $HOME/.ansible/tmp/ansible-1378283391.67-257013052215365 && echo $HOME/.ansible/tmp/ansible-1378283391.67-257013052215365'"'"''
<vm1> REMOTE_MODULE apt pkg=mysql-server,mysql-client state=installed
<vm1> PUT /tmp/tmpHWCen0 TO /root/.ansible/tmp/ansible-1378283391.67-257013052215365/apt
PUT file is 48995 bytes
EXEC COMMAND /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=hicqkdxwgfyrntmdqhvjrgzhiopvfgwe] password: " -u root /bin/sh -c '"'"'/usr/bin/python /root/.ansible/tmp/ansible-1378283391.67-257013052215365/apt; rm -rf /root/.ansible/tmp/ansible-1378283391.67-257013052215365/ >/dev/null 2>&1'"'"''
fatal: [vm1] => timed out while waiting to receive data

When I run the same playbook without acceleration then it takes really long but it doesn't fail.

[James Cammarata]

This is a known issue I'm working on correcting. Right now, I think we're just going to up the socket receive timeout to 5 minutes rather than the 30 seconds it is now. I'll be working on adding a keepalive mechanism that will still timeout connection failures quickly while allowing for more long-running connections, though we may not be able to get that in before Friday's release so it would be in 1.4.

[Michael DeHaan]

A timeout of 5 minutes for an unreachable host is definitely not going to be acceptable as it will pin up other tasks on other hosts, let's see what we can get in.

[Michael DeHaan]

Ok, I misread this, this is about long running tasks... this still may be something of a problem, but seems ok.   I'm not sure 5 minutes is the magic number, but use of async for long running tasks is usually a good idea anyway.

We should make extra sure we're not going to any other kinds of scenarios though.

[Jesse Keating]

On Sep 4, 2013, at 5:25 AM, Michael DeHaan <mic...@ansibleworks.com> wrote:
>
> Ok, I misread this, this is about long running tasks... this still may be something of a problem, but seems ok. I'm not sure 5 minutes is the magic number, but use of async for long running tasks is usually a good idea anyway.
>
> We should make extra sure we're not going to any other kinds of scenarios though.

From my reading of the code (and running into this pre-launch), there is a single timeout value for fireball2, and that's used for BOTH waiting for a host to respond AND for waiting for a task to finish. I suggested finding a way to separate these timeouts, much like ssh itself has, so that we can have a 5 second timeout on a dark host, but still allow for a 30 minute+ task to run.

-jlk

[Michel Sánchez Montells]

When I run ansible-playbook in accelerate mode and for some reason ansible task fail the next run will not connect with following error

PLAY [all] ******************************************************************** 

GATHERING FACTS *************************************************************** 

fatal: [xxx.xxx.xxx.xxx] => Failed to connect to xxx.xxx.xxx.xxx:5099

TASK: [role | task] *********************************************** 

FATAL: no hosts matched or all hosts have already failed -- aborting

PLAY RECAP ******************************************************************** 

           to retry, use: --limit @//task.retry

xxx.xxx.xxx.xxx                : ok=0    changed=0    unreachable=1    failed=0   

PLAY [all] ******************************************************************** 

GATHERING FACTS *************************************************************** 

fatal: [xxx.xxx.xxx.xxx] => Failed to connect to xxx.xxx.xxx.xxx:5099

TASK: [role | task] ********************************************* 

FATAL: no hosts matched or all hosts have already failed -- aborting

and so on...

Then in target host there is a ansible procces running. I run 

kill -9 $(ps -aux | grep accelerate | awk '{print $2}') for kill this proccess and next run success

[Michael DeHaan]

Hi,

FYI - You're replying to a thread that is almost a year old.

It sounds like you have a new issue, but I'm not sure what version of ansible you are running.

Please let us know this and what OSes you have on both sides of the equation for starters, and we can go from there.

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.

To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/198c02a4-c31c-4975-b117-05f18f4c3d5c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michel Sánchez Montells]

origin station

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=14.04

DISTRIB_CODENAME=trusty

DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS"

3.13.0-32-generic

target station

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=12.04

DISTRIB_CODENAME=precise

DISTRIB_DESCRIPTION="Ubuntu 12.04.4 LTS"

3.13.0-32-generic

ansible-playbook 1.6.10

--
You received this message because you are subscribed to a topic in the Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-project/40lqT6xp0y0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ansible-proje...@googlegroups.com.

To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgyc0Pt1hNEuAOtJrUk5mCo%2BegM6%3DyHg6F%3Drc0WJKhVvDQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Michel Sánchez Montells