looping over set of tasks (roles?)

[Nick Evgeniev]

Hi,

I need to perform (in a loop) some tasks on host 'A' till something happens on host 'B'. What is the 'best practice' of doing such things with ansible?

I saw loops over single task but this is not what I need (presumably), unless I write a shell script which perform all the tasks by connecting to different hosts and grabs the results. As the whole point of ansible (to me) is to pull all the ssh commands out of scripts (and get rid of these scripts too)

[Dmitry Makovey]

does that mean you're running cross-host function that takes params from A and B and returns value on B? And you expect certain value on B before you move forward? I'm not an expert but it doesn't seem to be Ansible domain. Maybe Fabric would be a better fit?

[Nick Evgeniev]

yep. exactly the case. at the moment this logic is implemented using .sh script.. so looks like the best option is to mix .sh with ansible (for pure deployment tasks). Will take a look into Fabric .. though I'd like to avoid introducing new tool for every simple task :)

[Michael DeHaan]

I would benefit by a more specific example so I can understand what you mean.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/63014730-309e-44a9-b2c6-400b9a7d5388%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Petros Moisiadis]

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/cb1bea17-3025-426e-b576-408a0dafad4c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Ansible playbooks have not a mechanism for repeating a part of a playbook, but this feature could be useful for your user case. For example, you could write a series of tasks and add at the end of them a 'goback' task (by creating a 'goback' action plugin) that lets you go back at the first task of that group of tasks you want to repeat. That 'goback' task would of course have the appropriate conditional, which would normally be set by the repeating tasks.

- name: Task one
  modulename: ...

- name: Task two
  modulename: ...

- name: Task three
  modulename: ...

- name: Go back to "Task one" (so, repeat tasks one to three)
  goback: task="Task one"
  when: somecondition

[Maciej Delmanowski]

You know what happens, when you use goto, right? https://xkcd.com/292/

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/53ABC4BB.8020901%40yahoo.gr.

[Michael DeHaan]

There is no goback task in Ansible.

There will not be one.

(I'd still benefit from more detail from the original poster, btw)

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/53ABC4BB.8020901%40yahoo.gr.

[Nick Evgeniev]

Adding some details:

1. node A is running some java app

2. node B is running another app which in turn can generate some 'load' for node A

in terms of states I need not only have app 'started' on node A but also have it 'warmed up'. as warming up a java process is a complex task the only way to put app in 'warmed up' state is to organize feedback loop between A and B and stop warmup runs (on B) as soon as number of compiled/recompiled methods by hotspot drops to certain level. At this point you'll get some 'confidence' that app is warmed up and you can proceed further

ansible has a way of repeating single task but for some (may be good) reason there is no way of repeating 'composite task' (which is what roles are to my understanding) 

[Jerome Wagner]

Hello,

I haven't tried it but maybe you can loop over an 'include' task ? have you tried it already ?

- include: warmup.yml
  register: result
  until: result.stdout.find("all systems go") != -1
  retries: 5
  delay: 10

[Michael DeHaan]

"Adding some details:

1. node A is running some java app

2. node B is running another app which in turn can generate some 'load' for node A

in terms of states I need not only have app 'started' on node A but also have it 'warmed up'. as warming up a java process is a complex task the only way to put app in 'warmed up' state is to organize feedback loop between A and B and stop warmup runs (on B) as soon as number of compiled/recompiled methods by hotspot drops to certain level. At this point you'll get some 'confidence' that app is warmed up and you can proceed further

ansible has a way of repeating single task but for some (may be good) reason there is no way of repeating 'composite task' (which is what roles are to my understanding) "

- hosts: java

  tasks:

     - # steps to wait for it to be started

- hosts: load_generators

  tasks:

     - # steps to generate load against java boxes, using delegate_to: "{{ item }}" and with_items: groups.java ?

   

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/b19d8e1d-db1a-4d82-8e82-ab17209767cb%40googlegroups.com.

[Petros Moisiadis]

On 06/26/14 19:58, Michael DeHaan wrote:

"Adding some details:

1. node A is running some java app

2. node B is running another app which in turn can generate some 'load' for node A

in terms of states I need not only have app 'started' on node A but also have it 'warmed up'. as warming up a java process is a complex task the only way to put app in 'warmed up' state is to organize feedback loop between A and B and stop warmup runs (on B) as soon as number of compiled/recompiled methods by hotspot drops to certain level. At this point you'll get some 'confidence' that app is warmed up and you can proceed further

ansible has a way of repeating single task but for some (may be good) reason there is no way of repeating 'composite task' (which is what roles are to my understanding) "

- hosts: java

  tasks:

     - # steps to wait for it to be started

- hosts: load_generators

  tasks:

     - # steps to generate load against java boxes, using delegate_to: "{{ item }}" and with_items: groups.java ?

This does not seem to be a solution to his problem. What he actually needs is a way to repeat a number of different, load generating tasks on host B until his java app on host A has been warmed up for good. It is that repetition that Ansible cannot currently handle.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgyiE4zPtS%3DBBr02QJL9HEP%3DFZLdnncxWDRCSsQ8HbXGPw%40mail.gmail.com.

[Michael DeHaan]

"It is that repetition that Ansible cannot currently handle."

I'm not sure that's true.

Ansible has loops and things like with_sequence, as well as do_until loops.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/53AC5662.3090801%40yahoo.gr.

[Petros Moisiadis]

On 06/26/14 20:24, Michael DeHaan wrote:

"It is that repetition that Ansible cannot currently handle."

I'm not sure that's true.

Ansible has loops and things like with_sequence, as well as do_until loops.

Ansible can do this kind of loops, but only for a single task. It does not offer a way to repeat a number of different tasks as a group.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgycGhiLGTS0wn6-FJywp1KKzxo3MevNd%3DyjNb17xGbJWQ%40mail.gmail.com.

[Michael DeHaan]

"Ansible can do this kind of loops, but only for a single task. It does not offer a way to repeat a number of different tasks as a group."

Agreed, we really aren't trying to create  a programming language.

Do you have to be as careful with generating exactly a certain amount of load?

I would ordinarily think a script (called by ansible of course) could be created that generates a certain amount of load and that would be predictable enough to warm up most things.   

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/53AC5993.8040201%40yahoo.gr.

[Nick Evgeniev]

It's not about being that careful but rather about getting to predictable result as soon as possible.

As I mentioned warming up of java program is a complex thing and depends heavily on jvm version/ switches as well as load 'patterns'. In my case everything could change.. So to answer your question yes it's possible to get guaranteed warm up ... but the cost of such guarantee would be high (for my use case) - for sure app will be warmed up after half an hour of running load generators..

i'm fine with hybrid solution (i.e to run ansible from within a shell script then run some 'imperative' logic .. just trying to pull as much 'ssh' out of scripts as possible)

btw not of immediate demand but what's the best way to get 'cached' ansible facts from within shell script? i think it might be useful in certain circumstances 

[Petros Moisiadis]

On 06/26/14 21:06, Michael DeHaan wrote:

"Ansible can do this kind of loops, but only for a single task. It does not offer a way to repeat a number of different tasks as a group."

Agreed, we really aren't trying to create  a programming language.

When a need to support a common task execution scenario arises and Ansible cannot handle it, I consider it a limitation. We are not trying to create a programming language, but we are all using Ansible to script task execution scenarios. I expect to be able to tell my machines "repeat that group of tasks until something becomes ready or reaches some limit".  This is speaking to my machines and telling them what to do, it's not programming.  I think Ansible can be improved on that aspect.

Do you have to be as careful with generating exactly a certain amount of load?

I would ordinarily think a script (called by ansible of course) could be created that generates a certain amount of load and that would be predictable enough to warm up most things.   

Instead of writing a shell script, another option would be to write another ansible playbook with his load generation tasks and call it repeatedly in a local action in the "main" playbook (with an 'until' directive). That way, he will not have to abandon Ansible's goodies (modules). Ansible inside Ansible!

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgwUcZAAQwvMUbRSRywBeZda8hubDwh-ejHHkUqd_eUVfA%40mail.gmail.com.

[Michael DeHaan]

On Thu, Jun 26, 2014 at 3:10 PM, 'Petros Moisiadis' via Ansible Project <ansible...@googlegroups.com> wrote:

On 06/26/14 21:06, Michael DeHaan wrote:

"Ansible can do this kind of loops, but only for a single task. It does not offer a way to repeat a number of different tasks as a group."

Agreed, we really aren't trying to create  a programming language.

When a need to support a common task execution scenario arises and Ansible cannot handle it, I consider it a limitation. We are not trying to create a programming language, but we are all using Ansible to script task execution scenarios. I expect to be able to tell my machines "repeat that group of tasks until something becomes ready or reaches some limit".  This is speaking to my machines and telling them what to do, it's not programming.  I think Ansible can be improved on that aspect.

I think you're picking at words a bit - it's programming computers to do things.  There are going to be cases where something like the script module are going to apply.

Having ansible evolve entirely into a general purpose programming language would make a very very bad programming language.

I'm ok if that's something we can't do, because it's going to be impossible to do /everything/.

Ansible is already more flexible than any other config system out there, because it's more stepwise, versus compiled-model based (even the coded ones use DSLs still compile down).

[Dmitry Makovey]

On Thursday, June 26, 2014 12:40:45 PM UTC-6, Nick Evgeniev wrote:

It's not about being that careful but rather about getting to predictable result as soon as possible.

As I mentioned warming up of java program is a complex thing and depends heavily on jvm version/ switches as well as load 'patterns'. In my case everything could change.. So to answer your question yes it's possible to get guaranteed warm up ... but the cost of such guarantee would be high (for my use case) - for sure app will be warmed up after half an hour of running load generators..

i'm fine with hybrid solution (i.e to run ansible from within a shell script then run some 'imperative' logic .. just trying to pull as much 'ssh' out of scripts as possible)

Here are some ideas that might work for you:

* use proper tool for load generation, something like JMeter and call out to it for the "warm-up"

* use Ansible as a library from within Python script and glue together tasks in whichever fashion you'd like.

* add more programmatic framework to the mix (I have mentioned Fabric already, there are others) 

bottom line - it sounds like your problem calls for more liberal use of technologies than just limiting yourself to one. Ansible is a good configuration management and orchestration system. Python is a programming language and Fabric is a framework written in it - Ansible has API you can call out to from within Fabric and get things done on a different level.

[Petros Moisiadis]

Ansible is powerful as a configuration system for bringing systems in a desired state and flexible for deploying software, but it could be more flexible as a task execution engine for things such as feedback-based repeated execution and bootstrapping of custom clustering environments (not using ready-made amazon's or google's or rackspace's cloud engine products/services) based on complex nested data. This flexibility could come from increasing flexibility in creating task loops.

It seems that you are not interested in going that direction with the excuse that more flexible loops are for programming languages. However, the fact that programming languages are very flexible in creating loops does not mean that other languages/systems (including Ansible) will automatically turn into programming languages if they become more flexible in that. Loops are natural. They are a very basic concept. They are everywhere in life, not only in programming, and people start learning about and using them from the very beginning. We have all played "Snakes and Ladders" as kids, which is exactly a loop of repeating steps until you reach a goal. So, having a 'goback' feature, for example, would be like being swallowed by a snake and going to repeat your previous steps :D. Nothing to worry about programming languages and other monsters...

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgxzeKsOvRPMMvbuaV72Cevm0-Nec0VFQedWLjBBU_UXNQ%40mail.gmail.com.

[Michael DeHaan]

"It seems that you are not interested in going that direction with the excuse that more flexible loops are for programming languages."

These are project directions, not excuses, and we're mincing at words here.

Just so this isn't misinterpreted, Ansible already provides the ability to write custom iterators, we have things like "with_nested", "with_sequence", etc.   It already can loop over single tasks quite fine, 

So what you are asking about is the ability to loop over more than one set of tasks - I would not implement this as a "goback" even if we did this, it would be a higher level language construct, more akin to a block with control structures.  Gotos are a gross construct nearly everywhere we are found. 

In most cases it's not needed - Modules already take care of most of the provisioning needs, for instance, "exact_count" on provisioning modules allow spinning up an arbitrary number of images, and then you can apply tasks to images using the host loop.   Thus we can ALREADY do all of that cluster spinup, and do that every day.

While looping over multiple tasks in a common loop could be something Ansible does in the future, we have a lot of more important things to chase first in the PR queue before we would consider such complexity when so few things need it.   So far the only example we have that we "can't" do is "warm up group of servers A from load of B until servers A reach threshold X", which strikes me as very arbitrary.

We've suggested a simple compromise of generating a set load, and that was ruled insufficient, or even calling out to a simple script - I see no point to going on about this infinitely given there are solutions.  It's a very niche scenario and I'm ok with not having a clear way to do *EXACTLY* that, when there are other ways to achieve the same ends, just not with the technical criteria stated about exactly how it should be implemented.

As long as there's one possible way, that's enough for me - we don't have to implement exactly this proposed way, especially when we think it would reduce the readability of the Ansible playbook language and encourage more complicated playbooks when Ansible's goal is encouraging simplicity and elegant playbook design.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/53AD22F8.5080101%40yahoo.gr.

[Petros Moisiadis]

On 06/27/14 15:42, Michael DeHaan wrote:

"It seems that you are not interested in going that direction with the excuse that more flexible loops are for programming languages."

These are project directions, not excuses, and we're mincing at words here.

Just so this isn't misinterpreted, Ansible already provides the ability to write custom iterators, we have things like "with_nested", "with_sequence", etc.   It already can loop over single tasks quite fine, 

This is true about custom iterators, though not all people can write custom iterators and I have personally witnessed some unjustified negativism for accepting in core small improvements for available iterators for everyone's benefit.

So what you are asking about is the ability to loop over more than one set of tasks - I would not implement this as a "goback" even if we did this, it would be a higher level language construct, more akin to a block with control structures.  Gotos are a gross construct nearly everywhere we are found. 

To be more specific, I was thinking of a "goback" feature as a very restricted 'goto'. It would allow to move only backwards and only in the same play and same file. That way all problems of 'goto' are gone.

That said, now I am thinking of another possible implementation:

- name: Repeat a group of tasks 5 times to warm up my servers
  repeat: tags="warmup" times=5

- name: Repeat a group of tasks to warm up my servers until a condition is true
  repeat: tags="warmup" times=0
  until: somecondition

By default 'times' would be 1. If times == 0 it would check that an 'until' condition has been set and repeat tagged tasks until that condition becomes true.

In most cases it's not needed - Modules already take care of most of the provisioning needs, for instance, "exact_count" on provisioning modules allow spinning up an arbitrary number of images, and then you can apply tasks to images using the host loop.   Thus we can ALREADY do all of that cluster spinup, and do that every day.

While looping over multiple tasks in a common loop could be something Ansible does in the future, we have a lot of more important things to chase first in the PR queue before we would consider such complexity when so few things need it.   So far the only example we have that we "can't" do is "warm up group of servers A from load of B until servers A reach threshold X", which strikes me as very arbitrary.

Other cases:
- Stress testing a group of hosts by repeating a group of tasks X times
- Run a time-dependent port knocking scheme X times to pass through a firewall
- Run X passes of a series of data-destructive tasks on some disk containing your secrets

And generally repeating any couple of "run task & check for output" tasks.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgz%3DH1zun_xfbFCU0buvAzSQVDSm8a6hwxsLRCh8AH3cug%40mail.gmail.com.

[Brian Coca]

we have a do/until already

--
Brian Coca
Stultorum infinitus est numerus
0110000101110010011001010110111000100111011101000010000001111001011011110111010100100000011100110110110101100001011100100111010000100001

Pedo mellon a minno

[Daniel Siechniewicz]

Hi,

Sorry to reanimate this, but I think I have a "legit" use case for looping over multiple tasks:

This is what I've got:

- name: install cq packages

  local_action: cqpkg name={{ item.pkg }}

     path={{ item.file }}

     state=present

     admin_user={{ admin_user }}

     admin_password={{ admin_pass }}

     host={{ ansible_eth1.ipv4.address }}

     port={{ listen_port }}

  with_items: cq_packages

But really I need (pretend syntax of course):

- name: install cq packages

  # Install package

  - local_action: cqpkg name={{ item.pkg }}

     path={{ item.file }}

     state=present

     admin_user={{ admin_user }}

     admin_password={{ admin_pass }}

     host={{ ansible_eth1.ipv4.address }}

     port={{ listen_port }}

  # Restart service if package requires it to be done immediately

  - service: name={{ service_name }} enabled=yes state=started

    when: {{ item.cqrestart }} == immediate

  # Wait for the port to come up (this can happen a while after restart)

  - wait_for: port={{ listen_port }} delay=1

  # Wait for the darned thing to come back, and this can take AGES

  - shell: "{{ aem_check }}"

    until: result.stdout.find("200 OK") != -1

    retries: 180

    delay: 10

    register: result

    changed_when: False

  # Rinse, repeat

  with_items: cq_packages

Because I'm using an ansible module to do actual installation (cqpkg) doing it as a script would require some serious rewriting OR doing all the restarts and waiting and checking within the module. Waiting and checking I'd be reasonably OK with, but not service restart, to be honest. The restart cannot be a handler either, because some packages require immediate restart (application can break beyond recovery if cq is not restarted there and then). As far as I managed to understand flush_handlers this won't work in a loop either, not to mention unintended consequences, as it's all or nothing.

This can, and most likely will be, separated out into a parameterised role, but it'll force fairly non-intuitive "flow.

This really seems to be a simple example of why loop over multiple tasks makes sense: I want to install a package, restart a service and wait for things to settle down before proceeding with installation of next package.

I'll need to come up with a solution pretty much today, so this is just for general discussion. 

Regards,

Daniel

[Daniel Siechniewicz]

Or is this a role for include + with_items that will be available in 2.0?

Regards,
Daniel

> You received this message because you are subscribed to a topic in the

> Google Groups "Ansible Project" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/ansible-project/tisr0c8eovc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

> ansible-proje...@googlegroups.com.
> To post to this group, send email to ansible...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/ansible-project/8cc731d9-bbc6-49b4-97c6-4afd568c1939%40googlegroups.com.

[Brian Coca]

yes

On Fri, Apr 17, 2015 at 10:33 AM, Daniel Siechniewicz
<dan...@siechniewicz.com> wrote:
> Or is this a role for include + with_items that will be available in 2.0?
>
> Regards,
> Daniel
>
>

--
Brian Coca

[Nick Tkach]

Can you expand on what you mean by "include + with_items"?  I have a use case that is almost identical to what Daniel gives above.  Need something like this (I'm sure it's not exactly right):

roles/httpd/tasks/main.yml:
---
- block:
  - include_vars: {{ item }}
  - debug: msg="{{ field1 }} from with list"
  - debug: msg="{{ field2 }} from with list"
  - template: src="templates/site.conf.j2" dest="site.conf"
  with_fileglob:
    - vars/*.yml

So basically pulling in a group of variables file one at a time (*not* merging) and running through an *entire* set of tasks on each, one at a time.

[Brian Coca]

- include: file.yml

   with_fileglob:

          - vars/*.yml

file.yml

  - include_vars: {{ item }}

  - debug: msg="{{ field1 }} from with list"

  - debug: msg="{{ field2 }} from with list"

  - template: src="templates/site.conf.j2" dest="site.conf"

----------

Brian Coca

[Nick Tkach]

Thank you very much!  No wonder I hadn't figured this out earlier!  I'm sure there's just some aspect I never looked into.  Just for the next person who comes along what I got to work was

playbook.yml

---
- hosts: localhost
  become: false

  tasks:
    - include: "roles/httpd/tasks/main.yml"
      with_fileglob:
        - vars/*.yml

roles/httpd/tasks/main.yml:

---
- include_vars: "{{ item }}"
- debug: msg="{{ site_name }}"

vars/site1.yml:

---
site_name: "site1"
http_port: 8888

vars/site2.yml:

---
site_name: "site2"
http_port: 8899

ansible-playbook playbook.xml does give the output I needed (snippet):

ok: [localhost] => {
    "msg": "site1"
}
ok: [localhost] => {
    "msg": "site2"

[Raghavendra Rao]

Hi. I think this situation hasn't changed in ansible v2.7 as well.

There is no way I can find to repeat a particular task or a set of tasks within an existing playbook.

It would have been great if we could use tags to call some "tagged-tasks again" wherever we want in a playbook OR create a new feature like 'goto or goback' as discussed in the above threads.

My scenario:

I have to deal with 3 tasks A, B and C. Here C is dependent on A and B's performance. i.e If C's output is not as expected, then again A and B have to be repeated again. And somehow I cannot fit this in a 'block-rescue-always' structure as its pointless.

Right now, I manage this using 'include_task' option along with 'when' conditional. I re-enter A and B tasks into a separate '.yml' file and then call it into this playbook using 'include_task' wherever I want.

This solved my purpose but is ridiculous especially when we have so many other options to effectively orchestrate things using ansilble.

Any inputs/thoughts on this would be helpful for me to explore more.

[Karl Auer]

If there is some condition that can be tested externally, then why not a simple script that calls ansible-playbook until that condition obtains?

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/61c4f6a8-100e-4829-8f01-cc0d72f21800%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Karl Auer

Email  : ka...@2pisoftware.com
Website: http://2pisoftware.com

GPG/PGP : 958A 2647 6C44 D376 3D63 86A5 FFB2 20BC 0257 5816
Previous: F0AB 6C70 A49D 1927 6E05 81E7 AD95 268F 2AB6 40EA