Ansible Roles in 1.8.4 - Still inferior to include files for complex playbooks?

[jack.ma...@gmail.com]

From Ansible doc:

Roles in Ansible build on the idea of include files and combine them to form clean, reusable abstractions

I really want to love roles and use them extensively in our system, but so far I'm getting a heap of frustration instead. I am trying to position Ansible at work as the preferred automation platform to manage a complex system with several internally developed apps, plus db, web server, and more. (together this is one product/system) Multiple environments too, as usual.

I'd think it makes sense to have roles like:
- appX (for X in 1 through N)
- web
- db
- etc (more.. skipped)

One seemingly basic thing I need to do is to restart this entire system occasionally. I would like to use Ansible to do that. Here's a simplified pseudo-code recipe of a playbook:
- web.startMaintenance()
- app1.stop()
- app2.stop()
- app2.start()
- app1.start()
- web.endMaintenance()

I want to run a single playbook command to do this. That seems clearly possible with include files but not possible with roles. The following doesn't work:

ansible-playbook --tags "app1.stop,app1.start,app2.start,app2.stop,web.startMaintenance,web.endMaintenance" playbooks/testRoleDeps.yml

That doesn't work because all plays in the playbook are filtered by the tags provided, and each play needs to have a particular tag applied (first stop then start later), not all of them.

My conclusion? As of 1.8.4, roles are inadequate for complex playbooks and include files need to be used instead.

That's unfortunate for code reuse reasons, etc. I would love to hear that this is not the case and I have overlooked something. Maybe a feature to resolve this is in the release plan?

I can't figure out how to do this even after reading the documentation repeatedly, reading the ansible O'Reilly book (pre-release copy), the code in ansible-examples, and various other resources.

[Timothy Gerla]

Some replies inline!

On Mon, Feb 23, 2015 at 1:41 PM, <jack.ma...@gmail.com> wrote:

From Ansible doc:

Roles in Ansible build on the idea of include files and combine them to form clean, reusable abstractions

I really want to love roles and use them extensively in our system, but so far I'm getting a heap of frustration instead. I am trying to position Ansible at work as the preferred automation platform to manage a complex system with several internally developed apps, plus db, web server, and more. (together this is one product/system) Multiple environments too, as usual.

I'd think it makes sense to have roles like:
- appX (for X in 1 through N)
- web
- db
- etc (more.. skipped)

That's usually the case for simple stuff, but for each app or component of the system, I usually end up with multiple roles. Maybe a role, a role to configure, a role to restart. When you start wanting to call individual parts of a single role separately, that's a good sign that you need to break that role up into multiple parts. For organizational purposes, you can definitely do:

roles/web/setup/*

roles/web/install/*

roles/web/config/*

roles/web/restart/*

 

One seemingly basic thing I need to do is to restart this entire system occasionally. I would like to use Ansible to do that. Here's a simplified pseudo-code recipe of a playbook:
- web.startMaintenance()
- app1.stop()
- app2.stop()
- app2.start()
- app1.start()
- web.endMaintenance()

I want to run a single playbook command to do this. That seems clearly possible with include files but not possible with roles. The following doesn't work:

ansible-playbook --tags "app1.stop,app1.start,app2.start,app2.stop,web.startMaintenance,web.endMaintenance" playbooks/testRoleDeps.yml

It seems like a single playbook for all of these different configurations might not be realistic. Why not have multiple playbooks, one per general operation, and call the roles as needed?

 

That doesn't work because all plays in the playbook are filtered by the tags provided, and each play needs to have a particular tag applied (first stop then start later), not all of them.

My conclusion? As of 1.8.4, roles are inadequate for complex playbooks and include files need to be used instead.

Roles are pretty simple--for more complex purposes, you may want to mix in playbook includes too. That's fine! I wouldn't get too hung up on getting all of the behavior for a certain part of your app in a single role. That's going to be confusing and complicated to use.

That's unfortunate for code reuse reasons, etc. I would love to hear that this is not the case and I have overlooked something. Maybe a feature to resolve this is in the release plan?

I can't figure out how to do this even after reading the documentation repeatedly, reading the ansible O'Reilly book (pre-release copy), the code in ansible-examples, and various other resources.

I'm not sure if I understand any specific requests for improvement here--besides being able to use a single playbook to drive multiple operations, which is not really Ansible-style. If I've misunderstood, feel free to let me know! :)

-Tim

[Jack Martin]

On Mon, Feb 23, 2015 at 5:17 PM, Timothy Gerla <t...@gerla.net> wrote:

Some replies inline!

Thanks for all the help! I'll reply inline too.
 

On Mon, Feb 23, 2015 at 1:41 PM, <jack.ma...@gmail.com> wrote:

From Ansible doc:

Roles in Ansible build on the idea of include files and combine them to form clean, reusable abstractions

I really want to love roles and use them extensively in our system, but so far I'm getting a heap of frustration instead. I am trying to position Ansible at work as the preferred automation platform to manage a complex system with several internally developed apps, plus db, web server, and more. (together this is one product/system) Multiple environments too, as usual.

I'd think it makes sense to have roles like:
- appX (for X in 1 through N)
- web
- db
- etc (more.. skipped)

That's usually the case for simple stuff, but for each app or component of the system, I usually end up with multiple roles. Maybe a role, a role to configure, a role to restart. When you start wanting to call individual parts of a single role separately, that's a good sign that you need to break that role up into multiple parts. For organizational purposes, you can definitely do:

roles/web/setup/*

roles/web/install/*

roles/web/config/*

roles/web/restart/*

This is very helpful. I have provided some sample code below showing how I have used this to do what I intended.

 

One seemingly basic thing I need to do is to restart this entire system occasionally. I would like to use Ansible to do that. Here's a simplified pseudo-code recipe of a playbook:
- web.startMaintenance()
- app1.stop()
- app2.stop()
- app2.start()
- app1.start()
- web.endMaintenance()

I want to run a single playbook command to do this. That seems clearly possible with include files but not possible with roles. The following doesn't work:

ansible-playbook --tags "app1.stop,app1.start,app2.start,app2.stop,web.startMaintenance,web.endMaintenance" playbooks/testRoleDeps.yml

It seems like a single playbook for all of these different configurations might not be realistic. Why not have multiple playbooks, one per general operation, and call the roles as needed?

The entire mission of Ansible is "simple IT automation", right? :) Wherever possible, my preference would be for a single Ansible playbook to handle the execution of a single high-level "goal"/"user request". If that isn't the case, then one needs to somehow bundle those externally, typically in a script. That requires the local "Ansible team" to decide on and maintain knowledge in such a scripting language, and requires implementing error checking and possibly error recovery in the scripting language. Sure.. it isn't strictly necessary, but I would rather just call a script "restart_system.bash" that makes a call to a single ansible playbook instead of seeing the bash script make a call to 6 ansible playbooks. What happens if the playbooks need to pass state or other data between each other? Much cleaner if it occurs in a single call, no?

 

That doesn't work because all plays in the playbook are filtered by the tags provided, and each play needs to have a particular tag applied (first stop then start later), not all of them.

My conclusion? As of 1.8.4, roles are inadequate for complex playbooks and include files need to be used instead.

Roles are pretty simple--for more complex purposes, you may want to mix in playbook includes too. That's fine! I wouldn't get too hung up on getting all of the behavior for a certain part of your app in a single role. That's going to be confusing and complicated to use.

That's unfortunate for code reuse reasons, etc. I would love to hear that this is not the case and I have overlooked something. Maybe a feature to resolve this is in the release plan?

I can't figure out how to do this even after reading the documentation repeatedly, reading the ansible O'Reilly book (pre-release copy), the code in ansible-examples, and various other resources.

I'm not sure if I understand any specific requests for improvement here--besides being able to use a single playbook to drive multiple operations, which is not really Ansible-style. If I've misunderstood, feel free to let me know! :)

One request would be that more complex cases/uses of Ansible be documented more clearly, and specifically what you wrote above regarding the use of "sub-roles": web/setup, web/restart, etc.

-Tim

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAH4wdVWc73Hc9x19AmiZ2CbZNJZPW2FyKv%3D7g-w88qnQtQNeQQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

I wrote some test roles to verify what you described would work:

ansible$ find roles/{web,app1,app2} -type f | sort
roles/app1/startServer/tasks/main.yml
roles/app1/stopServer/tasks/main.yml
roles/app2/startServer/tasks/main.yml
roles/app2/stopServer/tasks/main.yml
roles/web/startMaintenance/tasks/main.yml
roles/web/startServer/tasks/main.yml
roles/web/stopMaintenance/tasks/main.yml
roles/web/stopServer/tasks/main.yml

All of those files have what amounts to the the following code:

ansible$ cat roles/web/startMaintenance/tasks/main.yml
---
- name: Run first task for startMaintenance
  debug: msg="Doing first task"
- name: Run second task for startMaintenance
  debug: msg="Doing second task"

Each role gets a playbook for testing and when separate execution happens to be needed:

ansible$ find playbooks/test/ -type f | grep -v restart | sort
playbooks/test/app1_startServer.yml
playbooks/test/app1_stopServer.yml
playbooks/test/app2_startServer.yml
playbooks/test/app2_stopServer.yml
playbooks/test/web_startMaintenance.yml
playbooks/test/web_stopMaintenance.yml

Each of those has some trivial, essentially identical code:

ansible$ cat playbooks/test/web_startMaintenance.yml
---
- hosts: 127.0.0.1 # Imagine 'web'
  connection: local

  roles:
  - web/startMaintenance

Finally.. the restart playbook:

ansible$ find playbooks/test/ -type f | grep  restart
playbooks/test/restart_system.yml
ansible$ cat playbooks/test/restart_system.yml
---
- include: web_startMaintenance.yml
- include: app1_stopServer.yml
- include: app2_stopServer.yml
- include: app2_startServer.yml
- include: app1_startServer.yml
- include: web_stopMaintenance.yml

Is there something about the way restart_system.yml is written above that makes it "not Ansible-style" or that should be expected to lead to problems? Thanks again.

[Timothy Gerla]

Hey Jack,

The entire mission of Ansible is "simple IT automation", right? :) Wherever possible, my preference would be for a single Ansible playbook to handle the execution of a single high-level "goal"/"user request". If that isn't the case, then one needs to somehow bundle those externally, typically in a script. That requires the local "Ansible team" to decide on and maintain knowledge in such a scripting language, and requires implementing error checking and possibly error recovery in the scripting language. Sure.. it isn't strictly necessary, but I would rather just call a script "restart_system.bash" that makes a call to a single ansible playbook instead of seeing the bash script make a call to 6 ansible playbooks. What happens if the playbooks need to pass state or other data between each other? Much cleaner if it occurs in a single call, no?

Sorry, I think I misunderstood your intent here. I definitely agree with a single playbook for a single high level user request. I jumped to the conclusion that you wanted to encode a bunch of different user requests in a single playbook.

<snip>

I would probably structure this a little bit differently, going back a little bit on what I said above about a single playbook per user request. This might actually be a really good use case for tags.

playbooks/test/restart_system.yml:

---

- hosts: 127.0.0.1

  connection: local

  roles:

  - { role: web/startMaintenance, tags: ['web'] }

  - { role: app1/stopServer, tags: ['app1'] }

  - { role: app2/stopServer, tags: ['app2'] }

  - { role: app1/startServer, tags: ['app1'] }

  - { role: app2/startServer, tags: ['app2'] }

  - { role: web/stopMaintenance, tags: ['web'] }

Now you can invoke it with:

ansible-playbook restart_system.yml --tags app1

...to only restart app1, or without tags to restart everything. (Maybe the web start/stop should be untagged, so it runs every time--not sure. Depends on your app.)

This way you lose the intermediary playbooks, and end up with a single playbook to do all of your restart tasks.

If app1 and app2 roles are really close together, maybe you could parameterize them, too:

- { role: genericApp/stopServer, appName: "app1", tags: ["app1"] }

-Tim

[Tom Bamford]

I also find that smaller, job-oriented roles work well in scenarios like this. It makes playbooks more maintainable and they read better when you come back to them 3 months later.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAH4wdVX2Dfh3he3u%3Dza1heytT9Hy6shsDGawDQ6GS7RTTbnT5Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Tom Bamford

@Planet

ATPLANET (Pty) Ltd

atpla.net

Cell: +27 (0)79-095-7112

Fax: +27 (0)86-599-1310

[Brian Coca]

you can have it both ways, you can have smaller playbooks that do part
of those functions and a wrapper one that includes the smaller ones,
this is a common pattern with site.yml (run everything).




--
Brian Coca

[jack.ma...@gmail.com]

This is a fine refactor of my demo code, but the real production code can't do this because app1 is on "app1 server cluster" and app2 is on "app2 server cluster", etc You need a separate play for each call so that you can use the "hosts" field to indicate which server set it is being applied to. Right?

Now you can invoke it with:

ansible-playbook restart_system.yml --tags app1

Thanks for pointing this out.

...to only restart app1, or without tags to restart everything. (Maybe the web start/stop should be untagged, so it runs every time--not sure. Depends on your app.)

This way you lose the intermediary playbooks, and end up with a single playbook to do all of your restart tasks.

I don't dislike the intermediary playbooks, since they can be used by themselves and more easily composed together.

If app1 and app2 roles are really close together, maybe you could parameterize them, too:

- { role: genericApp/stopServer, appName: "app1", tags: ["app1"] }

I will try to find an appropriate place to use this technique. Thanks!

-Tim