All about Roles in 1.2 -- a great way to organize best-practices content

[Michael DeHaan]

Folks have been asking how to reuse pieces of playbooks easily, and I have also been wanting to squash the need to use relative or absolute paths all over the place.

Done!  Presenting roles! (in 1.2 devel)

Roles are mostly a way to save boilerplate -- using a role automatically puts content in tasks, vars_files, and handlers for you. 

I will admit first hand this structure looks a little bit like a puppet module, so I need to give credit where credit is due -- I got this idea from that originally, because people wanted an analog of that, but have improved it to make paths not as fiddly.  I hope this completely squashes the reuse of parts of playbooks now, and also enables git-submodule style workflows for those that want them.

There are some cool new things too, namely, variables are also loaded (via the vars_files internals), in addition to just tasks, so you can put less in group_vars now if you like.

You can also parameterize roles, so if you had, for example, a role for all the users you wanted to install, you could just include that list of users, and boom, it would work.

See details here and let me know what you think!

https://github.com/ansible/ansible/blob/devel/examples/playbooks/roletest.yml

I want to make it support lists of roles in the roles list, so I could do things like
define my roles in vars_files (or group_vars) for the users I want to include,

and then slurp them all in and make them exist.

Unlike included playbooks in playbooks, roles also minimize fact gathering, so I suspect the only use case for included playbooks now is when you want a site.yml that includes playbooks for all of your different group operations, and things like that.

Hope this is not too confusing and we will introduce this a lot more casually in the docs (probably on the best practices page).

Comments welcome!

--

Michael DeHaan <mic...@ansibleworks.com>
CTO, AnsibleWorks, Inc.
http://www.ansibleworks.com/

[Marco Vito Moscaritolo]

A.W.E.S.O.M.E.!!!!!!!!

I have no opportunity to try now (ansible can be run fom smartphone?? :) ) but do you know if it works with with_items statement?
Eg:

- roles:
     - { role : 'application', project: $item }
        with_items: $projects

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Michel Blanc]

On 06/04/2013 19:07, Michael DeHaan wrote:
> Folks have been asking how to reuse pieces of playbooks easily, and I
> have also been wanting to squash the need to use relative or absolute
> paths all over the place.
> Done! Presenting roles! (in 1.2 devel)
>

> Comments welcome!

Hello,

While I havent completely figured out how this will affect my setup here
(I'm the "playbook-in-playbook" type, so I will certainly use roles), it
looks like a neat addition !

However, when I saw the syntax, my first reaction was "eyebrow raising".
Why that mustache-enclosed sort of thing ?

AFAIK, this is the first time we have them beside Jinja and the new
Jinja style variables in playbooks (which is, btw, a really great
addition and opens very interesting possibilities like "variables
referencing vars" working eveywhere).

Wouldn't it be more expressive and ansible-looking like this :


---
- hosts: all
roles:
- foo:
vars:
param1: 1000
param2: 2000
- bar:
vars:
param1: 1000
param2: 2000

Agreed, it's (much) more verbose, but it looks more consistent with the
rest of Ansible. And it let's the door open to dict variables, and may
be other parameters passing besides 'vars:'.

On 07/04/2013 09:57, Marco Vito Moscaritolo wrote:
> do you know if it works with with_items statement?
> Eg:
>
> - roles:
> - { role : 'application', project: $item }
> with_items: $projects

Wow that would be great.


Michel.
--
Michel Blanc - netWorks
8A68 0871 747A 65B6 E87C 3BEA 187C 36BB 2CE5 68BD

[Michael DeHaan]

However, when I saw the syntax, my first reaction was "eyebrow raising".
Why that mustache-enclosed sort of thing ?

You can still spell out the hash, this has always been legal in Ansible YAML.

 

[Michael DeHaan]

It doesn't yet, but it's on my list.

I want some slightly more idiomatic variation of it though.

with_items (or something similar) should need to be inside the hash or something so the syntax is valid.

TBD!  (suggestions welcome!)

[Michel Blanc]

Oh ok, sorry, didn't know that.

May be it would be more expressive this way in the docs then :

---
- hosts: all
roles:

- role: foo
param1: 1000
param2: 2000
- role: foo
param1: 8000
param2: 9000

Less puzzling for non YAML experts !

M

[Michael DeHaan]

kind of liked it short, but will likely make it take the usual
key=value as well, yes!

-- Michael

[Walid]

Great!,  this is exactly one feature we would like to have

[GW]

Hi,

I would just like to check if I get it correctly, that this "roles" are
just a shortcut for those who do not want to explicitly write out
everything and actually do not add any new feature different to what
already exists?

It is therefore a less flexible and explicit, but otherwise exactly the
same
thing as doing:


# queues.yml (playbook for group "queues")
---
- include: app1/task_queue.yml hosts=all user=root


# app1/task_queue.yml (playbook for task_dispatcher flavor of the app1
role)
---
- hosts: ${hosts}
user: ${user}
vars_files:
- include: vars/task_queue.yml
tasks:
- include: tasks/task_queue.yml
handlers:
- include: handlers/task_queue.yml


In this example "app1" is a folder containing everything connected with
the "app1"
application. But the application consists of multiple parts that could
be deployed
on the same server or over a couple of servers, each making its part of
the job to
make "app1" work. Each of this parts therefore represents a subrole or
role flavor,
eg (this are all just playbooks further including appropriate tasks
list):

app1/web1.yml <-- responsible for configuring the web server and
deploying web accessible files for the first website of app1
app1/web2.yml <-- responsible for configuring the web server and
deploying web accessible files for the second website of app1
app1/web3.yml <-- responsible for configuring the web server and
deploying web accessible files for the third website of app1
app1/database.yml <-- responsible for configuring the database and
importing some initial data
app1/task_queue.yml <-- responsible for dispatching long running jobs
to other servers/workers
app1/worker.yml <-- responsible for executing the enqueued job
...
app1/main.yml <-- just a shortcut to put all above sub-playbooks
on the same server

For production where each server group runs on a different server there
are separate playbooks, eg:

webservers.yml <-- includes app1/web1.yml, app1/web2.yml...,
app2/web.yml, app3/web.yml...
queues.yml <-- includes app1/task_queue.yml, app2/task_queue.yml,
app3/task_queue.yml...
...
site.yml <-- just a shortcut to include all of the above
playbooks


For first I think that it would be better to not only support including
"main.yml"
files, because often a need arises to have multiple subroles or flavors
of roles.
(Yes, they could be split into multiple separate folder structures for
each subrole,
but this would just create an unnecessary mess, because large parts of
instructions
for each variant will probably be shared between all of them.)

To me the above is pretty straightforward, understandable and seems
much more flexible
than using the newly introduced roles.

The biggest problem (what I hoped some smart roles would solve) is the
same why at
Puppetlabs they created Hiera
(http://projects.puppetlabs.com/projects/hiera). That
there is no way for each role to have default values that can be
overriden per-host
or per-group. This consists of 2 problems:

1. It is not possible to define default variables in tasks list. Either
you hard code
the values, create an additional layer of playbooks (like above) or
force the playbook
using this task list to always define them all, or convince the
playbook user to manually
copy them to his "group_vars/all" file. (An end-user usually does not
want to care about
all the details that can be configured and would be happy with default
values a role he
found on the internet brings with it.)

2. The variable precedence order is wrong, because playbook vars and
vars_files have
precedence over hosts_vars and group_vars (what doesn't make much sense
to me). This is
actually the purpose of Puppetlabs Hiera - to have the ability to
override default module
values on per-host or per-group level.

To make such overriding simpler to use it would be great to have
namespaces, eg. per-role
namespace "app1" and than all variables within it could be accessible
or overriden with
something like "app1::database_ip" or without introducing new things
and using macros for
namespaces like "$ROLE_VAR(app1, database_ip)". Macros could also be
used to solve part of
the motivation why this new "roles" were introduced - using relative or

absolute paths all
over the place.

What do you think?

Greetings,
gw

> [1]

>
> I want to make it support lists of roles in the roles list, so I
> could do
> things like
> define my roles in vars_files (or group_vars) for the users I want to
> include,
> and then slurp them all in and make them exist.
>
> Unlike included playbooks in playbooks, roles also minimize fact
> gathering, so I suspect the only use case for included playbooks now
> is
> when you want a site.yml that includes playbooks for all of your
> different group operations, and things like that.
>
> Hope this is not too confusing and we will introduce this a lot more
> casually in the docs (probably on the best practices page).
>
> Comments welcome!
>
> --
>
> Michael DeHaan

> CTO, AnsibleWorks, Inc.
> http://www.ansibleworks.com/ [3]

>
> --
> You received this message because you are subscribed to the Google
> Groups "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it,
> send
> an email to ansible-proje...@googlegroups.com.

> For more options, visit https://groups.google.com/groups/opt_out [4].



Links:
------
[1]
https://github.com/ansible/ansible/blob/devel/examples/playbooks/roletest.yml
[2] mailto:mic...@ansibleworks.com
[3] http://www.ansibleworks.com/
[4] https://groups.google.com/groups/opt_out

[Michael DeHaan]

On Sun, Apr 7, 2013 at 9:06 AM, GW <gw....@tnode.com> wrote:

Hi,

I would just like to check if I get it correctly, that this "roles" are
just a shortcut for those who do not want to explicitly write out
everything

It's more than that.

They are a shortcut, and also make content reusable, and also allow easy relative imports of files and templates, making them reusable and easy to share.

 

and actually do not add any new feature different to what
already exists?

I'd consider upgrades to organization and reduction in boilerplate a new feature :)

 

It is therefore a less flexible and explicit, but otherwise exactly the same
thing as doing:

Your paths are not correct, but basically it does expand to a lot of imports.

The paths a 'foo' role would include are actually:

roles/foo/tasks/main.yml

roles/foo/handlers/main.yml

roles/foo/vars/main.yml

And no, it's not exactly the same, because if any of those files are *not* present, it's cool with that and would not produce an error.

You can of course import something else from main.yml if you want, and relative paths work GREAT in this case, which is what we want.

]nd group_vars (what doesn't make much sense to me). This is

actually the purpose of (...)

Nope, it's not.   It's completely not the purpose of that.  But I won't tell you what I really think as it's off topic to discuss that application on this list.

It doesn't make sense to have an inventory script drive roles.

I've always said "the purpose of a play is to map a set of hosts to the roles they should perform" (at a particular point in time), which is what this makes much more obvious, and it provides more elegant ways to reuse and share content, and also gets rid of a ton of games you have to play with "../" to find files and templates, or even reference things inside plays.

I think you would really like them if you used them, but if you have existing content you don't want to reorganize, you don't have to use it :)

[Ashley Penney]

Hey,

First I want to thank you for this feature as it's awesome!  I've been playing with roles over the weekend and reshaping the stuff I've been working on to fit better with roles.  I did however come across a use case I wanted to discuss with you and see how you envision it working.

I have a role, "application" that contains tasks/main.yml and deploy.yml.  The main.yml is the base configuration we want to constantly manage and deploy.yml is (unexpectedly) the deployment steps for updates.  With the current roles setup I can only get main.yml included, I believe.  Would you envision us putting the deployment steps into main.yml and perhaps tagging them and using tags with roles to select functionality, or expanding it so we can do: role: 'application:deploy' or something to select other yaml.

I know to a degree I'm bending the way ansible works, but with roles it seems like other people may have a similar use case to me, routine stuff they want included by the role repeatedly and then the ability to trigger other .yml files to do certain procedures.

I suspect there's a more ansible way to do this and I'm still just so influenced by Puppet that I can't see what it is.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[GW]

Hi,

I am also in the process of reorganizing things to roles and the approach I am currently using (also influenced by Puppet) is:

# web.yml

---
- hosts: all
  roles:

    ## Application roles
    - role: app1
      install: yes
      config: yes
      deploy: yes
      test: yes 

# app1/tasks/main.yml
---
- include: install.yml
  only_if: ${install}
- include: config.yml
  only_if: ${config}
- include: deploy.yml
  only_if: ${deploy}
- include: test.yml
  only_if: ${test} 

Is this the way it is meant to be used? Or am I again trying to go in the wrong direction?

Greetings,
   gw

[Michael DeHaan]

I have a role, "application" that contains tasks/main.yml and deploy.yml.  The main.yml is the base configuration we want to constantly manage and deploy.yml is (unexpectedly) the deployment steps for updates.  With the current roles setup I can only get main.yml included, I believe.  Would you envision us putting the deployment steps into main.yml and perhaps tagging them and using tags with roles to select functionality, or expanding it so we can do: role: 'application:deploy' or something to select other yaml.

This is not true.

In the middle of main.yml all you have to do is:

---

- include: deploy.yml tags=deploy

And that will include deploy.yml and auto-tag everything in it with the tag "deploy"

 

I suspect there's a more ansible way to do this and I'm still just so influenced by Puppet that I can't see what it is.

I am considering starting a rehab center in Antigua.

 

[Michael DeHaan]

On Mon, Apr 8, 2013 at 11:01 AM, GW <gw....@tnode.com> wrote:

Hi,

I am also in the process of reorganizing things to roles and the approach I am currently using (also influenced by Puppet) is:

# web.yml

---
- hosts: all
  roles:

    ## Application roles
    - role: app1
      install: yes
      config: yes
      deploy: yes
      test: yes 

 

# app1/tasks/main.yml
---
- include: install.yml
  only_if: ${install}
- include: config.yml
  only_if: ${config}
- include: deploy.yml
  only_if: ${deploy}
- include: test.yml
  only_if: ${test} 

Is this the way it is meant to be used? Or am I again trying to go in the wrong direction?

Yeah, this is a bit overwrought.    It sounds like you want to decide what to run each time, which are what tags do.

You should have one role for your app, and main.yml looks like:

----

- include: install.yml tags=install

- include: config.yml tags=config

- include: test.yml tags=test

And then by default it will run everything, but you can also do:

ansible-playbook site.yml --tags=config

to just run the config steps

 

Greetings,
   gw

 

[Ashley Penney]

On Mon, Apr 8, 2013 at 11:12 AM, Michael DeHaan <mic...@ansibleworks.com> wrote:

I have a role, "application" that contains tasks/main.yml and deploy.yml.  The main.yml is the base configuration we want to constantly manage and deploy.yml is (unexpectedly) the deployment steps for updates.  With the current roles setup I can only get main.yml included, I believe.  Would you envision us putting the deployment steps into main.yml and perhaps tagging them and using tags with roles to select functionality, or expanding it so we can do: role: 'application:deploy' or something to select other yaml.

This is not true.

In the middle of main.yml all you have to do is:

---

- include: deploy.yml tags=deploy

And that will include deploy.yml and auto-tag everything in it with the tag "deploy"

Hmm, but in this case I don't always want to include deploy.yml every time.  I want to only include it in special playbooks that do deploys, or perhaps when --extra_vars deploy=true is set, or something similar.  I guess I can probably do a with_boolean: with an include?  That would probably solve a lot of my dilemma by having deploy: false as the default and just overriding it when needed.  I could maybe copy the other examples I've seen have and have configuration: true/false as well.   I basically need to be able to -just- run deployment steps and ignore base configuration when needed.  I've been playing and restructuring and rewriting things every 10 minutes trying to find the best and most simple way to handle this.

[Michael DeHaan]

Ok, so if you don't want to include deploy except in playbooks that do deploys, don't put it as a role in that playbook!  :)

I think the greatest snag most people thing of in Ansible is they try to turn it into some kind of programming language, versus a model of what their systems should be or do.

In this case, roles say "these are the roles this group of hosts should perform", so if this playbook should not apply these roles at all, ever, they shouldn't be included!

You can of course break things into smaller roles if this works better for you.

Of course, tags are there, and are a great way to conditionally decide how to run ansible.

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[GW]

Hm, if it would be possible to access current tags as a variable one could do things like:

- debug: msg="foo"
  only_if: 'deploy' in $tags
  tags: deploy

This would execute if and only if --tags=deploy is given.

Or is it already possible to access hosts, user, tags in playbooks/task lists?

Greetings,
   gw

[Michael DeHaan]

If you do "--tags deploy" it will only run tasks or plays that are tagged with deploy already.

There is no need for the extra conditional.

[GW]

I mean: It will run only if "--tags deploy" is given, otherwise it will skip this step. Eg. if you do not pass any --tags or pass other --tags, the task will be skipped. It would function as a default skip policy if the right tag is not specified.

Anyway, it was just a thought...

Greetings,
   gw 

[Michael DeHaan]

This sounds like you would just want some configuration setting that defined some default tags, and the default in the config file could be all.

I would accept a patch for this, but your conditionals are not needed.

[Michael DeHaan]

The last piece that still feels missing to me (at least for my use case) is the templates and files. With the new roles structure the default location for these is relative to the role directory. This works fine when the template as is can be sufficiently represented with various jinja interpolations. And for files its okay when the file is model/role specific and not app+environment relevant. However I have use cases where the app+environment can(/may need to) override the whole template, and/or potentially use its own file/* ref'd in the playbook. Would it make sense to lookup templates and files first relative to the hosts file, then fallback to role location? 

You can still reference other paths in these cases by specifying a different path, it looks in two places, basically.

I do not think looking in a path relative to the host file is neccessarily a good idea, just because that I think crosses the streams too much between inventory data (this is something about my geography or hosts) and playbooks (this is a description about what I am deploying onto these hosts).

 

This is something I do today which allows the app to own its own templates & files (pulled from their git repo), if desired, via first_available_file and some path hackery. But it seems to make sense to have this capability 'built in' where the templates/files default to the roles subdir but can be overriden relative to the 'inventory' if necessary.

Hmm, yeah I understand, but am worried if we include too much magic the playbook could not be read "at first glance" to know exactly what it would do, and that behavior is also important to me.

 

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Stoflet Darryl]

Agreed regarding too much magic. I found one way to do this simply using the 1.2 devel branch:

- name: copy operation
  copy: src=$item dest=/tmp/roles_test1.txt
  first_available_file:
    - {{vars.inventory_dir}}/files/foo.txt
    - foo.txt

- name: template operation
  template: src=$item dest=/tmp/roles_test2.txt
 first_available_file:
    - {{vars.inventory_dir}}/templates/foo.j2
    - foo.j2
  notify:
    - blippy

I am happy! I hope I am not doing anything that will not be supported going forward.

[Stoflet Darryl]

Disregard this. The roles implementation at this point is inconsistent as far as template and copy go when using first_available_file. W/o and directly using src is will search relative to the role dir (as in the roletest.yml example). W/ first_available_file it will go back to using the playbook source dir :-(. 

Should I file a bug or is this implementation still in the works?

One other note: I noticed that using {{ vars.inventory_dir }} actually generated an error: recursive loop detected in template string: {{vars.inventory_dir}}. Oddly though using {{vars.inventory_dir}} as a debug msg value worked. However ${vars.inventory_dir} worked in both cases.

So the below works, but as mentioned above first_available_file is relative to playbook dir which kinda defeats the purpose of the roles dir structure.

$ /ansible/examples/playbooks$ cat roles/bar/tasks/main.yml 

---

- name: starting test

  debug: msg=${vars.inventory_dir}

- name: copy operation

  copy: src=$item dest=/tmp/roles_test1.txt

  first_available_file:

    - '${vars.inventory_dir}/files/foo.txt'

    - roles/bar/files/foo.txt

- name: template operation

  template: src=$item dest=/tmp/roles_test2.txt

  first_available_file:

    - '${vars.inventory_dir}/templates/foo.j2'

    - roles/bar/templates/foo.j2

  notify:

    - blippy

- name: demo that parameterized roles work

  shell: echo just FYI, param1={{ param1 }}, param2 ={{ param2 }}

[GW]

Hi,

What about the following (untested) approach of default values that is implemented as a hack, but could be part of the core:

# playbook.yml

---
- hosts: all
  roles:

    - role: foo
      tmpl: /alternative/templates/foo.j2
      var1: overriden value

# foo/tasks/main.yml
---
- name: template operation
  template: src={{ item }} dest=/etc/foo
  with_first_found:
    - "{{ tmpl }}"
    - default.j2

- name: set default value with a shell hack
  shell: test -n "{{ var1 }}" && echo "{{ var1 }}" || echo "default value"
  register: used_var1

- name: some other operation
  other: param={{ used_var1 }}

With this approaches everything about which default values should be overriden is still present in the playbook, not in inventory files. The template operation is obvious.

Instead of the hack to get the right thing into "used_var1" a "with_first_defined" could be defined. It would behave like "with_first_found" (first_available_file)
but instead of looking for files existance it checks if the value is fully defined/resolved and does not contain any unresolved variables (does not contain "{{" or "$\w+").

The above example with something like this:

- name: some other operation
  other: param={{ item }}
  with_first_defined:
    - "{{ var1 }}"
    - default value 

Thoughts?

Greetings,
   gw

[GW]

If a variable wants to be used throughout a role, then its default values and overrides could be stored back into a variable like this (although a better alternative to debug module would be nice):

    - debug: msg="{{ item }}"
      register: var1
      with_first_found:
        - "{{ var1 }}"
        - foo

    - name: access the variable whose default value can be overriden
      shell: echo {{ var1.results[0].msg }}

Greetings,
   gw

[Michael DeHaan]

I'm sorry, this doesn't really make sense, it is rather specific and would spam lots of data, and add lots of undue code.

Roles could contain variables that reference variables not yet defined -- Ansible allows this, so it would not be perhaps as useful as you may think.

Some folks may have 5 simple scalar values in a role, others may have a four page nested hash table.

On Tue, Apr 23, 2013 at 7:34 PM, Jaime Gago <gago...@gmail.com> wrote:

I've been refactoring my app playbook over the week end and the roles are great, one thing that could be nice is to have the an option to have detailed roles logging so I'd know where variables and tasks are coming from just by looking at stout. 

I'm thinking something like adding "--roles-logging" to a playbook run to get

PLAY [hosts] ****

Roles: db_server, common

Roles details:

db_server

vars:foobar,

files:

-

TASK: [making sure mysqld is up] ***

Role: db_servers

--

This as remote as it gets from being a critical feature but it'd be a nice addition =)

My 2cts to the roles discussion

J.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.