variable scope and roles

[Yves Dorfsman]

How is variable scope supposed to work with roles?

I have the same variable defined in different roles (say install_dir), when
running a playbook which includes several roles, the value assumed during the
execution of the tasks is always the one from the last defined role (the one
at the bottom of the list). I expected it to take its value from the
vars/main.yml for the role the task is defined in.

Is this how it is supposed to work, or a bug?

Thanks.

--
Yves. http://www.SollerS.ca/
Unix/Linux and Python specialist in Calgary.
http://blog.zioup.org/

[Michael DeHaan]

Roles pull in all variables from all vars directories before running any tasks in them.

It is not a bug.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Michael DeHaan <mic...@ansibleworks.com>
CTO, AnsibleWorks, Inc.
http://www.ansibleworks.com/

[John Jarvis]

I could see this causing behavior that may be difficult to troubleshoot.  

We have multiple people writing roles right now and I could see cases where it might be easy to stomp on each other.

I think it would make sense to error immediately if: 

* there are any variables defined with the same name in group_vars/<group-name>  (all files except for group_vars/all)

* there are any variables defined with the same name roles/*/vars/*

Thoughts?

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.

[Michael DeHaan]

You can configure variable merging in the config file if you want for hash values.

We clearly document what this does in the documentation -- this is an automation around tasks/handlers/vars_files.

[John Jarvis]

Yeah we use hash merging for precedence orders that are well defined such as the order of var_files or group_vars/all.

What I'm speaking about specifically are when two variables have the same name in group_vars/<group_name> where the precedence is not well defined (in that case the last file sorted alphabetically wins).

Another example is when two variables in two different roles have the same name in roles/<role-name>/vars/main.yml. 

I believe that should throw an error if precedence isn't well defined in that case.

[Michael DeHaan]

We are probably not going to do that.

[Kahlil Hodgson]

IIRC, variable assignment in child groups trump the assignment in the parent. Haven't tested extensively, but I believe you have to make the relationship explicit with '[groupname:children]' sections in the inventory file.

K

--

Kahlil (Kal) Hodgson                       GPG: C9A02289

Head of Technology                         (m) +61 (0) 4 2573 0382

DealMax Pty Ltd                            (w) +61 (0) 3 9008 5281

Suite 1415

401 Docklands Drive

Docklands VIC 3008 Australia

"All parts should go together without forcing.  You must remember that

the parts you are reassembling were disassembled by you.  Therefore,

if you can't get them together again, there must be a reason.  By all

means, do not use a hammer."  -- IBM maintenance manual, 1925

[Michael DeHaan]

Groups are included based on depth order.

Variables in a sub-group override those of the parent group.

On Thu, May 2, 2013 at 3:16 PM, Pol Llovet <pol.l...@gmail.com> wrote:

Yves,

I ran into the same issue with groups.  If a host is a member of multiple groups (say "physical", "database", and "datacenter1"), and each group has the same variable, it will use the last one assigned (the last group the host is listed in within the inventory file).  I think the key is that (as Michael mentioned) all variables are compiled and any collisions are all worked out in advance.  One could argue that this is easier to debug, but I couldn't find where it was explicitly defined in the documentation.  Maybe someone can point me in the right direction?

Also, IIRC, there is a trumping order (so it isn't *just* whichever is last in the files).  Like host vars trump group vars trump role vars or something like that.  I think the trumping order is just a consequence of the "last assigned" characteristic (role is populated, then group, then host or something like that).

-pol

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Michael DeHaan]

http://ansible.cc/docs/playbooks2.html#understanding-variable-precedence

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[John Jarvis]

@Pol - Yeah we ran into the same problem which may be more of an issue when you have group generation that is more dynamic (like when group definitions come from the ec2.py inventory scripts). 

In any case I think in cases where precedence isn't defined/documented the right thing to do is throw an exception but it sounds like mdehaan has his mind made up about it.

I need to be extra careful since we have different people developing roles right now to be sure we don't stomp on each other's variable namespace.

On Thu, May 2, 2013 at 3:16 PM, Pol Llovet <pol.l...@gmail.com> wrote:

Yves,

I ran into the same issue with groups.  If a host is a member of multiple groups (say "physical", "database", and "datacenter1"), and each group has the same variable, it will use the last one assigned (the last group the host is listed in within the inventory file).  I think the key is that (as Michael mentioned) all variables are compiled and any collisions are all worked out in advance.  One could argue that this is easier to debug, but I couldn't find where it was explicitly defined in the documentation.  Maybe someone can point me in the right direction?

Also, IIRC, there is a trumping order (so it isn't *just* whichever is last in the files).  Like host vars trump group vars trump role vars or something like that.  I think the trumping order is just a consequence of the "last assigned" characteristic (role is populated, then group, then host or something like that).

-pol

On Thursday, May 2, 2013 9:32:11 AM UTC-6, Yves Dorfsman wrote:

--

[Kahlil Hodgson]

Hi John,

On Fri, May 3, 2013 at 11:45 AM, John Jarvis <jo...@jarv.org> wrote:

I need to be extra careful since we have different people developing roles right now to be sure we don't stomp on each other's variable namespace.

Just exploring this at the moment, but I was thinking of making it a policy to prefix variables used in a role with the role name when you wanted to make sure they only applied to a role.  Enforcing by convention, but allowing explicit overrides when it made sense.  Is this the kind of thing you are doing?

[John Jarvis]

Yeah I think we should probably adopt that convention.

We also have an interesting challenge in that we are trying to keep our configuration open-source which means overrides for sensitive bits.

group_vars/all <-- variables that apply to all roles and sets a default/dummy $secure_dir path 

group_vars/<group_name>  <-- overides $secure_dir which will point to a place on disk that's not publically acceptable, may be different for different deploy groups.

role/*/vars <-- default variable definitions for roles

And then our playbooks look like this:

var_files:

  - {{ secure_dir }}/path/to/overrides.yml  <-- this overrides any of the default variables that are in role/*/vars 

[Michael DeHaan]

No, Ansible as originally designed (and still is) strives to be a minimal system.

This means as little as possible to remember in terms of conventions about things being named or labelled.

If you wish to implement namespacing by putting your variable down in a hash based on the role name, you are welcome and able to do so, which is also by design.

On Thu, May 2, 2013 at 9:57 PM, Kahlil Hodgson <kahlil....@dealmax.com.au> wrote:

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Michael DeHaan]

BTW, the no was to "we are not going to force this".

But really, {{ apache.1234 }} is pretty easy:

from roles/apache/vars/main.yml:

apache:

    port: 1234

much better than prefixing.

[Michael DeHaan]

Grr, I'm posting too late, sorry for spam.

I obviously mean {{ apache.port }}

[Kahlil Hodgson]

Ah, yes. That is a tidier way of namespacing :-)

[John Jarvis]

> This means as little as possible to remember in terms of conventions about things being named or labelled.

To play the devils advocate here having to make sure variable names are unique between roles to avoid unintended collisions where precedence isn't defined is something that adds to the things I need to remember.

Using a convention where my vars go into hashes that have the same name as the role sounds reasonable (as does a role prefix). 

[Michael DeHaan]

You must understand it takes huge acts balance to make something that pleases everyone.

As such, we are going to be very conservative when changing existing behaviors that are in place for thousands of users.

Variable overriding by group priority, hosts over groups, and the various default mechanisms in Ansible are important to everyone.

If there were theoretically some system that warned when the same variable were used in multiple groups at the same level, which is to say which is the right one and which is the warning?  And then, it's not an error, it's a warning, so how to you convey that

if you have 10000 hosts and it only clashes for some.

Basically the machinery for this would be obsenely complex.

I'll say this -- submit a clean patch and I'll think about it.  But it better be very very very clean and break nothing else, and still work with variable merging behavior on and off, and still work with group precedence, and not change anything with the precedence order, nor change anything to cause false alarms.   Many of these things are features where overriding is useful, you have identified *ONE* case where overriding may be bad... but might not be.   Some folks might actually rely on the variables of two overriding one.   So do you make the warning itself configurable?   I would hope you don't, as quickly the config file gets rather bloated quickly.

It's harder than you think.

So yes, we're conservative about making changes.

I would much rather just show in best practice type convention that if you want to define some variable "port" it is a good idea to namespace it somehow relative to the application.

--Michael

[Kahlil Hodgson]

Nice to have that as a 'convention' or 'best practice', but I'd not want to enforce it in any way.
Those who 'know what they are doing' should still have the freedom to do otherwise. 

[Yves Dorfsman]

On 2013-05-02 20:34, Michael DeHaan wrote:
> BTW, the no was to "we are not going to force this".
>
> But really, {{ apache.1234 }} is pretty easy:
>
> from roles/apache/vars/main.yml:
>
> apache:
> port: 1234
>

Which is exactly what I have started to do. Another surprise for me:

website:
port: 8080
host: example.com
path: /api
url: http://${host}:${port}/${path} <=== does not work
url: http://${website.host}:${website.port}/${website.path} <=== works

I really expected the first form to work.


> much better than prefixing.

I don't know... Because the tasks, templates and handlers are all local in
scope (You can't import a task from another scope without using full path), I
really expected vars to be local too. To me it seems more natural.

Another way I'm coping with this, is by adding the variables in my
tasks/imports. This was suggested in a different thread on this mailing list
months ago. I think we should add something to the documentation about this
(an example) as it isn't obvious, especially for a newcomer:

- import: /opt/shared/some_general_install_task
vars:
install_dir: /usr/local/myproduct

[Michael DeHaan]

Hi guys,

It's the way it's going to work.   Let's let it go :)  

--Michael

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[John Jarvis]

Update on this older thread since we recently ran into some gotchas with using hashes as a way to isolate variable namespace for roles.

For example if every role has a variable named "code_base_dir" you might name it "<role name>_code_base_dir" or put "code_base_dir" in a hash named <role_name>.

The latter option is tidier but it caused some problems so we are back to using underscores. 

For example this didn't seem to work for role "foo" and is probably not supposed to work:

foo:

  code_base_dir = /path/to/foo

  some_file = "{{ foo.code_base_dir }}/somefile"

Where this does work and something we do often in our vars/main.yml files:

foo_code_base_dir = /path/to/foo

foo_some_file = "{{ foo_code_base_dir }}/somefile"

Another problem was that if you need to override {{ foo.some_file }} you will need to turn on the hash merging setting which is fine but isn't by default turned on.

In general we also find it easier to troubleshoot since grep'ing through the playbooks for "foo_some_file" is easier than grepping for a hash.

So for now I think our convention will need to be that every variable defined in a role will need to be prefixed with the role name.

-John

[Michael DeHaan]

foo:

  code_base_dir = /path/to/foo

  some_file = "{{ foo.code_base_dir }}/somefile"

This doesn't work because it's not valid YAML syntax.  You need colons, not equal signs.

[John Jarvis]

Well that too (bad paste), but this was not working either:

foo:

  code_base_dir: /path/to/foo

  some_file: "{{ foo.code_base_dir }}/somefile"

If you think it should work I can take a look again.

[Michael DeHaan]

Templating probably only happens at one point in time here, rather than multiple times recursively, in order to resolve that.

Old style variables DO work that way, but it was something I observed to seem unneccessary in Jinja2 in at least top level variables, since Jinja2 is all about what variables/symbols you have in that top level hash.

So I bet if you put basedir a level up it probably might work, worth a shot:

foo_basedir: X

foo:

    code_base_dir:  "{{ foo_basedir }}/xxxxx"

[John Jarvis]

Yeah, putting the variable outside the hash works fine.

Does this qualify as a bug or as-designed or is it generally bad for us to be referencing variables in variable files like this to begin with?

[Michael DeHaan]

It's not a bug, more so something you would like to work that does not work.

Old style variables will allow referencing, but in general, I think it's a sign things should be modelled simpler.

[Michael DeHaan]

It's not a bug, more so something you would like to work that does not work.

Old style variables will allow referencing, but in general, I think it's a sign things should be modelled simpler when you are trying to do something like this.

If there's a "Zen of Ansible" here, it's "don't try to overcomplicate things" :)

[John Jarvis]

Yeah the ultimate goal is to definitely make things less complicated by coming up with scheme to organize variables namespace in a sane way.

I think the simplest approach will be for us to have a general rule that for roles you should prefix variable names with the role name.

> Old style variables will allow referencing,

The new style does as long as they aren't in hashes:

https://github.com/edx/configuration/blob/master/playbooks/roles/xqueue/vars/main.yml

I definitely want to know if this qualifies as something that you think is too complicated.

-John

[Michael DeHaan]

Looks pretty reasonable to me!

[Bowe Strickland]

Coming late to this discussion, also i'm clumsy with this forum interface, so i might be double posting, but 

i just discovered that roles don't implement true namespaces: http://pastebin.com/9W378CiQ, and found this

thread by searching.

The thread seems to express similar concern, with a proposal to resolve it with a "role prefix conveintion" for variables, 

with (over simplified) arguments against being a) simplicity is key, b) we don't want too many conventions, and c) there's 

years of existing implementation that would be hard to change.

I propose an alternative:  when you enter a role, you enter a true namespace by default.

a) in the 'postgres' role, $libdir = /var/lib/pgsql/data, while in the 'snmp' role, $libdir = /usr/share/snmp.

    (it's the global variables which should be prefixed....).  you don't get much simpler than that.

    to me, the most important thing roles can provide is encapsulation.

b) it follows established programming convention, so no need for new conventions.

c) roles, to me, are still new enough that we should get them right

the obvious downside:  it's hard, and would require banging on some core ansible code.

ansible has so much right, and is important enough to me, that i'd be willing to spend some time on this... so i guess

my question at this point is, does anyone share this vision?   if coded to appropriate standards, does it have a chance 

of being accepted by the community? 

--bowe 

On Thursday, May 2, 2013 11:32:11 AM UTC-4, Yves Dorfsman wrote:

How is variable scope supposed to work with roles?

I have the same variable defined in different roles (say install_dir), when
running a playbook which includes several roles, the value assumed during the
execution of the tasks is always the one from the last defined role (the one
at the bottom of the list). I expected it to take its value from the
vars/main.yml for the role the task is defined in.

Is this how it is supposed to work, or a bug?

Thanks.

[Michael DeHaan]

I think that's a reasonable thing to do, load stuff into the parameterized role namespace and also load them into the global namespace.

I suspect this is probably about a 5-7 line patch.

It would ensure the local variable is always used locally, but could still be referenced by other roles.

--Michael

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Krzysztof Zarzycki]

Hi! 

I know it's an old topic, but would like to dig it out for newcomers. 

I've noted, that role variables in Ansible 2.0 are namespaced! 

Have a look at my test: https://gist.github.com/kzarzycki/0a82497ed72756b722c56a62ab9f0ab5

That's the behavior I'm a fan of :) 

Cheers,

Krzysztof