Aggregating role facts

[Stephen Gargan]

I've a pattern that I want to apply and I'm wondering what the best practice might be for this. I have a situation where I apply a number of roles to a host and each have a number of facts or variables that I'd like to use in a final role, typically to generate a unified config for the host.

Say I'm applying a number of roles say a, b, c that are all distinct, but each exposes a set of metrics in a uniform way. As each gets played, I'd like to add facts about the metrics for the role in such a way that I can iterate over them in a final role d to generate some aggregate config to expose all the metrics. 

I have a number of situations that I'd like to apply this pattern so if there is a standard way to do this I'd love to hear it.

If not, I have been mulling over extending the set_fact module to allow something like the following 

Say each role could add a dictionary of values to an aggregate collection, aggregated_facts, so that in d I could do

{% for addedFactsForRole in aggregated_facts %}

// processing facts for key addedFactsForRole.key

{% for addedFact in addedFactsForRole.facts %}

// process addedFact.name

// process addedFact.value

{% endfor %}

{% endfor %}

I have used group variables to do something similar, but this tends to get crufty as it means replicating the data in every site that I use the role and I like to be able to have very generic roles that I can reuse in different sites. If the list could dynamically built like this it would be arguably more cohesive as now I could define all of the data for the role, within the role, and still have the ability to override it at the site level with group vars.

I'm thinking the task definition might look something like this.

  name: Add metrics to metrics aggregate 

  add_fact: group=host_metrics key=$role_key fact=$item

  with_items: $list_of_metrics

In the case above the list_of_metrics could be a list of single entries or a list of json blobs. I don't want to go re-inventing the wheel, this seems like a common enough scenario so I'm very curious how folks tackle it, before I go about putting this together.

Thanks in advance,

Steve.

[Michael DeHaan]

Not sure what a role fact is.   Role variables I'd assume?

"Say I'm applying a number of roles say a, b, c that are all distinct, but each exposes a set of metrics in a uniform way."

This seems very very abstract.   What is a non-distinct role?   What is a metric?

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Michael DeHaan <mic...@ansibleworks.com>
CTO, AnsibleWorks, Inc.
http://www.ansibleworks.com/

[Stephen Gargan]

> Not sure what a role fact is.   Role variables I'd assume?

Yes they are essentially variables. I just want to be able to aggregate them under a common variable name as each role is executed so that in final role execution I can use all the aggregated variables in a template.

"Say I'm applying a number of roles say a, b, c that are all distinct, but each exposes a set of metrics in a uniform way."

This seems very very abstract.   What is a non-distinct role?   What is a metric?

What I meant is that they are all different e.g. if they were java, A could be tomcat, B could be jboss, C could be a custom JVM process, but they all would expose 'metrics' or other data uniformly using JMX. My role D would take the aggregated 'metrics' specified by A, B, & C and create a config (in this case for jmxtrans) to forward them to graphite.

A better example might be ports, A exposes 8080, B 9090, C 1234. They would add these ports to a an aggregate variable "exposed_ports" that a firewall role D would use to create an aggregate firewall config. These exposed ports would be useful as 'facts' for the host as it is something that other hosts might interested in. 

I know at the site level I can make a variable 

exposed_ports:

- 8080

- 9090

- 1234

But i'd prefer to have the role add their ports dynamically as each role is applied so that If a role doesn't get played its ports don't get exposed.

 

I've put together an add_fact module similar to what I described above. I'm just tiding up the templating and will post a fuller example that illustrates what I'm thinking.

thanks,

Steve.

--
You received this message because you are subscribed to a topic in the Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-project/ObxLwj_egEI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ansible-proje...@googlegroups.com.

[Stephen Gargan]

Michael,

I've put together an implementation for an add_facts command a couple of examples that show what it might be useful for here https://github.com/sgargan/ansible-add-facts-examples. I think it explains the intention of the command a little better and gives a concrete example of the pattern I was attempting to explain above. I'm interested to hear what your thoughts are.

thanks,

Steve.

[Ned McClain]

Stephen,

I also desperately need a set_facts that works with with_items.  Have you had any response to these excellent use-case examples?

Ned.

[Michael DeHaan]

So that's a very old thread you've replied to, can you paste an example of what you are wishing to do?

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.

To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/d6bd87d4-d208-4890-9ff1-e17d81778e6f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ned McClain]

Definitely - I desire to have different roles append firewall rules (strings) to a single fact dictionary.  The final role will apply those generic rules to the appropriate iptables/ufw/ec2 security group based on the host's OS and cloud provider.  Allowing each role to append their own rules to the fact is a highly compostable solution.  It abstracts role-provided firewall rules from the host OS and hosting environments those roles are applied to.

I stumbled across Steven's super-detailed example that described the use case(s) very clearly:

https://github.com/sgargan/ansible-add-facts-examples

To me, the current outcome of using "set_fact" along with "with_items" very much defies the "Principle of least surprise".  I was sure surprised.

Note this request is in NO WAY for new "programming language" functionality in Ansible - this is simply a request to make "set_fact" work in the least surprising way when used with "with_items".

Thanks for considering this,

Ned.

[Stephen Gargan]

Ned,

There are better patterns for sharing data that what I'd originally suggested. As your system grows this approach gets untenable. If you need to share data variable files are a far more flexible solution. 

As Michael says, if you post another thread explaining of what you're trying to we'll gladly help you out.

Thanks,

Steve.

--
You received this message because you are subscribed to a topic in the Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-project/ObxLwj_egEI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ansible-proje...@googlegroups.com.

To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgxadOn-kkLwu6Fsr3SbER6bPAHeaeQgvqUsy9QygA3M2A%40mail.gmail.com.

[Michael DeHaan]

I'm open to making set_fact work with with_items (pull request welcome).

not having tested it recently - I'm actually surprised it didn't :)

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAPe9%3DrDZnoOL-UeD7ejcBY22uw%2BqJRpZkGzxAWJCsZUe_KygrQ%40mail.gmail.com.

[Ned McClain]

Michael, Stephen,

I appreciate you taking the time to respond to this old thread, and for your thoughts, and for all your hard work on Ansible.

I will work on a PR for this.

Ned.

[Stephen Gargan]

Can't take any credit, I'm just a very content user is all. Good luck with your PR.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/a764ab4a-7884-4a7b-845b-95218aa85481%40googlegroups.com.