Accessing Previously Set Facts in a Module

[Julian Berman]

I apologize for asking this because I see a number of previous threads, but none of them hit exactly this point, or the answers don't appear to have worked for me yet so I'm trying to figure out if I'm just completely missing something.

I have state that I need to persist between invocations of my module.

I'm following http://docs.ansible.com/developing_modules.html#module-provided-facts, but it appears to be pretty scarce (I know documentation is hard, so I'd love to contribute back the answer to the question I'm about to ask, since it'd probably fit there quite reasonably).

I have a list of strs which my module should read, possibly modify, and then store back.

The pattern for this, from various sequences of grepping around the example modules (which don't seem to do this), reading the PlayBook and Runner source, and just jumping back and forth, seems like it should be something like:

    ...

    facts = ansible_facts(module)

    existing_services = facts.get("services", {})

    new_services = do_stuff_with(services)

    module.exit_json(..., ansible_facts={"services" : new_services}

But after the module runs once, nothing new appears in the facts (specifically, there's no new "services" key, and no other new keys, and even `assert "somethingIknowshouldbethere" in str(facts)` fails so it doesn't appear to get added to some subobject in there.).

Where should my keys be going?

https://github.com/ansible/ansible/blob/devel/lib/ansible/playbook/__init__.py#L519 makes it seem like the answer is "top-level", but yeah, I don't see them, so I must be missing something.

If my example isn't concrete enough and in fact it looks correct, here's my exact module: (preface: my slightly embarrassing internal comment is now public due to me pushing that to get it working -- apologies, it's been a slightly long day of frustration): https://github.com/Magnetic/nix-ansible/blob/3d37bceefe7b022ee0cd9ad31e6c3b773e6b4fe4/nix.py

As a tangent, maybe this function (get_all_facts) isn't supposed to be public, I don't see any documentation for it, and specifically about what kind of object it wants as an argument, but https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/facts.py#L2716-L2717 seems slightly strange, it seems to be expecting a module with a "filter" param specifically. Does that "# ======" section break mean "from here on down is private" or something, or why is it making that assumption? (I see that that code appears to be copy-pasted from the `setup` module, so maybe the answer is "it shouldn't, and it was just copied over from there, which does have a filter parameter?")

Appreciated,

-Julian

[Jason DeTiberus]

On 08/04/15 15:52 -0700, Julian Berman wrote:
>I apologize for asking this because I see a number of previous threads, but
>none of them hit exactly this point, or the answers don't appear to have
>worked for me yet so I'm trying to figure out if I'm just completely
>missing something.
>
>I have state that I need to persist between invocations of my module.
>
>I'm
>following http://docs.ansible.com/developing_modules.html#module-provided-facts,
>but it appears to be pretty scarce (I know documentation is hard, so I'd
>love to contribute back the answer to the question I'm about to ask, since
>it'd probably fit there quite reasonably).
>
>I have a list of strs which my module should read, possibly modify, and
>then store back.
>
>The pattern for this, from various sequences of grepping around the example
>modules (which don't seem to do this), reading the PlayBook and Runner
>source, and just jumping back and forth, seems like it should be something
>like:
>
>
> ...
> facts = ansible_facts(module)

^ This will retrieve the base ansible facts, so unless you need to consume
them, you wouldn't want to do this (also, you'd need `from
ansible.module_utils.facts import *` for this to work as well)

> existing_services = facts.get("services", {})

^ This just returns an empty hash since services does not exist in the dict
returned fro ansible_facts above.

> new_services = do_stuff_with(services)
> module.exit_json(..., ansible_facts={"services" : new_services}

^ When specifying a dictionary this way, the quotes should be left off of
services. I suspect you are losing the dictionary you are passing back because
of the json conversion that exit_json is doing. At this point, the
ansible_facts variable should be set to a python dictionary.

>
>But after the module runs once, nothing new appears in the facts
>(specifically, there's no new "services" key, and no other new keys, and
>even `assert "somethingIknowshouldbethere" in str(facts)` fails so it
>doesn't appear to get added to some subobject in there.).

You will not see the services value from within your module, you are
generating it within your module. The approach I took for returning facts and
persisting data between runs was to use a combination of the module provided
facts and local facts:
http://docs.ansible.com/playbooks_variables.html#local-facts-facts-d

You can see how I'm doing it here:
https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_facts/library/openshift_facts.py
but basically, I'm storing data that I want to persist in
/etc/ansible/openshift.fact and then using that file to determine what "facts"
to return based on the local facts, base defaults, and provider specified
defaults.

>
>Where should my keys be going?
>
>https://github.com/ansible/ansible/blob/devel/lib/ansible/playbook/__init__.py#L519
>makes it seem like the answer is "top-level", but yeah, I don't see them,
>so I must be missing something.
>
>If my example isn't concrete enough and in fact it looks correct, here's my
>exact module: (preface: my slightly embarrassing internal comment is now
>public due to me pushing that to get it working -- apologies, it's been a
>slightly long day of frustration):
>https://github.com/Magnetic/nix-ansible/blob/3d37bceefe7b022ee0cd9ad31e6c3b773e6b4fe4/nix.py

Changing line 71 to: 'ansible_facts={services: nix.services},' should be all
that you need. You shouldn't need to import from ansible.module_utils.facts

Hope that helps,

Jason DeTiberus

[Julian Berman]

Hey Jason, thanks so much for the info. I'm going to have a look at your Openshift module tomorrow, so the only thing in your reply that I'm not sure I understand initially is what you mean about services and line 71 -- I'm passing a Python dictionary with one key, "services", and expecting to retrieve that later, when the module runs again a second time -- if I remove the quotes, I should get a NameError, because I have no local called `services` -- am I misunderstanding what you mean, or is there more special sauce somewhere that's doing something unexpected there?

[Jason DeTiberus]

On 08/04/15 20:33 -0700, Julian Berman wrote:
>Hey Jason, thanks so much for the info. I'm going to have a look at your
>Openshift module tomorrow, so the only thing in your reply that I'm not
>sure I understand initially is what you mean about services and line 71 --
>I'm passing a Python dictionary with one key, "services", and expecting to
>retrieve that later, when the module runs again a second time -- if I
>remove the quotes, I should get a NameError, because I have no local called
>`services` -- am I misunderstanding what you mean, or is there more special
>sauce somewhere that's doing something unexpected there?

The main thing is that any "facts" that you return in the ansible_facts
dictionary is only going to be available after your module is run, it won't be
persisted in any way by default. Your module will have to provide a way to
persist the data between runs. In my case, I used the local_facts approach
for storing the data so that it can also be retrieved through local_facts when
ansible is run against the host even if my openshift_facts module has not been
run.

Don't mind me on the syntax bit, I don't use a single language with enough
regularity to keep all the syntax clear in my head, I was thinking of the way
you would build a python dictionary using dict() instead of {}.

Hope that clarifies things a bit.
--
Jason

>--
>You received this message because you are subscribed to the Google Groups "Ansible Development" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to ansible-deve...@googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.

[Julian Berman]

Hah understandable.

What specifically is the lifecycle of facts then if they're not persisted? When you say "between runs" -- what I'm looking for is for 

`ansible-playbook foo.yaml`

where foo.yaml contains

- name: first step

  mymodule: service=foo

- name: second step

  mymodule: service=bar

and then goes off and runs some other task file as well, with service=baz,

that by the time I see service=baz, I should have a list of all three, "foo", "bar", and "baz" -- that's all the persistence I need, and I want it to disappear once ansible-playbook exits. Is even that not what is persisted with facts? If so what possibly can you use them for if they disappear as soon as the module exits? Might as well just use Python objects, which at least will persist until the process exits?

Thanks again,

-J

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-devel+unsubscribe@googlegroups.com.

[Jason DeTiberus]

On 09/04/15 10:03 -0400, Julian Berman wrote:
>Hah understandable.
>
>What specifically is the lifecycle of facts then if they're not persisted?
>When you say "between runs" -- what I'm looking for is for
>
>`ansible-playbook foo.yaml`
>
>where foo.yaml contains
>
>- name: first step
> mymodule: service=foo
>
>- name: second step
> mymodule: service=bar
>
>and then goes off and runs some other task file as well, with service=baz,
>
>that by the time I see service=baz, I should have a list of all three,
>"foo", "bar", and "baz" -- that's all the persistence I need, and I want it
>to disappear once ansible-playbook exits. Is even that not what is
>persisted with facts? If so what possibly can you use them for if they
>disappear as soon as the module exits? Might as well just use Python
>objects, which at least will persist until the process exits?

Okay, so the facts will be available to the rest of the playbook (and any
included roles, playbooks, tasks, etc after the module has been called). I'm
not sure how you would access the runtime facts from within the module, or if
it is even possible. If you try to retrieve the facts using
ansible.module_utils.facts, you will get the ansible generated facts only.

The main purpose of facts are to make decisions later within the playbook.
The current modules that aggregate actions (yum, apt, etc) are currently done
by workarounds in the ansible code and not natively in the module. I *believe*
that there are plans for v2 to make this more generic, but someone from
Ansible would have to confirm/deny that statement.

In my case, I need to persist the data between playbook runs, not just
multiple module runs within a playbook. You might have to require a parameter
to our module for the user to provide the output of
hostvars[inventory_hostname] to consume the current facts as a workaround.

--
Jason

>>> email to ansible-deve...@googlegroups.com.

[Brian Coca]

modules don't get data passed to them automatically, so if you add a
fact, you need to explicitly pas it to the module

about with_items optimization, this has been on our wishlist for a long time now


--
Brian Coca

[Julian Berman]

Ah cool, thanks for the confirmation. And I'd need to persist the fact myself right? Or how would I be able to pass it to the module, and whatever way that is, why does it need to involve ansible at all at that point? Might as well just pass Python objects around outside of ansible's framework (and AnsibleModule)?

To me I'd have expected ansible to pass me a Python object that it passes to each module during invocation where I can hang state, and then it disappears at the end of the run. Is that the kind of thing that's on the wishlist?

[Hank Beatty]

Hello Julian,

Did you ever come up with a solution to this? I'm wanting to do the exact same thing.

Thanks,
Hank

[Julian Berman]

I haven't yet, but I took a break to work on a few other things, so I'll be coming back to it next week I hope.

Ansible doesn't make persisting to a file trivial yet either from what I can tell, since you don't get access to `ansible_remote_tmp` (or I haven't found it yet), so I don't yet see a way to be able to even store state in a file during a run in a way that gets cleaned up at the end.

But if / when I come up with a solution I'll post it here.