Writing/Augmenting Host Variables from a Playbook.

[Marc Haber]

Let's assume I have a service that can listen on different combinations
or host/port:

---
host:
service:
listen:
- ip: "2001:db8:1:2::1"
port: "80"
- ip: "2001:db8:1:2::1"
port: "443"
- ip: "192.0.1.2"
port: "80"

let's also assume that the vast majority of cases use a rather simple
setup so that it would be tedious to write the standard configuration
over and over again, so it would be nice if one could just write:

---
host:
service:
hostname: "some-dns-name.example"
ports:
- "80"
- "443"

I would then like to have some component of my ansible setup to go to
DNS, look up A and AAAA records for the DNS name and generate the
detailed listen configuration.

Given some-dns-name.example would have an A record of 192.0.1.40 and an
AAAA record of 2001:db8:1::40, the data structure built would be

---
host:
service:
listen:
- ip: "2001:db8:1::40"
port: "80"
- ip: "2001:db8:1::40"
port: "443"
- ip: "192.0.1.40"
port: "80"
- ip: "192.0.1.40"
port: "443"

so that the tasks and templates could always act as if the fully
detailed configuration were explicitly given, while giving the admin the
possibility to always write doen the fully detailed configuration
explicitly AND the ease of writing down the easier form for the standard
cases.

I could have a program read in the inventory, write out a temporary
inventory with the "augmented" host variables and then have ansible run
from this, but I hope that this won't be necessary...

Greetings
Marc


--
-----------------------------------------------------------------------------
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421

[Stefan Hornburg (Racke)]

On 4/27/20 10:12 AM, Marc Haber wrote:
> Let's assume I have a service that can listen on different combinations
> or host/port:
>

Hello Marc,

nice to see you here :-). This is certainly doable:

---
- hosts: all

vars:
host:
service:
hostname: "google.com"

ports:
- "80"
- "443"

tasks:
- name: Get DNS records
set_fact:
ip_addresses: "{{ ip_addresses | default([]) + lookup('dig', host.service.hostname + '/' + item, wantlist=True) }}"
loop:
- A
- AAAA
when: "'listen' not in host.service"

- name: Determine all combinations of IP address and ports
set_fact:
ips_ports: "{{ ip_addresses | product(host.service.ports) | list }}"
when: "'listen' not in host.service"

- name: Turn list members into dictionaries
set_fact:
host:
service:
listen: "{{ host.service.listen | default([]) + [{ 'ip': item[0], 'port': item[1] }] }}"
loop: "{{ ips_ports }}"
when: ips_ports is defined

- debug:
msg: "{{ host }}"

Give it a try and let me know if it works. Or if you need help to understand how it works :-).

Regards
Racke

Ecommerce and Linux consulting + Perl and web application programming.
Debian and Sympa administration. Provisioning with Ansible.

[Marc Haber]

Hi Stefan,

thanks for your answer. I busied myself with working around the issue
temporarily because I had to get a project forward.

On Tue, Apr 28, 2020 at 09:48:23AM +0200, Stefan Hornburg (Racke) wrote:
> On 4/27/20 10:12 AM, Marc Haber wrote:
> > Let's assume I have a service that can listen on different combinations
> > or host/port:
> >
>
> Hello Marc,
>
> nice to see you here :-). This is certainly doable:
>
> ---
> - hosts: all
>
> vars:
> host:
> service:
> hostname: "google.com"
> ports:
> - "80"
> - "443"
> tasks:
> - name: Get DNS records
> set_fact:
> ip_addresses: "{{ ip_addresses | default([]) + lookup('dig', host.service.hostname + '/' + item, wantlist=True) }}"
> loop:
> - A
> - AAAA
> when: "'listen' not in host.service"

This sets a local fact to the list of IP addreses pulled from DNS.
Adding the IPv6 addresses is an easy enogh exercise.

> - name: Determine all combinations of IP address and ports
> set_fact:
> ips_ports: "{{ ip_addresses | product(host.service.ports) | list }}"
> when: "'listen' not in host.service"

That's magic. Cute, indeed. This returns a list, alternating between IP
adress and port, like [ ip1, port1, ip2, port2, ip3, port3 ]

> - name: Turn list members into dictionaries
> set_fact:
> host:
> service:
> listen: "{{ host.service.listen | default([]) + [{ 'ip': item[0], 'port': item[1] }] }}"
> loop: "{{ ips_ports }}"
> when: ips_ports is defined

This is where you're losing me. You're iterating over an array and get
array members, so item will be "ip1" in the first iteration, "port1" in
the second, "ip2" in the third. How can you index into an array
element???

Anyway, things have become a bit more complicated since my original
request. This is what I have now:

service:
sites:
default:
protocols:
- "http"
- "https"
listen:
- ip: "2a01:4f8:161:3541::32:100"
port: "80"
protocol: "http"
- ip: "2a01:4f8:161:3541::32:100"
port: "443"
protocol: "https"

The templates pull site configuration from the "sites" part of the
definition; the listen configuration from the "listen" part of the
definition. This is a strongly simplified example with reality being
much more complex, I just reduced this to the relevant parts. This is,
however, from a working setup (and yes, it's IPv6 only).

For better readability, I'd like to write this like:

service:
sites:
default:
protocols:
- "http"
- "https"
listen:
- ip: "2a01:4f8:161:3541::32:100"

With the possibility of defining explicit ports and protocols in the
listen clause. I think to massage one format into the other one in a
task is awfully hard, and to use local facts makes it impossible to
override the automatism by explicitly writing the desired result to
inventory proper.

Can a custom inventory plugin access what the previously running
inventory plugins have parsed, and can it augment structure that was
build by the predecessors? This way, I could have all processing power
and flexibility of imperative programming. I could think of a gazillion
of other places where this could be useful to simplify my templates
_AND_ my inventory.

Having this done inside ansible would allow me to take advantage of the
inventory reading logic that is already present in ansible. I could
write a preprocessor writing out the "augmented inventory" before
ansible is started, but I'd have to manually process the inventory file
-and- the contents of the host_vars and group_vars directories. I'd like
to avoid this.

[Stefan Hornburg (Racke)]

It already contains the IPv6 addresses (AAAA record).

>
>> - name: Determine all combinations of IP address and ports
>> set_fact:
>> ips_ports: "{{ ip_addresses | product(host.service.ports) | list }}"
>> when: "'listen' not in host.service"
>
> That's magic. Cute, indeed. This returns a list, alternating between IP
> adress and port, like [ ip1, port1, ip2, port2, ip3, port3 ]

It contains all IP and port combinations as nested list:

[[ip1,port1],[ip1,port2],[ip2,port1]]

>
>> - name: Turn list members into dictionaries
>> set_fact:
>> host:
>> service:
>> listen: "{{ host.service.listen | default([]) + [{ 'ip': item[0], 'port': item[1] }] }}"
>> loop: "{{ ips_ports }}"
>> when: ips_ports is defined
>
> This is where you're losing me. You're iterating over an array and get
> array members, so item will be "ip1" in the first iteration, "port1" in
> the second, "ip2" in the third. How can you index into an array
> element???
>

See above, each entry is a list of IP (item[0]) and port (item[1]).

Regards
Racke

[Marc Haber]

Idiot me, of course, I missed the loop.

> >> - name: Determine all combinations of IP address and ports
> >> set_fact:
> >> ips_ports: "{{ ip_addresses | product(host.service.ports) | list }}"
> >> when: "'listen' not in host.service"
> >
> > That's magic. Cute, indeed. This returns a list, alternating between IP
> > adress and port, like [ ip1, port1, ip2, port2, ip3, port3 ]
>
> It contains all IP and port combinations as nested list:
>
> [[ip1,port1],[ip1,port2],[ip2,port1]]

Now it all makes sense to me. Thanks for explaining. I'm still wondering
whether it would be possible for a more complex of variable data
definition.

> > Can a custom inventory plugin access what the previously running
> > inventory plugins have parsed, and can it augment structure that was
> > build by the predecessors? This way, I could have all processing power
> > and flexibility of imperative programming. I could think of a gazillion
> > of other places where this could be useful to simplify my templates
> > _AND_ my inventory.
> >
> > Having this done inside ansible would allow me to take advantage of the
> > inventory reading logic that is already present in ansible. I could
> > write a preprocessor writing out the "augmented inventory" before
> > ansible is started, but I'd have to manually process the inventory file
> > -and- the contents of the host_vars and group_vars directories. I'd like
> > to avoid this.

Any idea whether this would work?

[Stefan Hornburg (Racke)]

It is possible, but the Jinja expressions may become really complex
which you might want to prevent.

>>> Can a custom inventory plugin access what the previously running
>>> inventory plugins have parsed, and can it augment structure that was
>>> build by the predecessors? This way, I could have all processing power
>>> and flexibility of imperative programming. I could think of a gazillion
>>> of other places where this could be useful to simplify my templates
>>> _AND_ my inventory.
>>>
>>> Having this done inside ansible would allow me to take advantage of the
>>> inventory reading logic that is already present in ansible. I could
>>> write a preprocessor writing out the "augmented inventory" before
>>> ansible is started, but I'd have to manually process the inventory file
>>> -and- the contents of the host_vars and group_vars directories. I'd like
>>> to avoid this.
>
> Any idea whether this would work?

You could also write a custom module in Python which can transform the structure
as you wish. You can call your custom module in the first task to achieve the
augmenting.

Regards
Racke

[Marc Haber]

_That_ sounds totally interesting, can you point me to some example code
please?

[Stefan Hornburg (Racke)]

On 5/9/20 8:47 AM, Marc Haber wrote:
> On Sat, May 09, 2020 at 07:49:46AM +0200, Stefan Hornburg (Racke) wrote:
>> On 5/9/20 7:36 AM, Marc Haber wrote:
>>>>> Can a custom inventory plugin access what the previously running
>>>>> inventory plugins have parsed, and can it augment structure that was
>>>>> build by the predecessors? This way, I could have all processing power
>>>>> and flexibility of imperative programming. I could think of a gazillion
>>>>> of other places where this could be useful to simplify my templates
>>>>> _AND_ my inventory.
>>>>>
>>>>> Having this done inside ansible would allow me to take advantage of the
>>>>> inventory reading logic that is already present in ansible. I could
>>>>> write a preprocessor writing out the "augmented inventory" before
>>>>> ansible is started, but I'd have to manually process the inventory file
>>>>> -and- the contents of the host_vars and group_vars directories. I'd like
>>>>> to avoid this.
>>>
>>> Any idea whether this would work?
>>
>> You could also write a custom module in Python which can transform the structure
>> as you wish. You can call your custom module in the first task to achieve the
>> augmenting.
>
> _That_ sounds totally interesting, can you point me to some example code
> please?
>

I don't have a custom module around which I could share here. Searching the web
for "ansible custom module" should give you plenty of insights though.

Regards
Racke

[Marc Haber]

On Sat, May 09, 2020 at 08:53:37AM +0200, Stefan Hornburg (Racke) wrote:
> I don't have a custom module around which I could share here. Searching the web
> for "ansible custom module" should give you plenty of insights though.

Are you trying to say that a plain custom module called from a task (a)
has access to the full inventory including all variables and (b) can
write to it with following tasks seeing the changes it did?

[Stefan Hornburg (Racke)]

On 5/9/20 9:09 AM, Marc Haber wrote:
> On Sat, May 09, 2020 at 08:53:37AM +0200, Stefan Hornburg (Racke) wrote:
>> I don't have a custom module around which I could share here. Searching the web
>> for "ansible custom module" should give you plenty of insights though.
>
> Are you trying to say that a plain custom module called from a task (a)
> has access to the full inventory including all variables and (b) can
> write to it with following tasks seeing the changes it did?
>
> Greetings
> Marc
>

That's a good question :-). But you can also write filter plugins which is probably a better
idea.

This would allow you do augment your data structure with

"{{ data | augment }}"

You can find an example here: https://blog.oddbit.com/post/2019-04-25-writing-ansible-filter-plugins/

[Marc Haber]

On Sat, May 09, 2020 at 09:21:34AM +0200, Stefan Hornburg (Racke) wrote:
> On 5/9/20 9:09 AM, Marc Haber wrote:
> > On Sat, May 09, 2020 at 08:53:37AM +0200, Stefan Hornburg (Racke) wrote:
> >> I don't have a custom module around which I could share here. Searching the web
> >> for "ansible custom module" should give you plenty of insights though.
> >
> > Are you trying to say that a plain custom module called from a task (a)
> > has access to the full inventory including all variables and (b) can
> > write to it with following tasks seeing the changes it did?
>

> That's a good question :-). But you can also write filter plugins which is probably a better
> idea.
>
> This would allow you do augment your data structure with
>
> "{{ data | augment }}"
>
> You can find an example here: https://blog.oddbit.com/post/2019-04-25-writing-ansible-filter-plugins/

I am not sure whether this is a better idea, making the augmentation
dependent to the actual task being in quesiton. An independent approach
would, for example, allow people to use generic apache, letsencrypt,
proxy and DNS modules from The Galaxy while just writing the URL and
other web site data in the host definition, with independent code taking
the job of converting the simple host definition into input data
structures the generic modules can grok.