Complex process to generate template/lineinfile data?

[Richard Hector]

Hi all,

Well, it's not really that complex, but seems complex to do within the
constraints of ansible/jinja.

My scenario is that I run dirvish to back up multiple containers on
multiple hosts. The containers are backed up via the filesystems on the
hosts.

The root filesystem of the container lives under
/var/lib/lxc/<container>/rootfs.

Sometimes, extra filesystems are also used; they live under
/guestfs/<container>-<name>, where 'name' is a possibly shortened
version of the mountpoint.

Relevant sections of host_vars files look like this:

filesystems:
- { index: 1, suffix: srv, size: 2, container_mountpoint: '/srv/' }
backup_paths:
- { shortname: 'ROOT', path: '/' }
- { shortname: 'var', path: '/var/' }
- { shortname: 'home', path: '/home/' }
- { shortname: 'srv', path: '/srv/' }

So when I'm setting up backups, I need to treat any backup path for this
container starting with '/srv/' differently, in that I need to backup
something under /guestfs/ rather than under /var/lib/lxc.

I have a template that does this for the dirvish config:

client: {{ hostvars[container_host].fqdn }}
{% set ns = namespace(extra_fs='') %}
{% for mountpoint in filesystems %}
{% if item['path'] | regex_search(mountpoint['container_mountpoint']) %}
{% set ns.extra_fs = mountpoint %}
{% endif %}
{% endfor %}
{% if ns.extra_fs %}
tree: /guestfs/{{ inventory_hostname + '-' + ns.extra_fs['suffix'] + '/'
+ item['path'] |regex_replace(ns.extra_fs['container_mountpoint'], '') }}
{% else %}
tree: /var/lib/lxc/{{ inventory_hostname }}/rootfs{{ item['path'] }}
{% endif %}
rsh: ssh -i /root/.ssh/id_rsa_dirvish {{ hostvars[container_host].fqdn }}

{% for backup_path in (backup_paths | select('search', item['path'] +
'.*')) if not (backup_path['shortname'] == item['shortname']) %}
{% if loop.index == 1 %}
excludes:
{% endif %}
{{ backup_path['path'] }}*
{% endfor %}


As you can see, that's quite complicated.

Then in addition, I need to add to a list of acceptable paths on the
host (for my ssh forced command script to read).

That looks like this (for the same container described in the host_vars
above):

...
/var/lib/lxc/example-web1/rootfs/
/var/lib/lxc/example-web1/rootfs/var/
/var/lib/lxc/example-web1/rootfs/home/
/guestfs/example-web1-srv/
...

That isn't a template, because the file contains paths for all the
containers. I suppose I could modify my script to read any files in a
directory instead of a single file, but that would be a separate
project, and the template would be just as nasty as the one above.

I'm not even sure where to start with generating lines to add to the file.

Any suggestions?

Is this stuff reasonable to do with ansible, and a templating language?

Do I need to write a custom plugin of some kind?

If I had a dynamic inventory, then I could probably generate extra
variables at that stage, but that's further away too.

Cheers,
Richard

[Richard Hector]

On 15/05/21 6:39 pm, Richard Hector wrote:

> Any suggestions?
>
> Is this stuff reasonable to do with ansible, and a templating language?
>
> Do I need to write a custom plugin of some kind?
>
> If I had a dynamic inventory, then I could probably generate extra
> variables at that stage, but that's further away too.

I should give lots of credit to the nice people on irc who helped me get
as far as I have.

I've also thought that if I could pass data (as json perhaps?) to stdin
of a local script, and capture the output (a list of strings, as json
again?), that would probably work ok - I could then write the script in
a language I'm more familiar with, such as perl.

Is that doable?

Thanks,
Richard

[Richard Hector]

On 15/05/21 6:39 pm, Richard Hector wrote:

> Do I need to write a custom plugin of some kind?

I'm experimenting with writing a vars plugin, referring to these:

https://docs.ansible.com/ansible/latest/dev_guide/developing_plugins.html
https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/vars/host_group_vars.py

I've put it in the vars_plugins directory in my role.

So far, I can create variables, and access them from my role, which is good.

What I can't do, as far as I can see, is access existing facts/vars. Do
they exist at this point? Or is my plugin running at a lower level?

I was hoping they would be returned by the call to:

super(VarsModule, self).get_vars(loader, path, entities)

... but that returns None.

Is there a different kind of plugin that could do the job?

Thanks,
Richard

[Brian Coca]

plugins in general only get the variables they need, vars_plugins are
supposed to generate them, not change them, so they get very little
access to variables in general.

For data manipulation you want to create a filter plugin, you still
need to be explicit about the data you feed it.

--
----------
Brian Coca

[Richard Hector]

On 18/05/21 2:24 am, Brian Coca wrote:

> For data manipulation you want to create a filter plugin, you still
> need to be explicit about the data you feed it.

Thanks for that - this is what I've ended up with. There's probably some
copy/paste boilerplate that's not needed in my case.

Does it look reasonable?
I've had a suggestion that I should avoid lineinfile, but I'm not sure
what a good substitute is (other than rearranging my setup so I can
template out individual files, rather than editing the existing one).

Cheers,
Richard

------task----------------
- name: update dirvish-forced-command list
lineinfile:
dest: "/usr/local/etc/dirvish-forced-command/default.list"
regexp: "^{{ item }}$"
line: "{{ item }}"
insertafter: EOF
delegate_to: "{{ container_host }}"
with_items: "{{ backup_paths | gen_dfc_lines(inventory_hostname,
filesystems) }}"
tags:
- backup

------plugin------------

from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

from ansible.errors import AnsibleError, AnsibleFilterError,
AnsibleFilterTypeError
from ansible.module_utils.six import string_types, integer_types,
reraise, text_type

def gen_dfc_lines(backup_paths, container_name, filesystems):
'''Generate a list of lines for dirvish_forced_command'''
if not isinstance(filesystems, list):
raise AnsibleFilterTypeError('filesystems should be a list')
if not isinstance(backup_paths, list):
raise AnsibleFilterTypeError('backup_paths should be a list')

fixed_backup_paths = []
for path_entry in backup_paths:
found = False
for fs_entry in filesystems:
path = path_entry['path']
fs = fs_entry['container_mountpoint']
if path.startswith(fs):
# remove prefix from path
path = '/' + path[len(fs):]
# use guestfs path
fixed_path = "/guestfs/{cname}-{suffix}{bpath}".format(
cname = container_name,
suffix = fs_entry['suffix'],
bpath = path
)
found = True
if not found:
# use rootfs path
fixed_path = "/var/lib/lxc/{cname}/rootfs{bpath}".format(
cname = container_name,
bpath = path
)
fixed_backup_paths.append(fixed_path)
return fixed_backup_paths

class FilterModule(object):
''' backup helper filters '''

def filters(self):
return {
'gen_dfc_lines': gen_dfc_lines
}