Nested loop: referring to the value of the current item of the outer loop

[Tadej Janež]

Hi!

I'm trying to develop a simple backup solution with Ansible.

My inventory file contains:
[backup-hosts]
backup-host1
backup-host2

[backup-targets]
backup-target1
backup-target2
backup-target3
backup-target4

Each backup-target has a host_vars file with contents similar to the following:
# list of users and their directories that should be backed up
backup_target_users_dirs:
  - user: "alice"
    dirs:
      - "Maildir"
      - "Important"
  - user: "bob"
    dirs:
      - "Maildir"
      - "ProjectX"

I have a role that is played on each backup-host that contains
a task to prepare directories for each backup target:

- name: prepare directories for each backup target
  file:
    path: "~/{{ item[0] }}/{{ item[1]['dir'] }}"
    state: directory
  sudo: yes
  sudo_user: "{{ backup_host_user }}"
  with_nested:
    - groups["backup-targets"]
    - hostvars[item[0]]["backup_target_users_dirs"]

As it is, it doesn't work. What I would like to achieve is that "item[0]" in
the nested loop would refer to the value of the current item of the outer loop.
How can I do that?

Thanks and best regards,
Tadej

[Tadej Janež]

Hi!

Just a quick ping. Does anyone have any idea how I could solve the
problem described below? Maybe a work-around?

Thanks and best regards,
Tadej

[Henning Eggers]

Hi,

I think you might want to use with_subelements instead of with_nested.

http://docs.ansible.com/playbooks_loops.html#looping-over-subelements

Cheers,

Henning

[mic...@mcrilly.me]

Hi,

I was hoping to get more details from you regarding this issue. I appreciate what you're trying to do at Ansible's level, but what's going to do the data transfer? If it's rsync, then I would like to propose a new approach to this, but if it's not, I need to know before I can write some example code for you.

Cheers,

Mike.

[Tadej Janež]

Hi Henning,

On Thu, 2015-01-29 at 06:43 -0800, Henning Eggers wrote:

> I think you might want to use with_subelements instead of with_nested.

> http://docs.ansible.com/playbooks_loops.html#looping-over-subelements

thanks for your advice.

Unfortunately, I think 'with_subelements' is not powerful enough to do
what I want.
As explained in the documentation:
"Subelements walks a list of hashes (aka dictionaries) and then
traverses a list with a given key inside of those records."

What I want to do is a bit different. I need to extract information from
the 'hostvars' hash and I only want to extract the information for hosts
belonging to a specific group (i.e. groups["backup-targets"]).

Regards,
Tadej

[Tadej Janež]

Hi,

On Sun, 2015-02-01 at 03:39 -0800, mic...@mcrilly.me wrote:
> I was hoping to get more details from you regarding this issue. I
> appreciate what you're trying to do at Ansible's level, but what's
> going to do the data transfer? If it's rsync, then I would like to
> propose a new approach to this, but if it's not, I need to know before
> I can write some example code for you.

Yes, I intend to use rsync.

My plan is to use a Bash script template that uses rsync and use Ansible
to fill in the directories that need to be rsync-ed. I intend to use a
separate Bash script for each backup target.
Then I plan to use another Ansible task that will set up a cron job for
each of these backup targets.

Of course, I'm eager to hear new approaches to the problem I'm tackling.

Regards,
Tadej

[mic...@mcrilly.me]

Hi,

I think your approach has been overthought, a very common issue everyone runs into when designing a backup solution. I also think you should use an existing backup solution, such as http://www.rsnapshot.org. Ideally you want a solution that can be configured from a central point and left to do its job.

One thing to remember, also, is that Ansible is a Configuration Management suite. It's not designed to replace a scripting or programming language - it simply can't do the same things. Instead, use Ansible to configure a piece of software, like rsnapshot, and then maintain this state to fit your needs.

I believe you need to simplify the data structures in your example cases. Also, instead of having a list of users to backup along with their individual backup targets, use rsync's '--exclude-from' and '--include-from' to build a list of regular expressions or globs for matching what you do and do not want to backup, like:

+/home/alice/*
+/home/bob/my_files/*.txt
-/root/*

If you do this, you just need a set of templates for populating the exclude and include files, which you can then feed to rsync, alone with the remote system's (IP) address. Working with these patterns means you're not dealing with complex data structures from within Ansible, but instead have something simple such as this:

backup_target_systems:
  - system: "localdev"
    active: true
    backup_include_patterns:
      - "/home/alice/"
      - "/home/bob/files/"
    backup_exclude_patterns:
      - "/root/"

I hope this has helped. Please let me know if you need any further assistance.

- M