Failed to find file, with shared vars

[Michael Raugh - NOAA Affiliate]

I am hoping to leverage the wisdom of the group here.  For over a year, I have been running a playbook that starts out by pulling variables from a separate role:

# Load required variables

- import_role:

    name: sitevars

  tags: always

Basically, that extracts the things that vary by site, like the DHCP server, monitoring server, Nessus scanner, NTP, etc.  It's worked flawlessly for something line 15 months.  Then, today, I tried to add a new site.

-rw-rw-r--+ 1 mraugh mraugh   28 Jan  2 14:44 main.yml

-rw-rw-r--+ 1 mraugh mraugh  918 Jun 25 10:59 sitevars-Asheville.yml

-rw-rw-r--+ 1 mraugh mraugh 1150 Jun 25 10:59 sitevars-Boulder.yml

-rw-rw-r--+ 1 mraugh mraugh  971 Jun 25 10:59 sitevars-NJO.yml

-rw-rw-r--+ 1 mraugh mraugh 1076 Jun 25 10:59 sitevars-NSOF.yml

-rw-rw-r--+ 1 mraugh mraugh 1138 Jun 25 10:59 sitevars-Red_Forest.yml

-rw-rw-r--+ 1 mraugh mraugh 1138 Jun 25 10:59 sitevars-Silver_Spring.yml

-rw-rw-r--+ 1 mraugh mraugh  918 Jun 25 10:59 sitevars-STAR.yml

-rw-rw-r--+ 1 mraugh mraugh  922 Jun 25 10:59 sitevars-Stennis.yml

-rw-rw-r--+ 1 mraugh mraugh  927 Jun 26 13:54 sitevars-Wallops.yml

Wallops (sitevars-Wallops.yml) is the new file.  But ansible insists that it's not there:

fatal: [atlas-wal]: FAILED! => {"ansible_facts": {"site": {}}, "ansible_included_var_files": [], "changed": false, "message": "Could not find or access 'sitevars-Wallops.yml'\nSearched in:\n\t/etc/ansible/roles/sitevars/vars/sitevars-Wallops.yml\n\t/etc/ansible/roles/sitevars/sitevars-Wallops.yml\n\t/etc/ansible/tower/baselines/roles/rhel7-base/vars/sitevars-Wallops.yml\n\t/etc/ansible/tower/baselines/roles/rhel7-base/sitevars-Wallops.yml\n\t/etc/ansible/roles/sitevars/tasks/vars/sitevars-Wallops.yml\n\t/etc/ansible/roles/sitevars/tasks/sitevars-Wallops.yml\n\t/etc/ansible/tower/baselines/vars/sitevars-Wallops.yml\n\t/etc/ansible/tower/baselines/sitevars-Wallops.yml on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}

fatal: [bran-wal]: FAILED! => {"ansible_facts": {"site": {}}, "ansible_included_var_files": [], "changed": false, "message": "Could not find or access 'sitevars-Wallops.yml'\nSearched in:\n\t/etc/ansible/roles/sitevars/vars/sitevars-Wallops.yml\n\t/etc/ansible/roles/sitevars/sitevars-Wallops.yml\n\t/etc/ansible/tower/baselines/roles/rhel7-base/vars/sitevars-Wallops.yml\n\t/etc/ansible/tower/baselines/roles/rhel7-base/sitevars-Wallops.yml\n\t/etc/ansible/roles/sitevars/tasks/vars/sitevars-Wallops.yml\n\t/etc/ansible/roles/sitevars/tasks/sitevars-Wallops.yml\n\t/etc/ansible/tower/baselines/vars/sitevars-Wallops.yml\n\t/etc/ansible/tower/baselines/sitevars-Wallops.yml on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}

It's not griping about the content, but it seems convinced that the file does not exist, even though it plainly does.  Any ideas?

<MR>

[Kai Stian Olstad]

On 26.06.2019 20:54, 'Michael Raugh - NOAA Affiliate' via Ansible

This doesn't help us much since this could be a listing of any
directory, the placement of the directory i crucial for Ansible.
Maybe the files is duplicated in several directory, and only one of them
is the correct directory.
Maybe your main.yml is incorrect written.
But is does say that you are using ACL.

> Wallops (sitevars-Wallops.yml) is the new file. But ansible insists
> that
> it's not there:
> fatal: [atlas-wal]: FAILED! => {"ansible_facts": {"site": {}},
> "ansible_included_var_files": [], "changed": false, "message": "Could
> not
> find or access 'sitevars-Wallops.yml'\nSearched
> in:\n\t/etc/ansible/roles/sitevars/vars/sitevars-Wallops.yml\n\t/etc/ansible/roles/sitevars/sitevars-Wallops.yml\n\t/etc/ansible/tower/baselines/roles/rhel7-base/vars/sitevars-Wallops.yml\n\t/etc/ansible/tower/baselines/roles/rhel7-base/sitevars-Wallops.yml\n\t/etc/ansible/roles/sitevars/tasks/vars/sitevars-Wallops.yml\n\t/etc/ansible/roles/sitevars/tasks/sitevars-Wallops.yml\n\t/etc/ansible/tower/baselines/vars/sitevars-Wallops.yml\n\t/etc/ansible/tower/baselines/sitevars-Wallops.yml
> on the Ansible Controller.\nIf you are using a module and expect the
> file
> to exist on the remote, see the remote_src option"}
> fatal: [bran-wal]: FAILED! => {"ansible_facts": {"site": {}},
> "ansible_included_var_files": [], "changed": false, "message": "Could
> not
> find or access 'sitevars-Wallops.yml'\nSearched
> in:\n\t/etc/ansible/roles/sitevars/vars/sitevars-Wallops.yml\n\t/etc/ansible/roles/sitevars/sitevars-Wallops.yml\n\t/etc/ansible/tower/baselines/roles/rhel7-base/vars/sitevars-Wallops.yml\n\t/etc/ansible/tower/baselines/roles/rhel7-base/sitevars-Wallops.yml\n\t/etc/ansible/roles/sitevars/tasks/vars/sitevars-Wallops.yml\n\t/etc/ansible/roles/sitevars/tasks/sitevars-Wallops.yml\n\t/etc/ansible/tower/baselines/vars/sitevars-Wallops.yml\n\t/etc/ansible/tower/baselines/sitevars-Wallops.yml
> on the Ansible Controller.\nIf you are using a module and expect the
> file
> to exist on the remote, see the remote_src option"}

So is this one of the paths that the files is in?

> It's not griping about the content, but it seems convinced that the
> file
> does not exist, even though it plainly does. Any ideas?

Since your are using ACL, ACL might deny you access to the file so you
should check the actual permission.


--
Kai Stian Olstad

[Michael Raugh - NOAA Affiliate]

The directory is the first one checked in the path:  /etc/ansible/tower/sitevars/vars/sitevars-Wallops.yml.  The FACL on it matches all the other site files, so it's not that.  The file is not duplicated anywhere other that the Git repository.  Checked permissions there, too.

My main.yml for tasks contains only this:

---

# tasks file for sitevars

- include_vars: 

    file: sitevars-{{ site }}.yml

    name: site

Which seems to be working for the other 8 sites just fine, and has been for a while.

Still perplexed.

<MR>

[Kai Stian Olstad]

On 02.07.2019 20:04, 'Michael Raugh - NOAA Affiliate' via Ansible Project wrote:
> The directory is the first one checked in the path:
> /etc/ansible/tower/sitevars/vars/sitevars-Wallops.yml. The FACL on it
> matches all the other site files, so it's not that. The file is not
> duplicated anywhere other that the Git repository. Checked permissions
> there, too.
>
> My main.yml for tasks contains only this:
> ---
> # tasks file for sitevars
>
> - include_vars:
> file: sitevars-{{ site }}.yml
> name: site
>
> Which seems to be working for the other 8 sites just fine, and has been for
> a while.
>
> Still perplexed.

One option is that a non ascii has snuck in somewhere, that and a lot more is easy to check with strace

strace -f -x -o ansible.strace ansible-playbook playbook.yml

Then you should have stat(..) line with the filename in ansible.strace and openat(...)
and because of -x it will show any non ascii character if there are any.


--
Kai Stian Olstad

[Michael Raugh - NOAA Affiliate]

Thought of that; nope.  I even deleted the file and started over, no dice.  Also tried backing off the ansible version from 2.8.1 to 2.4.2, and that didn't help.  Neither did setting SELinux to Permissive.  

I have to think it's in the mechanism that retries the sitevars from Git. The latest version is in there, though.  I'm pulling my hair out, and I don't have much left as it is.

<MR>

[Michael Raugh - NOAA Affiliate]

Sure enough, moving the sitevars into the same playbook with the baseline worked around the problem.  So I at least know where the problem is and I have a workaround for now.

<MR>