How to handle lists of files and avoid unnecessary changes
19 views
Skip to first unread message
Marc Haber
Nov 8, 2017, 2:47:22 PM11/8/17
to ansible...@googlegroups.com
Hi,
on Debian systems, one can configure repositories in an
/etc/apt/sources.list.d directory with multiple files. I usually name
the files after the distribution, so that, in a simplyfied example, I
might have a file /etc/apt/sources.list.d/stable.list on systems running
Debian stable, and /etc/apt/sources.list.d/testing.list on systems
running Debian testing. In the real case, there are additional
distribution-depending files (security, backports, local packages in
different stages).
A similiar setup can usually be found in /etc/yum.repos.d/ on Red Hat
based systems.
A local admin might choose to place additional files into those
directories manually, for example adding special repositories for
third-party software that doesn't come with the distribution.
But back to the Debian case.
When I change a system from testing to stable, I want the all
testing*.list to vanish, and the stable*.list files to appear, unless
they're already there, in which case I want them untouched. Locally
placed files should also be untouched.
My first approach was to concentrate all the ansible-managed *.list
files under a common prefix, zda, giving, for example,
zda-unstable.list.
I then wrote code to first remove all zda*.list files (and their laegacy-named
instances), and code to deploy the correct zda-foo.list file:
- name: search for sources.list files
find:
paths: "/etc/apt/sources.list.d"
patterns: "zda-*.list,exp-mc.list,sid-mc.list,sid-zg-stable-mc.list,sid-zg-unstable-mc.list,stretch-mc.list,stretch-security.list,stretch-zg-stable-mc.list,stretch-zg-unstable-mc.list,buster-mc.list,stretch-zg-stable-mc.list,stretch-security.list"
register: sourceslistfiles
- name: delete sources.list files
file:
path: "{{ item.path }}"
state: absent
with_items: "{{ sourceslistfiles.files }}"
and finally code to roll out new list files:
- name: include repositories
tags:
repos
include_tasks:
"{{distribution}}/{{distribution_version}}/repos.yml"
$ cat roles/common/tasks/debian/stretch/repos.yml
---
- name: zda-stretch-mc.list
tags:
- repos
- stretch
copy:
dest: /etc/apt/sources.list.d/zda-stretch-mc.list
owner: root
group: root
mode: 0644
content: |
deb http://debian.debian.zugschlus.de/debian/ stretch main contrib
notify: apt update
- name: zda-stretch-security.list
tags:
- repos
- stretch
copy:
dest: /etc/apt/sources.list.d/zda-stretch-security.list
owner: root
group: root
mode: 0644
content: |
deb http://debian-security.debian.zugschlus.de/debian-security/ stretch/updates main contrib
deb http://security.debian.org/ stretch/updates main contrib
notify: apt update
But of course this does not quiet what I want, as it first zaps all
files, and then proceeds to roll out the "new" files again, resulting in
ctime and inode values being changed etc.
How about the following:
- build the list of files found on the remote system
- roll out the new contents, remove the name of the file from the list
- delete the remainder of files
or
- roll out the new contents, build a list of files being rolled out
- build the list of files found on the remote system
- delete files that are in list 2 but not in list 1.
Will that work? How would I write that? Is it a common idiom? Or am I
better off with living with ever-changing inode numbers?
I don't want to keep a list of files in the ansible code as this will
never be current, and I don't want to use the same file names regardless
of which distribution is used (I like to be able to look in the
directory to see what I'll find inside the files).
Greetings
Marc
--
-----------------------------------------------------------------------------
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
on Debian systems, one can configure repositories in an
/etc/apt/sources.list.d directory with multiple files. I usually name
the files after the distribution, so that, in a simplyfied example, I
might have a file /etc/apt/sources.list.d/stable.list on systems running
Debian stable, and /etc/apt/sources.list.d/testing.list on systems
running Debian testing. In the real case, there are additional
distribution-depending files (security, backports, local packages in
different stages).
A similiar setup can usually be found in /etc/yum.repos.d/ on Red Hat
based systems.
A local admin might choose to place additional files into those
directories manually, for example adding special repositories for
third-party software that doesn't come with the distribution.
But back to the Debian case.
When I change a system from testing to stable, I want the all
testing*.list to vanish, and the stable*.list files to appear, unless
they're already there, in which case I want them untouched. Locally
placed files should also be untouched.
My first approach was to concentrate all the ansible-managed *.list
files under a common prefix, zda, giving, for example,
zda-unstable.list.
I then wrote code to first remove all zda*.list files (and their laegacy-named
instances), and code to deploy the correct zda-foo.list file:
- name: search for sources.list files
find:
paths: "/etc/apt/sources.list.d"
patterns: "zda-*.list,exp-mc.list,sid-mc.list,sid-zg-stable-mc.list,sid-zg-unstable-mc.list,stretch-mc.list,stretch-security.list,stretch-zg-stable-mc.list,stretch-zg-unstable-mc.list,buster-mc.list,stretch-zg-stable-mc.list,stretch-security.list"
register: sourceslistfiles
- name: delete sources.list files
file:
path: "{{ item.path }}"
state: absent
with_items: "{{ sourceslistfiles.files }}"
and finally code to roll out new list files:
- name: include repositories
tags:
repos
include_tasks:
"{{distribution}}/{{distribution_version}}/repos.yml"
$ cat roles/common/tasks/debian/stretch/repos.yml
---
- name: zda-stretch-mc.list
tags:
- repos
- stretch
copy:
dest: /etc/apt/sources.list.d/zda-stretch-mc.list
owner: root
group: root
mode: 0644
content: |
deb http://debian.debian.zugschlus.de/debian/ stretch main contrib
notify: apt update
- name: zda-stretch-security.list
tags:
- repos
- stretch
copy:
dest: /etc/apt/sources.list.d/zda-stretch-security.list
owner: root
group: root
mode: 0644
content: |
deb http://debian-security.debian.zugschlus.de/debian-security/ stretch/updates main contrib
deb http://security.debian.org/ stretch/updates main contrib
notify: apt update
But of course this does not quiet what I want, as it first zaps all
files, and then proceeds to roll out the "new" files again, resulting in
ctime and inode values being changed etc.
How about the following:
- build the list of files found on the remote system
- roll out the new contents, remove the name of the file from the list
- delete the remainder of files
or
- roll out the new contents, build a list of files being rolled out
- build the list of files found on the remote system
- delete files that are in list 2 but not in list 1.
Will that work? How would I write that? Is it a common idiom? Or am I
better off with living with ever-changing inode numbers?
I don't want to keep a list of files in the ansible code as this will
never be current, and I don't want to use the same file names regardless
of which distribution is used (I like to be able to look in the
directory to see what I'll find inside the files).
Greetings
Marc
--
-----------------------------------------------------------------------------
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Kai Stian Olstad
Nov 8, 2017, 3:58:57 PM11/8/17
to ansible...@googlegroups.com
On 08.11.2017 20:47, Marc Haber wrote:
> When I change a system from testing to stable, I want the all
> testing*.list to vanish, and the stable*.list files to appear, unless
> they're already there, in which case I want them untouched. Locally
> placed files should also be untouched.
How you can implement this depends on the feature you need
> When I change a system from testing to stable, I want the all
> testing*.list to vanish, and the stable*.list files to appear, unless
> they're already there, in which case I want them untouched. Locally
> placed files should also be untouched.
1. one server having a several repo stable, testing and unstable
2. one server only having stable, testing or unstable.
> My first approach was to concentrate all the ansible-managed *.list
> files under a common prefix, zda, giving, for example,
> zda-unstable.list.
>
> I then wrote code to first remove all zda*.list files (and their
> laegacy-named
> instances), and code to deploy the correct zda-foo.list file:
>
> - name: search for sources.list files
> find:
> paths: "/etc/apt/sources.list.d"
> patterns:
> "zda-*.list,exp-mc.list,sid-mc.list,sid-zg-stable-mc.list,sid-zg-unstable-mc.list,stretch-mc.list,stretch-security.list,stretch-zg-stable-mc.list,stretch-zg-unstable-mc.list,buster-mc.list,stretch-zg-stable-mc.list,stretch-security.list"
> register: sourceslistfiles
> - name: delete sources.list files
> file:
> path: "{{ item.path }}"
> state: absent
> with_items: "{{ sourceslistfiles.files }}"
Ansible do the work.
contain stable, testing or unstable.
Then your tasks would look something like this
- apt_repository:
repo: deb http://debian.debian.zugschlus.de/debian/ stretch main
contrib
filename: zda-stretch-mc
state: "{{ (repo_type == 'stable') | ternary('present', 'absent')
}}"
State present creates the file, and absent remove the file so Ansible
will clean up the files so nothing is left behind when you change
repo_type.
You would need one for each repo and repo_type.
But with with_items and facts you could probably make a loop to deal
with most of them.
With feature 2 you could have something like stable_state, testing_state
and unstable_state and set them to absent or present, and then just use
that variable in state.
Hopefully this make some sense.
--
Kai Stian Olstad
Marc Haber
Nov 9, 2017, 5:17:57 AM11/9/17
to ansible...@googlegroups.com
On Wed, Nov 08, 2017 at 09:58:32PM +0100, Kai Stian Olstad wrote:
> On 08.11.2017 20:47, Marc Haber wrote:
> > When I change a system from testing to stable, I want the all
> > testing*.list to vanish, and the stable*.list files to appear, unless
> > they're already there, in which case I want them untouched. Locally
> > placed files should also be untouched.
>
> How you can implement this depends on the feature you need
> 1. one server having a several repo stable, testing and unstable
> 2. one server only having stable, testing or unstable.
2, but with execptions. The general function that I want is "zap
> On 08.11.2017 20:47, Marc Haber wrote:
> > When I change a system from testing to stable, I want the all
> > testing*.list to vanish, and the stable*.list files to appear, unless
> > they're already there, in which case I want them untouched. Locally
> > placed files should also be untouched.
>
> How you can implement this depends on the feature you need
> 1. one server having a several repo stable, testing and unstable
> 2. one server only having stable, testing or unstable.
everything matching a glob, execpt files that would be written to by
another task"
> > My first approach was to concentrate all the ansible-managed *.list
> > files under a common prefix, zda, giving, for example,
> > zda-unstable.list.
> >
> > I then wrote code to first remove all zda*.list files (and their
> > laegacy-named
> > instances), and code to deploy the correct zda-foo.list file:
> >
> > - name: search for sources.list files
> > find:
> > paths: "/etc/apt/sources.list.d"
> > patterns:
> > "zda-*.list,exp-mc.list,sid-mc.list,sid-zg-stable-mc.list,sid-zg-unstable-mc.list,stretch-mc.list,stretch-security.list,stretch-zg-stable-mc.list,stretch-zg-unstable-mc.list,buster-mc.list,stretch-zg-stable-mc.list,stretch-security.list"
> > register: sourceslistfiles
> > - name: delete sources.list files
> > file:
> > path: "{{ item.path }}"
> > state: absent
> > with_items: "{{ sourceslistfiles.files }}"
>
> To clean up legacy I would only run this once and after that make Ansible do
> the work.
I want to be able to switch a host from one distribution to another with
the files belonging to the "old" one cleaned up automatically.
> Feature 1 only need one variable, call it "repo_type", this could contain
> stable, testing or unstable.
>
> Then your tasks would look something like this
>
> - apt_repository:
> repo: deb http://debian.debian.zugschlus.de/debian/ stretch main contrib
> filename: zda-stretch-mc
> state: "{{ (repo_type == 'stable') | ternary('present', 'absent') }}"
few hosts that have multple repositories would get the extra repos
outside of ansible. I think I can live with that, but it's not pretty.
> With feature 2 you could have something like stable_state, testing_state and
> unstable_state and set them to absent or present, and then just use that
> variable in state.
But I have to admit that my sense of prettiness is not yet tuned to
ansible.
Marc Haber
Nov 9, 2017, 10:17:26 AM11/9/17
to ansible...@googlegroups.com
On Wed, Nov 08, 2017 at 08:47:08PM +0100, Marc Haber wrote:
> - roll out the new contents, build a list of files being rolled out
> - build the list of files found on the remote system
> - delete files that are in list 2 but not in list 1.
This is what I did and it works. Here my solution:
> - roll out the new contents, build a list of files being rolled out
> - build the list of files found on the remote system
> - delete files that are in list 2 but not in list 1.
- name: search for sources.list files
find:
paths: "/etc/apt/sources.list.d"
patterns: "zda-*.list,exp-mc.list,sid-mc.list,sid-zg-stable-mc.list,sid-zg-u
- name: set up fact list for wanted sources.list.d entries
set_fact:
wantedsourceslistfiles:
- name: include repositories
tags:
repos
include_tasks:
"{{distribution}}/{{distribution_version}}/repos.yml"
tags:
repos
include_tasks:
"{{distribution}}/{{distribution_version}}/repos.yml"
- name: delete sources.list files
file:
path: "{{ item.path }}"
state: absent
with_items: "{{ presentsourceslistfiles.files }}"
file:
path: "{{ item.path }}"
state: absent
when: item.path not in wantedsourceslistfiles
debian/sid/repos.yml:
- name: create list of sid wantedsourceslistfiles
set_fact:
sidwantedsourceslistfiles:
- /etc/apt/sources.list.d/zda-sid-mc.list
- /etc/apt/sources.list.d/zda-sid-zg-stable-mc.list
- /etc/apt/sources.list.d/zda-sid-zg-unstable-mc.list
- name: add sid files to wantedsourceslistfiles
set_fact:
wantedsourceslistfiles: "{{ wantedsourceslistfiles }} + {{ sidwantedsourceslistfiles }}"
- name: zda-sid-mc.list
copy:
dest: /etc/apt/sources.list.d/zda-sid-mc.list
owner: root
group: root
mode: 0644
content: |
deb http://debian.debian.zugschlus.de/debian/ sid main contrib
group: root
mode: 0644
content: |
notify: apt update
- name: zda-sid-zg-stable-mc.list
copy:
dest: /etc/apt/sources.list.d/zda-sid-zg-stable-mc.list
owner: root
group: root
mode: 0644
content: |
deb http://zg20150.debian.zugschlus.de/zg20150/ sid-zg-stable main contrib
group: root
mode: 0644
content: |
notify: apt update
- name: zda-sid-zg-unstable-mc.list
copy:
dest: /etc/apt/sources.list.d/zda-sid-zg-unstable-mc.list
owner: root
group: root
mode: 0644
content: |
deb http://zg20150.debian.zugschlus.de/zg20150/ sid-zg-unstable main contrib
group: root
mode: 0644
content: |
notify: apt update
Reply all
Reply to author
Forward
0 new messages