generic 'package' module?

[Michael Johnson]

It has always sort of bothered me that ansible does not have a generic 'package' module that you then can tell it what 'provider' to use if you want to override the default like other tools such as puppet.

While doing some searching on the topic, I found this:

  https://github.com/ansible/ansible/issues/4261

That issue asks about this same question and the core of the response seems to be the last part of the last sentence:

  "as you have to make exemptions so many times, it really causes problems down the road."

While I can somewhat understand this philosophy, it does not seem to been one that is internally consistent.  Take the 'service' module for example.  Services can require just as many exemptions as packages.  If this philosophy is really the sticking point, why don't we have 'systemd_service', 'smf_service', 'upstart_service', etc.?

There may very well be a good explanation for the apparent inconsistency, but I am not seeing it.  Can anyone explain why these two things which seem very similar to me are treated very differently in ansible?

Would the ansible project be open to having a unified 'package' module?

[Matt Martz]

You can already do:

- action: "{{ ansible_pkg_mgr }}" name=some-pkg-name

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/56b70dd6-6b0c-44c5-a35f-32af8d149826%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Matt Martz
@sivel
sivel.net

[Brian Coca]

Yes, it is a contradiction, sadly it is very hard to remove 'service'
now, it is a huge pathway of detection and fallbacks and I would be
very happy to create a systemd, upstart, initd and other modules to
substitute it.

You just have to look at the tickets to see that every time we do a
minor modification to service it has many unintended consequences, it
is fragile, hard to read and very hard to debug. That and other
reasons made me keep I kept the svc/daemontools module separate.


--
Brian Coca

[Michael Johnson]

Ah!  That works wonderfully!

On Sunday, June 7, 2015 at 12:03:03 PM UTC-7, Matt Martz wrote:

You can already do:

- action: "{{ ansible_pkg_mgr }}" name=some-pkg-name

On Sunday, June 7, 2015, Michael Johnson <ther...@gmail.com> wrote:

It has always sort of bothered me that ansible does not have a generic 'package' module that you then can tell it what 'provider' to use if you want to override the default like other tools such as puppet.

While doing some searching on the topic, I found this:

  https://github.com/ansible/ansible/issues/4261

That issue asks about this same question and the core of the response seems to be the last part of the last sentence:

  "as you have to make exemptions so many times, it really causes problems down the road."

While I can somewhat understand this philosophy, it does not seem to been one that is internally consistent.  Take the 'service' module for example.  Services can require just as many exemptions as packages.  If this philosophy is really the sticking point, why don't we have 'systemd_service', 'smf_service', 'upstart_service', etc.?

There may very well be a good explanation for the apparent inconsistency, but I am not seeing it.  Can anyone explain why these two things which seem very similar to me are treated very differently in ansible?

Would the ansible project be open to having a unified 'package' module?

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.
To post to this group, send email to ansible-project@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/56b70dd6-6b0c-44c5-a35f-32af8d149826%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Johnson]

Got it, this makes perfect sense.  Always nice to have the background on inconsistencies like this (for me a least it removed a mental block).  With this background coupled with the notion of using 'action', perhaps 'service' could be deprecated and eventually removed in favor of different modules.

[Dick Davies]

Good point on the disparity (and despite generally working pretty well
for a big case statement,
service will occasionally give confusing errors when it guesses wrong).

Would it make sense to migrate to splitting out each service subtype
into its own module?
I might be being a bit optimistic but it'd allow a phased deprecation
- there are no existing modules
called 'upstart', 'init' etc so that should be straightforward (right?).

> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-proje...@googlegroups.com.
> To post to this group, send email to ansible...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/ansible-project/0f620dac-ea10-4af5-921b-c0edba118acf%40googlegroups.com.

[Brian Coca]

Yep, this is what I am thinking .. just need the time and resources to
make it so.

On Mon, Jun 8, 2015 at 3:18 AM, Dick Davies <di...@hellooperator.net> wrote:
> Would it make sense to migrate to splitting out each service subtype
> into its own module?
> I might be being a bit optimistic but it'd allow a phased deprecation
> - there are no existing modules
> called 'upstart', 'init' etc so that should be straightforward (right?).

--
Brian Coca

[Adam Kosmin]

On Sunday, June 7, 2015 at 3:03:03 PM UTC-4, Matt Martz wrote:

You can already do:

- action: "{{ ansible_pkg_mgr }}" name=some-pkg-name

I have my doubts as to how wise this approach is given the different set of parameters supported by each package module. I feel much better stuffing these OS-specific modules into their own files which get included based on things such as ansible_os_family.

Best,
Adam

[Brian Coca]

@Adam,

Agreed, this workaround just offsets the work of having 2 tasks to
having 2 data structures and is more limited than the other version,
this is one of the reasons why we do not recommend it, but it is
available if people insist going down that path.


--
Brian Coca

[Christian Thiemann]

I want all my Apache installation to behave the same on all OSs -- I want to write one logical model of how Apache should work, and all OS differences should just be different views of that model.

Thus, this is how my "install Apache packages" task looks like -- one task, one data structure, thanks to some YAML magic:

- include: install_packages.yml
  pkgs: !platform
    - apache::
        Debian: apache2
        RedHat: httpd
    - mod_authnz_external::
        Debian: libapache2-mod-authnz-external
    - python-passlib::  # needed for Ansible's htpasswd module
        Gentoo: dev-python/passlib
    - pwauth
  tags:
    - apache/setup

If I had separate include files per OS, then my list of packages would be duplicated in multiple files -- I mean the logical list of packages (i.e., "apache" + "mod_authnz_external" + "python's passlib"). If I have to add another mod_foobar in the future, I only have to add it to one list instead of having to remember to maintain multiple OS-dependent representations in multiple places.

[Giovanni Tirloni]

On Wed, 10 Jun 2015 08:43 -0700, Christian Thiemann
<thiem...@gmail.com> wrote:
> I want all my Apache installation to behave the same on all OSs -- I want
> to write one logical model of how Apache should work, and all OS
> differences should just be different views of that model.
>
> Thus, this is how my "install Apache packages" task looks like -- one
> task,
> one data structure, thanks to some YAML magic:
>
> - include: install_packages.yml
> pkgs: !platform
> - apache::
> Debian: apache2
> RedHat: httpd
> - mod_authnz_external::
> Debian: libapache2-mod-authnz-external
> - python-passlib:: # needed for Ansible's htpasswd module
> Gentoo: dev-python/passlib
> - pwauth
> tags:
> - apache/setup

All this magic feels like more typing and wasted brain cells than
keeping separate lists per distribution.

I've come to the conclusion that in Ansible you either hide complexity
in modules, roles or playbooks, in that order.

If the module isn't featured/mature enough (and you don't want or can't
improve it), you write roles that do the magic you need. It gets all
hidden from you and the playbooks look very clean.

Otherwise you let the imperfection fall through and deal with it in the
playbooks, with command-line options and what not.

I don't know what's best for your situation. It seems a role
encapsulating these "imperfections" might be enough.. or you could write
a module for completely managing Apache (from install to deploying new
websites).

They say, Ansible is a radically simple IT automation engine... I think
it's much more flexible than simple (perhaps simpler compared to the
absolute insanity of other automation tools). That comes with its
drawbacks.

Giovanni

[Dick Davies]

I've seen that attempted with Puppet and it works out badly in practice.

Different distros put files in different locations, use different init
systems, have
different conventions for 'includes', name packages differently.

Having one 'magic' class that makes it easy to type a config seems like it'll
save you time. until you have to debug it.

> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-proje...@googlegroups.com.
> To post to this group, send email to ansible...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/ansible-project/c138e7cc-9bac-46c4-bf75-deca40e290f0%40googlegroups.com.

[Brian Coca]

So to put this to bed:
http://docs.ansible.com/package_module.html

I still think it is not what you want or need, but now you have the option.


--
Brian Coca

[Adam Kosmin]

Brian,

Very cool. I am personally a big fan of this effort and am curious to see how it evolves.

Dick,

C'mon man, that's not true at all. The "works out poorly in practice" portion of your comment is more about any given implementation rather than the tool itself. We all know any of these tools *can* get the job done and in most cases, the cobweb of damage is constructed by the persons using it. I myself have used this type of abstraction to manage thousands of distributed systems with several OS's in site-specific areas all over the country. There are many other people out there with similar successes and I'd argue each of those wins goes back to 3 things: tools, design, and work flow. No tool is going to be a magic cure-all when proper design and work flows are lacking.

Best,
Adam

[Brian Coca]

The reason this works poorly in practice is that the undertaking is
like trying to set in stone a moving target.

packages are not named the same, the same app can appear under various names
packages are not split the same way, one app can be 1 package in one
distro or 3 in another
packages don't stay the same inside of a distro, they get renamed,
split, joined, etc, this mostly happens at distro version boundaries,
but not always
distros do not always have version boundaries, rolling release distros!
packages with the same name are not always the same app
even more fun, throw in virtual and meta packages, package groups!
some distros name per language milestone python2.5-package python3.4-package
others cut off at major versions py2-, py3, others at old/current py2, py-

now add external/custom repos outside the tighter control of the
distributions and hope that when they overlap ... it is actually what
you want.

now add per programming language package managers, some languages have multiple!

now try to remember what my sanity looked like.

This is what people really want solved, they want to say 'install
apache' and have something magically navigate the above quagmire and
give them what they wanted, not what they asked for. Sadly that is
much more than what an ansible module was designed to do, you required
a central database that is constantly updated and amended with every
variation, OS, distribution and version, a 'wikipackagesmedia' which
is a much larger project than I'm willing to attempt at this point in
time.

--
Brian Coca

[Christian Thiemann]

I don't necessarily want to save time, but have a well-designed setup that saves me from having headaches / making mistakes later on. Take the following task from my apache role:

- name: "disable SSLv2 & SSLv3"
  lineinfile:
    dest: !platform
      Debian: /etc/apache2/mods-available/ssl.conf
      Gentoo: /etc/apache2/vhosts.d/00_default_ssl_vhost.conf
      RedHat: /etc/httpd/conf.d/ssl.conf
    regexp: "{{ item.regexp }}"
    line: "{{ item.line }}"
  with_items:
    - { regexp: '^\s*SSLProtocol ', line: 'SSLProtocol ALL -SSLv2 -SSLv3' }
  notify:
    - reload apache
  tags:
    - apache/setup

It's our policy to not use SSLv2 or v3. That policy is expressed in one place (and one place only). If I had one lineinfile task for each OS (possibly in separate files), I'd have to duplicate the policy to all those places. If we want to disable TLS 1.0 in the future, it would have to be changed in multiple places -- something that programmers avoid rigorously, so why should it be different in a devops context? The thing I want to achieve is "install Apache on a server", not "install Apache on Debian" and "install Apache on RedHat" and "install Apache on Gentoo".

Ansible (and other cfg mgmt systems as well, I assume) imposes quite some annoying restrictions on ops work ("it would be so much easier to just SSH in and fix this manually"), but that's ok because it provides reproducibility and system documentation -- and OS abstraction. To me, that is one of the major points of using config management in the first place -- reproducibility and documentation can be achieved using shell scripts, possibly with less effort than by using cfg mgmt.

I'd say that the OS abstraction in ops is like the database abstraction layer in dev work. I've seen a few posts in this group which recommend creating separate Apache roles for each distro -- to me, they all sound like: "Database abstraction is really hard -- if your application needs to support both MySQL and Postgres as a backend, you should create two totally independent forks of your application, one for each database."

@Brian: I think the package module is a very good thing. For most use cases, this is the perfect module -- for everything else one can still use yum, apt, etc. in combination with "when: ansible_os_family == 'foobar'". The same design could be applied to the service module: split into separate modules so you can use all the fancy features of specific init systems if you want/need to, but also provide an "ansible_svc_mgr" fact and a "service" module / action plugin that work like "ansible_pkg_mgr" and the new "package" module.

[Brian Coca]

@Christian, those are my thoughts for the future of the service
module, this package module will serve as a template on how to split
it and keep backwards compatibility using the generic module as a
placeholder. Just need to add service system detection to facts and
then use those internally.

As for keeping the policy in a single place, the same can be achieved
by just making the ssl cipher string into a common variable and then
have each template reference it. This does not work for all cases but
I find it better to keep specific tasks for each OS and then abstract
the common data, than the reverse which abstracts the common task but
requires abstracting the non-common data.

Again, I'm all for choice, in this case I think it is an illusion as
you update 1 common yaml file and 2 non-common yaml files vs 2 common
yaml files and 1 non-common yaml file.


--
Brian Coca

[Christian Thiemann]

On Wednesday, June 10, 2015 at 11:17:25 PM UTC+2, Brian Coca wrote:

@Christian, those are my thoughts for the future of the service
module, this package module will serve as a template on how to split
it and keep backwards compatibility using the generic module as a
placeholder. Just need to add service system detection to facts and
then use those internally.

Cool!
 

As for keeping the policy in a single place, the same can be achieved
by just making the ssl cipher string into a common variable and then
have each template reference it. This does not work for all cases but
I find it better to keep specific tasks for each OS and then abstract
the common data, than the reverse which abstracts the common task but
requires  abstracting the non-common data.

That's only a partial solution -- you still have multiple copies of the policy "we set the SSL cipher string to something custom." The thing I want to achieve is "disable SSLv3 on our Apache servers", not "define a cipher string that disables SSLv3" and "set that cipher string on Debian" and "set that cipher string on RedHat" etc.

Say, you want to add support for Arch in the future. You add another include file but happen to forget to add the lineinfile task. Your playbooks run fine, POODLE happened a long time ago and you don't even think about SSLv3 anymore, but all your Arch servers will offer it (assuming Arch's default config still has it enabled, of course...).

My playbooks will fail on Arch hosts if I forget to add Arch's config filename to my "disable SSLv3" task, reminding me about my "no SSLv3" policy.
 

Again, I'm all for choice, in this case I think it is an illusion as
you update 1 common yaml file and 2 non-common yaml files  vs 2 common
yaml files and 1 non-common yaml file.

I am only editing 1 common yaml file -- roles/apache/tasks/main.yml -- that includes all the relevant data ("!platform" is a YAML tag that I add to the YAML parser (mis)using a vars_plugin, which, granted, is somewhat hacky and will hopefully still work in v2 ;-) ).

Re your other comment about package versions: yes, we will always have to manually curate some OS details. But cfg mgmt should help making that easier, not say "package NAMES are different anyways, so we won't even let you abstract the package MANAGER."

My task for installing PHP (different number of packages in different distros) looks like this:

- include: install_packages.yml
  pkgs: !

platform
    - php::
        Debian: php5
        RedHat: php55w
    - Debian: php5-cli
      RedHat: php55w-cli
    - Debian: php5-curl
  notify:
    - reload apache
  tags:
    - apache/php/setup

That installs the logical "php" package (which is provided by "php5" on Debian, and "php55w" on RedHat, and "php" everywhere else, i.e. Gentoo in my case). In addition, it will install the CLI SAPI on Debian and RedHat (that's included in "php" on Gentoo), and the curl extension on Debian (included in "php55w" on RedHat). All in one single yaml file.

Different versions of distros can be treated as distinct distros. Until recently (before we killed our last Ubuntu 10 and 12 servers), I had this in my Apache role:

  lineinfile:
    dest: !platform
      Ubuntu14: /etc/apache2/sites-enabled/000-default.conf
      Ubuntu12: /etc/apache2/sites-enabled/000-default
      Ubuntu10: /etc/apache2/sites-enabled/000-default

This wasn't perfect: there's still some duplication, but at least it's very close together and not spread over multiple files.

Rolling-release distributions are tricky -- luckily I am able to keep all my (few) Gentoo servers in sync, so I don't need to support multiple "versions" of Gentoo.

[Brian Coca]

Any intent on sharing that vars plugin? it sounds like a nice way to
unify vars in a single by distro vs per distro file.


--
Brian Coca

[Christian Thiemann]

Yes, been meaning to do that for some time, but haven't gotten around to properly cleaning it up etc.  Happy to create a PR if you are interested -- for now, here's the rough version:

https://gist.github.com/cthiemann/220c128e298614a869f2

The filter_plugin returns the value of a dictionary that exists for a list of keys. Example:

# vars file
foo:
  Debian: bar
  Ubuntu: baz
  Gentoo: abc

# Jinja template
{{ foo|selectby(['Gentoo']) }} == abc
{{ foo|selectby(['Debian', 'Ubuntu']) }} == bar
{{ foo|selectby(['Ubuntu', 'Debian']) }} == baz

The vars_plugin defines a variable for each host that's a list of ansible_distribution + version, ansible_os_family, etc. in descending order of specificity, and modifies the YAML parser to recognize dictionaries tagged !platform, which will be converted into Jinja templates using the |selectby filter. Example:

# vars file
foo: !platform
  Debian: bar
  Ubuntu: baz
  Gentoo: abc

# Jinja template
{{ foo }} == abc if ansible_os_family == 'Gentoo'
{{ foo }} == bar if ansible_os_family == 'Debian' and not ansible_distribution == 'Ubuntu'
{{ foo }} == baz if ansible_distribution == 'Ubuntu'

The |selectby filter is a bit more complex than just this, it also transparently iterates over lists, so that these two are equivalent:

foo1:
  - !platform
    Debian: bar
  - !platform
    Debian: barbara

foo2: !platform
  - Debian: bar
  - Debian: barbara

And it allows to define a default value:

foo: !platform
  default-value::
    Debian: override-value

{{ foo }} == ('override-value' if ansible_os_family == 'Debian' else 'default-value')

[Roman Belyakovsky]

Hi Christian,

I didn't get how did you do the OS specific package installation. I've ended up with a temporary solution:

- name: install python-passlib

  action: "{{ ansible_pkg_mgr }} name={{ item.name }}"

  with_items:

      - { os: Gentoo, name: dev-python/passlib }

  when: ansible_os_family  == item.os

But I do like the way you've shown. Could I possibly ask you to give me an example with the 'action' module? And should it work in ansible 1.9?