Template/File path for dependencies

[Paul Oyston]

Hi guys,

I had a bit of an idea surrounding the template and file path for dependencies, wondering what everyone thinks of this.

At the moment when you use meta/main.yml - dependencies the template path and file path is set to the role that you're running in. For example if you were to use a example.nginx role the template path would be something like /etc/ansible/roles/example.nginx/templates.

When you're calling dependencies it's likely that you may want to override a template or pass a template in to the role. For example you may wish to do something like:

# file: /etc/ansible/roles/example.app/meta/main.yml
---
dependencies:
  - role: example.nginx
    server_template: server.conf.j2

Where the server.conf.j2 file is contained within the example.app role. At the moment the only workaround to this is to use the absolute path which may not be very portable.

# file: /etc/ansible/roles/example.app/meta/main.yml
---
dependencies:
  - role: example.nginx
    server_template: /etc/ansible/roles/example.app/templates/server.conf.j2

I don't mind implementing this feature myself if it's considered to be a good idea.

[Michael DeHaan]

I don't understand what you are suggesting to implement, though this is probably a good discussion for ansible-devel instead of the main list.

We would like to see what syntax you are proposing though it *sounds* like you might be saying to search the various role dependency chain items to find the first file up the chain starting at the current role.

Really, role deps shouldn't be looked at like inheritance though - they are simply NOT inheritance.  They are things to do *first*.

It might be better to pass a parameter into the role instead (like you have) with what filename to use, I'm unclear as to why you think it needs an absolute path though.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/f4f5ca9c-fc82-4d55-a471-2f3cdee70f03%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Petros Moisiadis]

On 05/27/14 20:26, Michael DeHaan wrote:
> Really, role deps shouldn't be looked at like inheritance though -
> they are simply NOT inheritance. They are things to do *first*.
>
> It might be better to pass a parameter into the role instead (like you
> have) with what filename to use, I'm unclear as to why you think it
> needs an absolute path though.

But we could make role dependencies allow for template overrides /
injections. Judging by discussions in several related topics, it seems
that this would be a feature that people would want to have, so I do not
understand why you are against some little level of inheritance for the
templates. In fact, I can see great potential if we make role
dependencies smart enough to allow template inheritance between
dependent roles. Jinja already has this power built-in. Why don't we
take advantage of it? Imagine an 'apache' role that provides a vhost
template with a "{% block extra %}{% endblock %}" jinja block and a
'wsgi_app' role which depends on 'apache' role and provides content for
that 'extra' block. That would allow for a really modular design of
roles, instead of having to maintain huge, monolithic roles that do
everything with a lot of conditional includes and passing "strange" role
parameters. Role parameters are exactly what you want for things such as
'http_port' or 'bind_address', but using them for practically choosing a
"more specific role for the role" is far from elegant design.

[Michael DeHaan]

"But we could make role dependencies allow for template overrides /
injections"

I find this kind of language hard to understand.

", I can see great potential if we make role

dependencies smart enough to allow template inheritance between
dependent roles. "

I don't even know what this means.

Ansible isn't a programing language, I would find this easier if we did not try to use the phrase "inheritance" where it didn't make sense.

Did you mean including a template in another template?

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/5384D232.9000106%40yahoo.gr.

[Petros Moisiadis]

On 05/27/2014 11:04 PM, Michael DeHaan wrote:

"But we could make role dependencies allow for template overrides /
injections"

I find this kind of language hard to understand.

", I can see great potential if we make role

dependencies smart enough to allow template inheritance between
dependent roles. "

I don't even know what this means.

Ansible isn't a programing language, I would find this easier if we did not try to use the phrase "inheritance" where it didn't make sense.

Did you mean including a template in another template?

The phrase "template inheritance" is what is officially used in the jinja2 docs (http://jinja.pocoo.org/docs/templates/#template-inheritance). I am copying from jinja2 docs:
"Template inheritance allows you to build a base “skeleton” template that contains all the common elements of your site and defines blocks that child templates can override."

So, what I am suggesting is adding a new possibility on top of role dependencies, which would allow to write a template in a dependent role that extends a template of a role it depends to. In other words, a role would be allowed to have child templates that provide content for blocks defined in a 'skeleton' template of a role it depends to. That means when a role is being deployed because of a role dependency, its template tasks could use blocks possibly provided by any role that depends on it and is included in the active dependency chain. I have already given an example of a 'wsgi_app' role that depends on an 'apache_role'. The 'apache_role' would provide a template with all common things as well as an "extra" block, which could be overridden by the "wsgi_app" role in order to provide wsgi-specific functionality to the apache configuration.

All this allows for a really modular design of roles. An 'apache' role with a good skeleton template would allow to have a great variety of dependent roles such as 'fastcgi_app', 'wsgi_app', 'django_app', 'ruby_on_rails_app', etc., without having to repeat common functionality on each role and avoiding the creation of a huge 'apache' role that would try to do everything driven by a great amount of parameters for choosing the appropriate templates and a spaghetti of conditional includes.

Ansible is not a programming language, but it already contains a powerful template language. Why not take full advantage of it for the benefit of a better role design?

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgwKB6HCf%2Bj_6P4X2h8ZYcgc6dyT45fj8%3DYvfH90bOM2kw%40mail.gmail.com.

[Michael DeHaan]

"That means when a role is being deployed because of a role dependency, its template tasks could use blocks possibly provided by any role that depends on it and is included in the active dependency chain."

The problem is 95% of our users including myself have trouble with this wording :)

We avoid things we can't explain, and that would confuse everyone when they read it.

It's worked pretty well as a guideline in making every language decision in this project so far :)

I would still maintain that there are some (usually mostly coming from Chef) that try to solve too many problems with role dependencies.  They were added to end a certain class of repeated question, but in general, they are overused and mostly *not* needed.

Role dependences are not an inheritance mechanism, they are a "run these roles before these roles" mechanism.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/53859C4F.8070205%40yahoo.gr.

[James Cammarata]

Absolutely. Role dependencies are more like package dependencies - if package A depends on package B, it means you need to install package B first. 

In no circumstances should package B include files that package A is going to overwrite. It would make things extremely confusing and impossible to implement, especially when you start talking about multiple levels of dependencies, and what should happen if there are multiple dependencies (package B and C require package A, but both override a file in A, who wins?).

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgxo04S-s8qdo_aOBeBmQJT8h4oTT-sBue-c7ecAi14N0g%40mail.gmail.com.

[Petros Moisiadis]

On 05/28/14 15:04, Michael DeHaan wrote:

"That means when a role is being deployed because of a role dependency, its template tasks could use blocks possibly provided by any role that depends on it and is included in the active dependency chain."

The problem is 95% of our users including myself have trouble with this wording :)

We avoid things we can't explain, and that would confuse everyone when they read it.

It's worked pretty well as a guideline in making every language decision in this project so far :)

I would still maintain that there are some (usually mostly coming from Chef) that try to solve too many problems with role dependencies.  They were added to end a certain class of repeated question, but in general, they are overused and mostly *not* needed.

Role dependences are not an inheritance mechanism, they are a "run these roles before these roles" mechanism.

I am not sure if that "95%" is valid... I don't think that template inheritance is rocket science. Thousands of users (e.g. including people in the great community of Django, which has a template language very similar to jinja2) are already using it for better design and reusability.

Please, think of this workflow:

- You define an empty block called 'extra' in a template file named "mytemplate.j2" belonging to role A.

- You write a role B that depends on role A. In the dependency statement you state that:
   a)  Role B depends on role A
   b)  Templates of role A can be overridden by templates provided by role B.

- You define a block named "extra" in a template file also named "mytemplate.j2", but belonging to role B. Obviously, you put in that block content that is specific for role B.

- When role B is applied, it first calls role A.

- Role A, when it is called by role B, renders "mytemplate.j2" with the "extra" block overridden with content provided by role B's "mytemplate.j2".


Do you really think that the above is something that normal users could not understand?

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgxo04S-s8qdo_aOBeBmQJT8h4oTT-sBue-c7ecAi14N0g%40mail.gmail.com.

[Petros Moisiadis]

On 05/28/14 15:13, James Cammarata wrote:

Absolutely. Role dependencies are more like package dependencies - if package A depends on package B, it means you need to install package B first. 

In no circumstances should package B include files that package A is going to overwrite. It would make things extremely confusing and impossible to implement, especially when you start talking about multiple levels of dependencies, and what should happen if there are multiple dependencies (package B and C require package A, but both override a file in A, who wins?).

I don't think your concerns are valid. The last role in the dependency chain should always win. This is how it already works in systems where template inheritance can be applied (e.g. Django). Also, I propose an _explicit_ directive to enable template inheritance in a role dependency statement (turned off by default), so people will always know what is going on.
 

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAMFyvFhOo_vsgRpbfg%3DuBJ_hhbeQ_Tnw0H9RZ%3Dp_OZvw_F%3DsPA%40mail.gmail.com.

[Michael DeHaan]

Ok so it's still not "inheritance" (this is an uneducated misuse of the term by Jinja, it's encapsulation, quite a different concept), ansible roles themselves are definitely not inheritance.

(I've also written a lot in Django)

Ultimately the problem is we have an irresolveable communication gap about how you are describing your technical idea. 

It seems to me you are saying when a template includes another template it should also look in the "template/" directories that belong to the dependent roles, so it can find things there, but this should only apply when a template includes another template.

Ultimately, I don't think this applies well to Ansible.

Ansible looks at role dependencies just like package managers look at package dependencies.  They are things that need to be applied first, but there is no concept of search paths or inheritance/encapsulation applied.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/5385E149.8010804%40yahoo.gr.

[Adam Morris]

For the specific example given: apache role and wsgi role there may be a simple, elegant, solution. Apache can include other configuration files, I don't know if it can include directories. If so include a directory. If not use the assemble module to create your included file.

Then the wsgi role just needs to drop a fragment or a complete file.

Template encapsulation seems like it could get complicated fast. Particularly if you have dependency chains... how far back should it go?

So is there another use case that we can discuss.

I'm thinking that they wish they'd called them prerequisites not dependencies now. :-)

Adam

[Petros Moisiadis]

On 05/28/14 16:21, Michael DeHaan wrote:

Ok so it's still not "inheritance" (this is an uneducated misuse of the term by Jinja, it's encapsulation, quite a different concept), ansible roles themselves are definitely not inheritance.

(I've also written a lot in Django)

Ultimately the problem is we have an irresolveable communication gap about how you are describing your technical idea. 

I used 'template inheritance' to have exact reference to a Jinja's feature that I propose to take advantage of in role dependencies. Sure, terminology could be wrong and that's why I am always using examples.

It seems to me you are saying when a template includes another template it should also look in the "template/" directories that belong to the dependent roles, so it can find things there, but this should only apply when a template includes another template.

Basically, yes. I am thinking of something that is similar to what is done with django's "template/" directories, but only when it is explicitly asked in the dependency statement and only if a role is triggered due to a dependency statement of that kind.

Ultimately, I don't think this applies well to Ansible.

Ansible looks at role dependencies just like package managers look at package dependencies.  They are things that need to be applied first, but there is no concept of search paths or inheritance/encapsulation applied.

Please, give it some extra thought. Think of what I have already said about better modularity in the design of roles. Think of a great potential to avoid complicating roles as variety in functionality increases. Now, people often end up in writing monolithic roles that are controlled by "choose your template" parameters and piled up include statements (think of the 'apache' role example). This could be done better.
Also, since I propose for an explicit directive in the dependency statement, current Ansible projects are not going to be affected and, without that directive being explicitly set, Ansible would still look at role dependencies the way it does now.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgzGpgMENkBgxG96C1CLR34AsMyZR%3D6J93EibXao-8gjtQ%40mail.gmail.com.

[Petros Moisiadis]

On 05/28/14 16:50, Adam Morris wrote:
> For the specific example given: apache role and wsgi role there may be a simple, elegant, solution. Apache can include other configuration files, I don't know if it can include directories. If so include a directory. If not use the assemble module to create your included file.
>
> Then the wsgi role just needs to drop a fragment or a complete file.
>
> Template encapsulation seems like it could get complicated fast. Particularly if you have dependency chains... how far back should it go?

It is true that software that has support for include directories can
help, but it become less helpful as you add more levels in ther hierarchy

For example, consider a tree like this:

'apache' role
|
+-------------------------+-------------------------+
| |
|
'wsgi_app' role 'php_app' role
'passenger_app'
| |
+-----------------------+
+------------------------+
| | |
|
'django_app' role 'flask_app' role 'redmine_app'
role 'gitlab_app' role


Deploying all or a subset of the above roles on the same server could be
most easily done if my proposal for template encapsulation was available.

> So is there another use case that we can discuss.

I have a 'common' role that deploys /etc/hosts. In 'cluster node' role
luster nodes I also add lines for all nodes in the cluster. Now I am
using a really ugly loop around lineinfile for this that took me some
time before I made it place the content properly. Try to use lineinfile
with a relatively complex data structure and you will remember my words.
Instead, it would be far more easy to write an "extra_hosts" block and
do things in pure jinja.

[Petros Moisiadis]

Oops, the line-wrapping algorithm destroyed my tree:
http://pastebin.com/SH5nTFEk

[Maciej Delmanowski]

Sorry, replying from phone.

28 maj 2014 16:59 "'Petros Moisiadis' via Ansible Project" <ansible...@googlegroups.com> napisał(a):
>
> On 05/28/14 16:50, Adam Morris wrote:
> > For the specific example given: apache role and wsgi role there may be a simple, elegant, solution.  Apache can include other configuration files, I don't know if it can include directories.  If so include a directory. If not use the assemble module to create your included file.
> >
> > Then the wsgi role just needs to drop a fragment or a complete file.
> >
> > Template encapsulation seems like it could get complicated fast. Particularly if you have dependency chains... how far back should it go?
>
> It is true that software that has support for include directories can
> help, but it become less helpful as you add more levels in ther hierarchy
>
> For example, consider a tree like this:
>
>                                        'apache' role
>                                              |
>                    +-------------------------+-------------------------+
>                    |                         |
> |
>              'wsgi_app' role           'php_app' role
> 'passenger_app'
>                    |                                                   |
>         +-----------------------+
> +------------------------+
>         |                       |                           |
>                   |
> 'django_app' role       'flask_app' role            'redmine_app'
> role       'gitlab_app' role
>
>
> Deploying all or a subset of the above roles on the same server could be
> most easily done if my proposal for template encapsulation was available.

Maybe most easily, but definitely not impossible. Currently I can deploy gitlab, gitlab-ci, phpmyadmin, phpipam, etherpad, owncloud, all on the same server behind nginx. I don't see how including templates in nginx role from other roles to make that possible could help me.

>
> > So is there another use case that we can discuss.
>
> I have a 'common' role that deploys /etc/hosts. In 'cluster node' role
> luster nodes I also add lines for all nodes in the cluster. Now I am
> using a really ugly loop around lineinfile for this that took me some
> time before I made it place the content properly. Try to use lineinfile
> with a relatively complex data structure and you will remember my words.
> Instead, it would be far more easy to write an "extra_hosts" block and
> do things in pure jinja.

Why not deploy for example dnsmasq? I mean, using /etc/hosts for DNS in a multiple host environment is so 1960...

>
> > I'm thinking that they wish they'd called them prerequisites not dependencies now. :-)

Seconded. :)

Maciej

> >
> > Adam

> >
>
> --
> You received this message because you are subscribed to the Google Groups "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
> To post to this group, send email to ansible...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/5385F9DB.9030903%40yahoo.gr.

[Petros Moisiadis]

On 05/28/14 18:17, Maciej Delmanowski wrote:

Sorry, replying from phone.

28 maj 2014 16:59 "'Petros Moisiadis' via Ansible Project" <ansible...@googlegroups.com> napisał(a):
>
> On 05/28/14 16:50, Adam Morris wrote:
> > For the specific example given: apache role and wsgi role there may be a simple, elegant, solution.  Apache can include other configuration files, I don't know if it can include directories.  If so include a directory. If not use the assemble module to create your included file.
> >
> > Then the wsgi role just needs to drop a fragment or a complete file.
> >
> > Template encapsulation seems like it could get complicated fast. Particularly if you have dependency chains... how far back should it go?
>
> It is true that software that has support for include directories can
> help, but it become less helpful as you add more levels in ther hierarchy
>
> For example, consider a tree like this:
>
>                                        'apache' role
>                                              |
>                    +-------------------------+-------------------------+
>                    |                         |
> |
>              'wsgi_app' role           'php_app' role
> 'passenger_app'
>                    |                                                   |
>         +-----------------------+
> +------------------------+
>         |                       |                           |
>                   |
> 'django_app' role       'flask_app' role            'redmine_app'
> role       'gitlab_app' role
>
>
> Deploying all or a subset of the above roles on the same server could be
> most easily done if my proposal for template encapsulation was available.

Maybe most easily, but definitely not impossible. Currently I can deploy gitlab, gitlab-ci, phpmyadmin, phpipam, etherpad, owncloud, all on the same server behind nginx. I don't see how including templates in nginx role from other roles to make that possible could help me.

I am sure that you can deploy all these and that you are happy with the result. However, I took a look at your nginx role and it seems you are doing exactly what I believe people could (and should) avoid with template encapsulation. It seems you are creating a "super-role" merging together what could be at least 3 separate roles connected through dependencies: nginx, fastcgi_app, php_app. If you prefer that monolithic way of doing things, it's fine. You will be one of those who will not be excited by a feature like the one I propose and probably never use it.

>
> > So is there another use case that we can discuss.
>
> I have a 'common' role that deploys /etc/hosts. In 'cluster node' role
> luster nodes I also add lines for all nodes in the cluster. Now I am
> using a really ugly loop around lineinfile for this that took me some
> time before I made it place the content properly. Try to use lineinfile
> with a relatively complex data structure and you will remember my words.
> Instead, it would be far more easy to write an "extra_hosts" block and
> do things in pure jinja.

Why not deploy for example dnsmasq? I mean, using /etc/hosts for DNS in a multiple host environment is so 1960...

Because it would be overkill for a 2 or 3 - node cluster. But you are missing the point. dnsmasq would still have a common configuration on most hosts, as well as an extra configuration needed for cluster nodes.

>
> > I'm thinking that they wish they'd called them prerequisites not dependencies now. :-)

Seconded. :)

Maciej

> >
> > Adam
> >
>
> --
> You received this message because you are subscribed to the Google Groups "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
> To post to this group, send email to ansible...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/5385F9DB.9030903%40yahoo.gr.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAEnKK1y7B3ecBT3KC%2B1pUQxKNv94krF2sQ1Ziboix3Rm5BnosA%40mail.gmail.com.

[Maciej Delmanowski]

2014-05-28 17:55 GMT+02:00 'Petros Moisiadis' via Ansible Project <ansible...@googlegroups.com>:

On 05/28/14 18:17, Maciej Delmanowski wrote:

> It is true that software that has support for include directories can

> help, but it become less helpful as you add more levels in ther hierarchy
>
> For example, consider a tree like this:
>
>                                        'apache' role
>                                              |
>                    +-------------------------+-------------------------+
>                    |                         |
> |
>              'wsgi_app' role           'php_app' role
> 'passenger_app'
>                    |                                                   |
>         +-----------------------+
> +------------------------+
>         |                       |                           |
>                   |
> 'django_app' role       'flask_app' role            'redmine_app'
> role       'gitlab_app' role
>
>
> Deploying all or a subset of the above roles on the same server could be
> most easily done if my proposal for template encapsulation was available.

Maybe most easily, but definitely not impossible. Currently I can deploy gitlab, gitlab-ci, phpmyadmin, phpipam, etherpad, owncloud, all on the same server behind nginx. I don't see how including templates in nginx role from other roles to make that possible could help me.

I am sure that you can deploy all these and that you are happy with the result. However, I took a look at your nginx role and it seems you are doing exactly what I believe people could (and should) avoid with template encapsulation. It seems you are creating a "super-role" merging together what could be at least 3 separate roles connected through dependencies: nginx, fastcgi_app, php_app. If you prefer that monolithic way of doing things, it's fine. You will be one of those who will not be excited by a feature like the one I propose and probably never use it.

Actually my playbook does exactly what you suggest - 'fastcgi_app' role (say, gitlab), uses 'nginx' role as dependency and passes its configuration via hash variable. Similarly, PHP5-based role (like ownCloud), uses 'php5' role as a dependency to create custom php5-fpm pool for itself, and 'nginx' role as another dependency similarly with its own custom configuration passed via hash variable.

Inside 'nginx' role, a server template generates server configuration for each "parent" role, puts it in its own file in /etc/nginx/sites-available/ and enables it by symlinking it to /etc/nginx/sites-available/.

The idea behind this was to abstract nginx configuration and management away from gitlab and owncloud roles, so that you could define for example your own SSL certificates or other server-wide configuration in one place. I treat Ansible roles which support dependencies like these like blackboxes (from the perspective of a role that uses them as dependency) with common interface - how nginx configures itself shouldn't matter for gitlab. That way I could replace nginx with apache2 (with similar interface prepared beforehand) and gitlab role should be none the wiser.

In fact, I stumbled on a similar problem you describe just recently, while planning WordPress integration  - you can see here http://codex.wordpress.org/Nginx that proper installation requires a lot more configuration in nginx that should be necessary to include in base nginx role. Because of that I'm thinking about creating a directory inside /etc/nginx/ where other roles could place their own nginx configuration (either via copy or template) and include it in their nginx server configuration - but not on Ansible level, but using nginx "include" option.

As you can hopefully see, I don't want to create a super-role for nginx, but instead I want to include just enough logic and room in that role so that other roles could use it comfortably. It might not be optimal or simple, as Michael would suggest, but it's a tradeoff between simplicity and customization which I feel will be beneficial for me in the long run - instead of creating specific nginx roles for many servers, I'm trying to create one general and reusable one.

> I have a 'common' role that deploys /etc/hosts. In 'cluster node' role
> luster nodes I also add lines for all nodes in the cluster. Now I am
> using a really ugly loop around lineinfile for this that took me some
> time before I made it place the content properly. Try to use lineinfile
> with a relatively complex data structure and you will remember my words.
> Instead, it would be far more easy to write an "extra_hosts" block and
> do things in pure jinja.

Why not deploy for example dnsmasq? I mean, using /etc/hosts for DNS in a multiple host environment is so 1960...

Because it would be overkill for a 2 or 3 - node cluster. But you are missing the point. dnsmasq would still have a common configuration on most hosts, as well as an extra configuration needed for cluster nodes.

Fair enough - I don't know what are your needs. I'm using dnsmasq on a laptop for 3-4 virtual machines in my development environment and I don't feel it's a waste of resources, but a convenience. :) I can define a CNAME or A in one place (in my inventory, in fact) and all hosts immediately know about it.

As for dnsmasq configuration on hosts other than the server (I hope you didn't plan to install it on all your servers), it's just one line in /etc/resolv.conf which is managed by operating system anyway, via DHCP. So, no extra configuration for me.

Maciej

[Petros Moisiadis]

On 05/28/14 20:55, Maciej Delmanowski wrote:

2014-05-28 17:55 GMT+02:00 'Petros Moisiadis' via Ansible Project <ansible...@googlegroups.com>:

On 05/28/14 18:17, Maciej Delmanowski wrote:

> It is true that software that has support for include directories can

> help, but it become less helpful as you add more levels in ther hierarchy
>
> For example, consider a tree like this:
>
>                                        'apache' role
>                                              |
>                    +-------------------------+-------------------------+
>                    |                         |
> |
>              'wsgi_app' role           'php_app' role
> 'passenger_app'
>                    |                                                   |
>         +-----------------------+
> +------------------------+
>         |                       |                           |
>                   |
> 'django_app' role       'flask_app' role            'redmine_app'
> role       'gitlab_app' role
>
>
> Deploying all or a subset of the above roles on the same server could be
> most easily done if my proposal for template encapsulation was available.

Maybe most easily, but definitely not impossible. Currently I can deploy gitlab, gitlab-ci, phpmyadmin, phpipam, etherpad, owncloud, all on the same server behind nginx. I don't see how including templates in nginx role from other roles to make that possible could help me.

I am sure that you can deploy all these and that you are happy with the result. However, I took a look at your nginx role and it seems you are doing exactly what I believe people could (and should) avoid with template encapsulation. It seems you are creating a "super-role" merging together what could be at least 3 separate roles connected through dependencies: nginx, fastcgi_app, php_app. If you prefer that monolithic way of doing things, it's fine. You will be one of those who will not be excited by a feature like the one I propose and probably never use it.

Actually my playbook does exactly what you suggest - 'fastcgi_app' role (say, gitlab), uses 'nginx' role as dependency and passes its configuration via hash variable. Similarly, PHP5-based role (like ownCloud), uses 'php5' role as a dependency to create custom php5-fpm pool for itself, and 'nginx' role as another dependency similarly with its own custom configuration passed via hash variable.

Inside 'nginx' role, a server template generates server configuration for each "parent" role, puts it in its own file in /etc/nginx/sites-available/ and enables it by symlinking it to /etc/nginx/sites-available/.

The idea behind this was to abstract nginx configuration and management away from gitlab and owncloud roles, so that you could define for example your own SSL certificates or other server-wide configuration in one place. I treat Ansible roles which support dependencies like these like blackboxes (from the perspective of a role that uses them as dependency) with common interface - how nginx configures itself shouldn't matter for gitlab. That way I could replace nginx with apache2 (with similar interface prepared beforehand) and gitlab role should be none the wiser.

But it seems you do have some specialized functionality embedded in your nginx role. For example you do fastcgi stuff. By fastcgi stuff I mean things such as configuration sections, structures and directives (e.g fastcgi_index, fastcgi_pass, etc.). I do not mean configuration data (such as {{ document_root }} or {{ fastcgi_script_name }}). If I could, I would abstract it further and put fastcgi stuff in a separate role that depends on nginx role. You would ask why to do that? Shouldn't the nginx know how to deploy fastcgi applications? Well, as you saw while planning for WordPress integration, sooner or later you will come across with a fastcgi (or php, or whatever) application A that needs different fastcgi configuration than the fastcgi configuration needed by a fastcgi application B. So you would need to abstract your fastcgi logic further. But this will also happen with your wsgi logic (which could also have its own universe with one app being deployed with modwsgi, another one with uwsgi, etc.), your php logic, your phusion passenger logic, etc. If you keep all this stuff in a single role, you will end up with a difficult to read and maintain project. So, I think splitting your deployment in multiple roles would make your life much easier. That said, I really appreciate the way you are thinking, as you do recognize the need for abstracting your basic roles, and I see that you have done what the current tools allow you to do for that purpose. It is just that Ansible does not (yet) give you a mechanism to modularize your roles in a cleaner way.

In fact, I stumbled on a similar problem you describe just recently, while planning WordPress integration  - you can see here http://codex.wordpress.org/Nginx that proper installation requires a lot more configuration in nginx that should be necessary to include in base nginx role. Because of that I'm thinking about creating a directory inside /etc/nginx/ where other roles could place their own nginx configuration (either via copy or template) and include it in their nginx server configuration - but not on Ansible level, but using nginx "include" option.

Since my suggestion for allowing template encapsulation between roles is not (yet) implemented, I would encourage you to go that way and use the nginx "include" option. However, I believe that jinja's "template inheritance" would be more flexible than the basic "include" mechanism provided by systems like nginx. For example, with jinja you could have a section with some desired content when the role would be deployed standalone, or override that section with some different (more specific) content when the role is deployed as part of a dependency. I am not sure if that can be done (easily and cleanly) with an ngix type of "include" mechanism. Also, keep in mind that there are systems that they do NOT provide such an "include" mechanism at all.

As you can hopefully see, I don't want to create a super-role for nginx, but instead I want to include just enough logic and room in that role so that other roles could use it comfortably. It might not be optimal or simple, as Michael would suggest, but it's a tradeoff between simplicity and customization which I feel will be beneficial for me in the long run - instead of creating specific nginx roles for many servers, I'm trying to create one general and reusable one.

> I have a 'common' role that deploys /etc/hosts. In 'cluster node' role
> luster nodes I also add lines for all nodes in the cluster. Now I am
> using a really ugly loop around lineinfile for this that took me some
> time before I made it place the content properly. Try to use lineinfile
> with a relatively complex data structure and you will remember my words.
> Instead, it would be far more easy to write an "extra_hosts" block and
> do things in pure jinja.

Why not deploy for example dnsmasq? I mean, using /etc/hosts for DNS in a multiple host environment is so 1960...

Because it would be overkill for a 2 or 3 - node cluster. But you are missing the point. dnsmasq would still have a common configuration on most hosts, as well as an extra configuration needed for cluster nodes.

Fair enough - I don't know what are your needs. I'm using dnsmasq on a laptop for 3-4 virtual machines in my development environment and I don't feel it's a waste of resources, but a convenience. :) I can define a CNAME or A in one place (in my inventory, in fact) and all hosts immediately know about it.

As for dnsmasq configuration on hosts other than the server (I hope you didn't plan to install it on all your servers), it's just one line in /etc/resolv.conf which is managed by operating system anyway, via DHCP. So, no extra configuration for me.

Maciej

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAEnKK1xpOWkaxH-3sMWUhUaNdQLf-aDSJMWHikFL3ZT0XCCzNg%40mail.gmail.com.

[Maciej Delmanowski]

2014-05-28 21:42 GMT+02:00 'Petros Moisiadis' via Ansible Project <ansible...@googlegroups.com>:

But it seems you do have some specialized functionality embedded in your nginx role. For example you do fastcgi stuff. By fastcgi stuff I mean things such as configuration sections, structures and directives (e.g fastcgi_index, fastcgi_pass, etc.). I do not mean configuration data (such as {{ document_root }} or {{ fastcgi_script_name }}). If I could, I would abstract it further and put fastcgi stuff in a separate role that depends on nginx role. You would ask why to do that? Shouldn't the nginx know how to deploy fastcgi applications? Well, as you saw while planning for WordPress integration, sooner or later you will come across with a fastcgi (or php, or whatever) application A that needs different fastcgi configuration than the fastcgi configuration needed by a fastcgi application B. So you would need to abstract your fastcgi logic further. But this will also happen with your wsgi logic (which could also have its own universe with one app being deployed with modwsgi, another one with uwsgi, etc.), your php logic, your phusion passenger logic, etc. If you keep all this stuff in a single role, you will end up with a difficult to read and maintain project. So, I think splitting your deployment in multiple roles would make your life much easier. That said, I really appreciate the way you are thinking, as you do recognize the need for abstracting your basic roles, and I see that you have done what the current tools allow you to do for that purpose. It is just that Ansible does not (yet) give you a mechanism to modularize your roles in a cleaner way.

I know that different applications or frameworks require different configuration. For this purpose, in my 'nginx' role I've included a way to select which server template is used to generate nginx configuration. Currently there's only one (https://github.com/ginas/ginas/blob/master/playbooks/roles/ginas.nginx/templates/etc/nginx/sites-available/default.conf.j2 which is selected by default or by setting item.type: 'default'), but that doesn't mean there couldn't be more - I plan to write separate one for wordpress, for example.

Yes, with time, number of these templates will grow. But hopefully I will be able to write them abstract enough that each one will be usable for many applications. And you can even combine different files via Jinja extension blocks - I currently don't use it in 'nginx', but you can see example in my 'apt' role which generates Debian Preseed configuration for normal and destructive install (https://github.com/ginas/ginas/tree/master/playbooks/roles/ginas.apt/templates/srv/www/sites/default/public/d-i/wheezy) - files need to be in the same directory, and AFAIK using different directories doesn't really work, but I haven't tested that extensively. I plan to use similar mechanism in the future to build nginx server configurations from parts (in fact my 'type' variable was planned with this scenario in mind).

Other way around this would be to split nginx role into base and templates and use them somehow via dependency. Problem with that is current nginx configuration structure - you would probably want to create separate directory for templating role, like /etc/nginx-templates/ and keep your generated configuration there to not interfere with base nginx config files - otherwise you could end up with chicken and egg problem where one role requires /etc/nginx derectory which is created by a different role. Of course Ansible could mitigate this with file: module, careful permissions and still keeping your generated templates in non-standard directories inside /etc/nginx/... It's a tricky question to answer, but I still feel that adding separate mechanism in Ansible dependency functions to support jinja extension blocks is a stretch and doesn't look too intuitive.

Maciej 

[ra...@future500.nl]

Hi All,

The "inheritance" discussion keeps popping up, appearantly there's value in there somewhere.. I'd like to add my 2ct, but just with regards to using galaxy roles.

First: it's obviously not "inheritance", I'd like to say "override" (like variables override eachother according to precedence), and limit the scope only to files/templates.

The case is, I can find a prefect role in Galaxy, with one single problem:  I'd like to use my own template instead of the included template. Currently, I cannot use the role, I must duplicate the entire thing and replace the template.

It would be nice if there was a way to use my own template, in place of the role's template. The point has been made that this is already possible with a variable for the template name, but this would require other people to change their roles [which is absolutely an option, considering they're open source].

The way I would expect it to work is just adding the files in a roles_path that has precedence...

It was outlined here in more detail:

https://groups.google.com/d/msg/ansible-project/9xRyYIOdSro/aGF5L7F03PoJ

But kind of got buried in talk of multiple reversed depedency inheritance injection encaptulation.

Kind regards,

Ramon

[Adam Morris]

On Wednesday, May 28, 2014 12:42:52 PM UTC-7, Ernest0x wrote:

 dependency. I am not sure if that can be done (easily and cleanly) with an ngix type of "include" mechanism. Also, keep in mind that there are systems that they do NOT provide such an "include" mechanism at all.

And yet it appears that Apache and Nginx the two examples raised so far do both include an "include" mechanism.  For those that don't it is always (and I don't think that has been emphasised enough yet) possible to use Assemble to add together multiple configuration fragments created by different roles...  

e.g. My main widget role puts down some basic configuration pieces, my widget instance role adds some more bits to the same directory, my gadget role adds a few more bits, and finally a configure role pulls all of the pieces together into one configuration file.  If something changes it's piece the only parts that would run would be the relevant role and the configure role to pull all of the new bits together.  If you don't like the idea of having a separate role then you make sure that the main widget role is called last and it does the assembly...  or you run the playbook twice (or more) until it settles down.

Adam

[Paul Oyston]

Just to confirm, I wasn't suggesting any form of inheritance in any way, it's just that quite often when you want to include dependencies then you may want to pass a template to that particular role. In many circumstances (such as nginx) it's not possible to provide this via yaml configuration simply because of the complexity of that configuration. 

I completely agree with the way that dependencies currently work, I just think there's a better way to pass information to these dependencies so that you can represent data in ways other than yaml. At the moment as I say it is possible, it's just that you have to provide the absolute path to the template.

The way that I saw it working was that the role has it's dependencies, and the path of the role that defined those dependencies is added to the normal template path.

To give a completely practical example say you have a graphite installation, and you want graphite to sit behind nginx. So you create an example.graphite role that has a meta file that lists the dependency of example.nginx. Now you want to provide a conf file for that graphite installation, now lets say that you've got a few conditionals in your config file, there's no real clean way to provide this information via yaml, so you basically want to create a template file for your example.graphite installation. The logical place for this is obviously the example.graphite/templates directory. But in your example.nginx dependency you need to accept in the template file, at the moment the only way is to provide the absolute path to this file.

What may benefit re-usability is just to accept a string that represents a file within the example.graphite/templates directory, if that doesn't exist fall back to example.nginx/templates and if that doesn't exist, fail.

This also has a lot of benefits for galaxy role re-usability, because evidently one of the issues with Puppet's Forge is that a lot of these modules are un-usable because they don't accept the parameter in that you need to set so you end up copying it and then hacking with it to add in the option that you need. But if you just accept a template in then the user has full control over what they need to set and this issue with the lack of a configurable option disappears.

[Adam Morris]

On Thursday, May 29, 2014 1:52:16 AM UTC-7, Paul Oyston wrote:

Just to confirm, I wasn't suggesting any form of inheritance in any way,...

 

 I understand what you are saying, but unless the role that you run first (avoiding the d word) knows to expect a template from the later role then you have an issue.  If all roles were written to add fragments and assemble configuration files then they could work together (or for applications that have configuration directories that's already solved for you).  But then you are either going to have to go in and retrofit all existing roles or only work with a subset.  There is no enforced coding standard for roles so I think that we are going to have to live with the current situation.  

If you write your roles to be accepting of fragments then that's the best that you can do...

Adam

[Paul Oyston]

I'm not at all saying that the role HAS to know about the template path, I'm saying that during the passing of the parameters of that role you could pass in a string that represents a file and that file could either exist in the role that has the dependency declaration or the role that you're calling. It's a simple change to the template path when you're calling roles within the dependency list. It could allow you to override templates that are present in the dependant roles without having to copy out, or change a potentially third party role. This also has the benefit in that you don't change templates or files within roles, and can potentially update them if they're from a third party without causing any conflicts or storing your own template files within that role.

To me it seems the role should be responsible for it's own configuration files and setting their parameters rather than dropping files into directories and calling handlers externally, but perhaps that's just my perception coming from other systems?