Ansigenome: A tool to help you gather information and manage your roles

[Nick Janetakis]

What do you guys think about https://github.com/nickjj/ansigenome. I released it yesterday and would appreciate some feedback.

Here's a snippet from the readme:

Ansigenome is a command line tool designed to help you manage your Ansible roles. It does 6 things:

scan a path containing Ansible roles and report back useful stats
rebuild a path of roles which auto generates a `README.md` and `meta/main.yml` from templates
run shell commands inside of each role's directory
init new roles with a travis-ci test already made for you
export a path of roles to a file to be consumed by `ansible-galaxy install -r`
dump a json file containing every stat it gathers from a directory

Also do you think something like this belongs as a core tool in ansible? I imagine the json dump could be useful for generating stats on the galaxy. I also feel like it's a good idea to push people into best practices for releasing ansible roles. It would be great if we can all agree on some standard and then stick to it. I tried to help with this situation by adding automatic travis tests to each new role you generate with Ansigenome. It also exposes a git branch in the meta file to hint that you should really be tagging your role versions.

[Michael DeHaan]

No offense intended, I saw this on twitter yesterday, and honestly, I didn't get it.

First off, ansible-galaxy is already a command line tool help manage (create, stub out, and download ansible roles).  ansible-galaxy is already open source as a command line tool, and it's not galaxy specific (just named as such).  We do welcome contributions, but would like to talk about syntax and usage first a bit.   Some of the things we've talked about was supporting private git repos.  Contributions have been slight, however.

Things like "run a shell command inside each role directory" seem really abstract to me, and I had difficulty following the documentation to understand why I would want to do the above things listed on the PyPi page.

I don't know what "rebuild a role" would mean.   Roles don't have to be built.

Travis integration doesn't seem to make sense because it's a very limited environment for running integration tests.

So, yeah, I'd rather have seen this as a discussion for what facilities are lacking in the /usr/bin/ansible-galaxy tool, and perhaps one of those might have been a "--list-roles" subcommand, that could use something like your stats output.

I'm still not understanding things like "run" or "rebuild".

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/75244ddb-4a71-4a3c-b872-ef9c37178386%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nick Janetakis]

Thanks for the feedback, I'll try to address some of your concerns:

Re: "run":
The stemmed from the ginas project. It's currently 50 roles inside of a single git repo. The author of ginas recently decided he's going to try and break up each role into stand alone git repos and release whatever roles he can on the galaxy. Now... he is a unix beast and could write a script to walk each directory, create a new git repo, add a remote origin to it, etc.. but it's annoying to write all of that stuff. Wouldn't it be cool to just type:

`ansigenome run -m 'git init && git remote add origin https://github.com/yourname/%role_name.git`

That's all you would need to write to make 50 git dirs and add a custom remote to each one. %role_name gets replaced with the current role in the loop.

Re: "rebuild":
Also he wanted to make a readme standard for all his roles. He wanted an easy way to adjust a readme template for all his roles at once. By providing just a tiny bit of information in your meta file it will create an interesting readme file for you based on the stats it found by inspecting your role.

Since last night I've added a bit more documentation on rebuilding:
https://github.com/nickjj/ansigenome#example-rebuild-output

You can see an example output there and how you would use it.

Re: travis:
There's a lot of low quality roles on the galaxy that aren't idempotent. The init command of ansigenome uses the standard ansible-galaxy init structure but it adds a travis test that runs your role twice and checks to make sure there are 0 changes. That's pretty useful to have for free. If you're not a unix wizard (I'm not) and fairly new to ansible then you might not be sure how to test ansible roles either. I just wanted to give people a stepping stone.

---

I tried to stick to the ansible-style way of making command line tools so that maybe merging certain things into ansible-galaxy would be easier. What was wrong with the syntax btw? I went for a stock pep8 setup.

Thanks again for ansible Michael.

[Michael DeHaan]

On Tue, Jul 22, 2014 at 8:56 AM, Nick Janetakis <nick.ja...@gmail.com> wrote:

Thanks for the feedback, I'll try to address some of your concerns:

Re: "run":
The stemmed from the ginas project. It's currently 50 roles inside of a single git repo. The author of ginas recently decided he's going to try and break up each role into stand alone git repos and release whatever roles he can on the galaxy. Now... he is a unix beast and could write a script to walk each directory, create a new git repo, add a remote origin to it, etc.. but it's annoying to write all of that stuff. Wouldn't it be cool to just type:

`ansigenome run -m 'git init && git remote add origin https://github.com/yourname/%role_name.git`

I'd probably go "find . | xargs" myself, maybe bash alias it.

For most internal projects, keeping one role per repo is really complicated.  It works great for galaxy because it allows for seperate versions and forking, and easier pull requests.

Also I may point out it appears you are using a new templating system :)

 

That's all you would need to write to make 50 git dirs and add a custom remote to each one. %role_name gets replaced with the current role in the loop.

Re: "rebuild":
Also he wanted to make a readme standard for all his roles. He wanted an easy way to adjust a readme template for all his roles at once. By providing just a tiny bit of information in your meta file it will create an interesting readme file for you based on the stats it found by inspecting your role.

This seems a bit specialized at first glance.   It is true that ansible-galaxy init does instantiate a readme template for a role, contributions to make that configurable would be welcome, I'm not sure "stats based" are the ideal way to go about it.

 

Since last night I've added a bit more documentation on rebuilding:
https://github.com/nickjj/ansigenome#example-rebuild-output

You can see an example output there and how you would use it.

Re: travis:
There's a lot of low quality roles on the galaxy that aren't idempotent. The init command of ansigenome uses the standard ansible-galaxy init structure but it adds a travis test that runs your role twice and checks to make sure there are 0 changes. That's pretty useful to have for free. If you're not a unix wizard (I'm not) and fairly new to ansible then you might not be sure how to test ansible roles either. I just wanted to give people a stepping stone.

What about roles that require authentication to external sources, or spin up new cloud instances?

This is why I'm saying public integration tests is not feasible for many roles.

See our testing guide at http://docs.ansible.com/test_strategies.html for some pointers on testing in general.   I don't think people would need to adopt an extra ecosystem tool in order to test things.

---

I tried to stick to the ansible-style way of making command line tools so that maybe merging certain things into ansible-galaxy would be easier. What was wrong with the syntax btw? I went for a stock pep8 setup.

Which syntax?

 

Thanks again for ansible Michael.

You are welcome.   I don't mean to be harsh above, but I do question when "ecosystem" tools instantly show up in PyPi without some attempt to discuss them first, or an attempt to build a stronger core.

We were built on "batteries included" to a great degree.   I agree we move slower now as we must be more conservative about making sure things are simple and robust than two years ago, and because we have thousands of people trying to throw code at Ansible all at once.   But I generally don't believe recreating the problems, of say, other ecosystems with metaphor-filled tool names of varying utility, are ultimately good in the long run, because someone feels they need to use everything.

There's a desire for minimalism, and making sure everything fits in.   Right down to making sure we use the same ways of replacing variables.   

I don't mean to be a control freak - this may very well serve a purpose, but I think the needs of it are not the needs of many, per se, because most large internal projects will have roles in all in one repo, and in many cases, Travis doesn't apply, etc.   

So some decisions made are very specific to a few very specific use cases.

That is not to say I think it should not exist.   I'm just questioning how to make something more generally accessible to everyone, and also how to continue to evolve those tools around a hub everybody can collaborate around.

Hope this makes sense.

-- 

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/1942657a-1067-4001-9d45-624b22a66485%40googlegroups.com.

[Michael DeHaan]

On Tue, Jul 22, 2014 at 9:09 AM, Michael DeHaan <mic...@ansible.com> wrote:

On Tue, Jul 22, 2014 at 8:56 AM, Nick Janetakis <nick.ja...@gmail.com> wrote:

Thanks for the feedback, I'll try to address some of your concerns:

Re: "run":
The stemmed from the ginas project. It's currently 50 roles inside of a single git repo. The author of ginas recently decided he's going to try and break up each role into stand alone git repos and release whatever roles he can on the galaxy. Now... he is a unix beast and could write a script to walk each directory, create a new git repo, add a remote origin to it, etc.. but it's annoying to write all of that stuff. Wouldn't it be cool to just type:

`ansigenome run -m 'git init && git remote add origin https://github.com/yourname/%role_name.git`

I'd probably go "find . | xargs" myself, maybe bash alias it.

For most internal projects, keeping one role per repo is really complicated.  It works great for galaxy because it allows for seperate versions and forking, and easier pull requests.

Also I may point out it appears you are using a new templating system :)

 

That's all you would need to write to make 50 git dirs and add a custom remote to each one. %role_name gets replaced with the current role in the loop.

Re: "rebuild":
Also he wanted to make a readme standard for all his roles. He wanted an easy way to adjust a readme template for all his roles at once. By providing just a tiny bit of information in your meta file it will create an interesting readme file for you based on the stats it found by inspecting your role.

This seems a bit specialized at first glance.   It is true that ansible-galaxy init does instantiate a readme template for a role, contributions to make that configurable would be welcome, I'm not sure "stats based" are the ideal way to go about it.

 

Since last night I've added a bit more documentation on rebuilding:
https://github.com/nickjj/ansigenome#example-rebuild-output

You can see an example output there and how you would use it.

Re: travis:
There's a lot of low quality roles on the galaxy that aren't idempotent.

Wanted to address this, BTW.   This is not /always/ the case.

Ansible is also designed to model a wide variety of actions.    "reboot" is a valid role.    "restart_caching_layer" could be a valid role.   "rotate_ssh_keys" could be one as well, as could even "erase_disks_to_government_standards".

Not all operations in ansible have to be idempotent.   We achieve this easily by having modules that (usually, in their most commonly used states) have declarative properties.

But one of the great powers of ansible is the ability to break out of that model, and use it for more than config management, and for things that actually are also stepwise.

Galaxy is naturally a commons, and content there can be of varying nature, but that's why the review system is in place.   I'd recommend people use this to comment on roles they like, as this helps folks find better content.

Ansible's niceness is that you can be declarative when you want, but it also lets you be imperative when you want.   (As such, with great power comes great responsibility... except when you are doing "cloud like cloud" and are just throwing images/instances away on upgrade, in which case, not so much).

[Nick Janetakis]

Also I may point out it appears you are using a new templating system :)

I'm not. I'm using jinja2 and the tool itself has no extra dependencies other than what ansible requires. The %foo syntax is only used on the command line so you as the scripter have access to the current role name in your script. It can't just read the folder name because it might be prefixed with your git username to conform to the galaxy's way of naming roles.

In the above example you might have your role at yourname/ansible-myrole on github but it's "yourname.myrole" on the galaxy. %role_name returns just "myrole" as a convenience.

It is true that ansible-galaxy init does instantiate a readme template for a role, contributions to make that configurable would be welcome, I'm not sure "stats based" are the ideal way to go about it.

I think there's a misunderstanding about what the rebuild command does because you looked at the docs before I had documentation written about it. It's not stats based really.

Look, the default ansible-galaxy readme template expects you to enter:

1. your default variables

This means you have your defaults listed in both your meta file AND your readme and it's annoying to keep both in sync by copy/pasting. Ansigenome solves this by just reading in your defaults from the meta file and dumps it under a defaults header in the readme. You don't have to configure anything to get this functionality but optionally you can overwrite the defaults output in the readme by changing your meta file. It also doesn't clutter your meta file with the automated readme contents either.

2. a list of dependencies

Yikes, here's another thing I need to copy/paste from my meta file without Ansigeome. Ansigenome solves this by just looking at your dependencies list in the meta file and outputs them cleanly in the readme without configuration.

3. example playbook

Ansigenome addresses this in a different way. By default it doesn't output anything like this to the readme since it's very custom and impossible to guess from just looking at your role. So instead all you have to do is goto the meta file and fill out the "inventory" field and it'll automatically create a quick start guide in your readme listing out whatever you wrote there.

4. requirements

Ansignome takes a peek at your platforms in the meta file and automatically builds a list of platforms for you in the readme. You don't need to copy/paste them into your readme anymore.

You guys clearly thought some of those things were annoying so you built that into the galaxy role page, like the platforms, etc.. The problem is I don't want to have to look at a git repo AND the galaxy to figure out what a role does or needs to work.

As a role user looking to use a public role I want to see:

1. What defaults are available?
2. What facts are set?
3. What platforms does it work on?
4. What dependencies do I need?
5. Is it idempotent?
6. Is it well tested and in working condition?

As an aside I also don't want to have to look at both a galaxy profile and a git repo. I want a single point of truth that's close to home (git repo).

As an open source roler build I want to:

1. Not have to copy/paste a ton of stuff into a readme file.
2. Easily be able to create unique content in my readme when needed.
3. Have as much documentation written automatically as possible.

Ansigenome addresses all of the above issues and more with the rebuild/init commands and does it in a way that requires no configuration at all from the user for both role consumers and authors.

What about roles that require authentication to external sources, or spin up new cloud instances? This is why I'm saying public integration tests is not feasible for many roles.

Ok, if you don't want an idempotency test added by default then just delete the .travis.yaml file in the role's directory. It also won't even run unless you specifically login to travis and turn the repo on. Having tests like this is a benefit to the community IMO and adding your own tests that ignore idempotency is super easy and standard to do. It shows how to use the role since you need to set it up on travis. It acts as fool proof documentation.

because most large internal projects will have roles in all in one repo, and in many cases, Travis doesn't apply, etc.

Then just don't use travis then, problem solved heh? Also you can have roles in 1 repo and benefit from Ansignome in the same way as having separate roles. It gathers stats on all roles it finds in a directory you give it. It doesn't care about any git directories. I used that run example of adding git directories as an example of the type of things you can do with the run commands. By no means do you need git dirs for each role. I want to make that super clear.

[Nick Janetakis]

Oops, in that last reply I meant you have your defaults listed in both your defaults/main.yml file and the readme. It also reads it from the defaults/main.yml file to generate the readme, or if you have something in the "defaults" field in the meta file it uses that instead.

[Michael DeHaan]

On Tue, Jul 22, 2014 at 10:06 AM, Nick Janetakis <nick.ja...@gmail.com> wrote:

Also I may point out it appears you are using a new templating system :)

I'm not. I'm using jinja2 and the tool itself has no extra dependencies other than what ansible requires. The %foo syntax is only used on the command line so you as the scripter have access to the current role name in your script. It can't just read the folder name because it might be prefixed with your git username to conform to the galaxy's way of naming roles.

Yep, I was referring to "%foo"

 

In the above example you might have your role at yourname/ansible-myrole on github but it's "yourname.myrole" on the galaxy. %role_name returns just "myrole" as a convenience.

It is true that ansible-galaxy init does instantiate a readme template for a role, contributions to make that configurable would be welcome, I'm not sure "stats based" are the ideal way to go about it.

I think there's a misunderstanding about what the rebuild command does because you looked at the docs before I had documentation written about it. It's not stats based really.

Look, the default ansible-galaxy readme template expects you to enter:

1. your default variables

This means you have your defaults listed in both your meta file AND your readme and it's annoying to keep both in sync by copy/pasting. Ansigenome solves this by just reading in your defaults from the meta file and dumps it under a defaults header in the readme. You don't have to configure anything to get this functionality but optionally you can overwrite the defaults output in the readme by changing your meta file. It also doesn't clutter your meta file with the automated readme contents either.

With regard to the default variable entry (which is so far unused in galaxy, and can be left blank), it seems it would be better to just improve the ansible-galaxy CLI in this place so the usage of two tools together was not required.

 

2. a list of dependencies

Yikes, here's another thing I need to copy/paste from my meta file without Ansigeome. Ansigenome solves this by just looking at your dependencies list in the meta file and outputs them cleanly in the readme without configuration.

The galaxy dependency field here is free form, and can include things like packages that are required.  It is not role specific.

 

3. example playbook

Ansigenome addresses this in a different way. By default it doesn't output anything like this to the readme since it's very custom and impossible to guess from just looking at your role. So instead all you have to do is goto the meta file and fill out the "inventory" field and it'll automatically create a quick start guide in your readme listing out whatever you wrote there.

Unclear of how inventory and example playbooks are related.

 

4. requirements

Ansignome takes a peek at your platforms in the meta file and automatically builds a list of platforms for you in the readme. You don't need to copy/paste them into your readme anymore.

You guys clearly thought some of those things were annoying so you built that into the galaxy role page, like the platforms, etc.. The problem is I don't want to have to look at a git repo AND the galaxy to figure out what a role does or needs to work.

I'd like to see an attempt at not bifurcating the tooling here.

 

As a role user looking to use a public role I want to see:

1. What defaults are available?
2. What facts are set?
3. What platforms does it work on?
4. What dependencies do I need?

So far the webpage handles all of this.

 

5. Is it idempotent?

This is not something a machine can really test.   It is something a machine can approximate.

 

6. Is it well tested and in working condition?

Even if you have a green light or not, it doesn't indicate the quality of said tests.   

Galaxy is in many cases just a starting point and set of examples, it's fully expected that people will need to modify roles from time to time to make use of them.   

 

What about roles that require authentication to external sources, or spin up new cloud instances? This is why I'm saying public integration tests is not feasible for many roles.

Ok, if you don't want an idempotency test added by default then just delete the .travis.yaml file in the role's directory. It also won't even run unless you specifically login to travis and turn the repo on. Having tests like this is a benefit to the community IMO and adding your own tests that ignore idempotency is super easy and standard to do. It shows how to use the role since you need to set it up on travis. It acts as fool proof documentation.

To be fair, tests aren't documentation either. 

[Nick Janetakis]

Unclear of how inventory and example playbooks are related.

In a lot of cases the playbook itself is always the same except for the name of the role. I have around 20 roles on the galaxy and pretty much all of them have an identical playbook in the example.

The thing that changes is overwriting or defining a few variables in your inventory. Like "here's the bare minimum to get rolling with the role", rather than just show someone 50 default variables up front. That is where the inventory name comes into play. I consider group_vars/etc. as part of my inventory.

I'd like to see an attempt at not bifurcating the tooling here.

I wanted to merge some of this into core if you guys felt it was reasonable but the ansible-galaxy script is untouchable IMO. It is already massive with a bunch of mixed concerns. I don't think it can sustain me dropping in 1k+ lines of python without it exploding and to keep it all broken up would have required coming up with my own directory structure which is fine for my project but I wasn't sure that would fly over well with ansible.

What parts do you like and don't like?
How much of it wouldn't ever get into core?
Would ansible-galaxy need to be renamed?

[Michael DeHaan]

On Tue, Jul 22, 2014 at 6:47 PM, Nick Janetakis <nick.ja...@gmail.com> wrote:

Unclear of how inventory and example playbooks are related.

In a lot of cases the playbook itself is always the same except for the name of the role. I have around 20 roles on the galaxy and pretty much all of them have an identical playbook in the example.

This is probably a sign they might need better examples :)

 

I'd like to see an attempt at not bifurcating the tooling here.

I wanted to merge some of this into core if you guys felt it was reasonable but the ansible-galaxy script is untouchable IMO. It is already massive with a bunch of mixed concerns.

Strongly disagree with this statement.  It's quite easy to add new functions, and it doesn't matter what functions or files things are in.  It's also easy enough to make a galaxy_utils/ subpackage in there too.

I 

 

I don't think it can sustain me dropping in 1k+ lines of python without it exploding and to keep it all broken up would have required coming up with my own directory structure which is fine for my project but I wasn't sure that would fly over well with ansible.

What parts do you like and don't like?
How much of it wouldn't ever get into core?

We are going to have editorial views about what Ansible should be.   All being said, we do appreicate and welcome attempts to contribute and work towards good combined tooling.  Your cases seem a bit niche for the mainstream, and that's fine, you can absolutely keep using it - but we seem to be at an impasse and should call it as such, I think.

 

Would ansible-galaxy need to be renamed?

It has no reason to be. 

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/2ed77b14-bf13-4151-9e5a-e83643f12639%40googlegroups.com.

[Nick Janetakis]

I try to set sane defaults. Almost all of my playbooks look like:

- name: Do something
  hosts: some_group
  sudo: true

  roles:
    - { role: my.role, tags: }

I have about 15 roles that work with that identical setup minus the name. What changes between them are a few inventory variables.

So I guess you're saying it's not worth merging anything then?

[Michael DeHaan]

Perhaps some of the scan/report stuff might be interesting.

I'm more interesting in seeing support for non-github git repos (git:// and ssh://) and a possibly some refinements in versioning role downloads (some people ask about multiple versions of roles and workflow, and things like cook-librarian type features) a bit first.

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/c5cc0067-9505-4f96-abe5-56d8ee199000%40googlegroups.com.