Best practices for using Ansible with and without Vagrant

[Steve Moyer]

First a little background ...

I'm building my staging and production provisioning using Ansible and organized as suggested in the Ansible best practices section of the documentation.  My Ansible playbooks are kept in an SCM (Git) repository called infrastructure.

I recently started using Vagrant on my development workstation to allow me to easily create or recreate a clean environment (as a guest VM) on my workstation.  I'm storing the required Vagrantfile in SCM (Git) along with my project's files and I want to use Ansible to also provision the Vagrant environment.  Since much of the configuration of these guest VMs matches the production environment (that's the whole point right?), I want to use the roles (and associated tasks) that are used to provision the production and staging servers when provisioning the VMs.

So the problem I have is ...

The bulk of my provisioning is organized as recommended in the Infrastructure project's repository, but I need access to those files from each of the projects that spins up guest VM(s) on my development system.  Obviously I can check out the infrastructure project and refer to it from the other project (if I'm careful to check the projects out in the same directories), but that's not at all portable.  Another alternative is to check out the Infrastructure project as one of the tasks before the Vagrant specific provisioning happens, but that may lead to having multiple copies of the same repository on my HD (not a space problem) and I'm afraid I'd miss updating all of them if the infrastructure project has changes.

Is anyone else using Ansible for both their production provisioning and with Vagrant?  What are the best strategies for coping with playbooks/plays/tasks/etc that reside in different Git (or other SCM) repositories?

Thanks,

Steve

[Michael DeHaan]

You can easily use the same ansible tree with seperate inventory files, just pass in the "-i" flag in each case.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Michael DeHaan <mic...@ansibleworks.com>
CTO, AnsibleWorks, Inc.
http://www.ansibleworks.com/

[Steve Moyer]

Yeah ... and the inventory file for any of my projects is pretty short.

The real question is (perhaps) more related to managing two Git projects together.  I want a convenient way of checking out my  infrastructure project into the same workspace as my development project.

Steve

[Michael DeHaan]

"I want a convenient way of checking out my  infrastructure project into the same workspace as my development project."

I don't know what a workspace is.

[Mark Mandel]

This sounds more like an SCM issue than an Ansible issue, no?

Some thoughts:

1) Combine the two projects into one. TBH - not sure why you have them separate, it makes this sort of thing far harder, as you can see.

2) Git Submodules (http://git-scm.com/book/en/Git-Tools-Submodules). They can be a little tricky, but they really are designed for this sort of issue.

Mark

E: mark....@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

2 Devs from Down Under Podcast

http://www.2ddu.com/

[Ali Asad Lotia]

+1 Mark's advice. If your project is laid out appropriately submodules would be a very good fit.
Ali

[Jimmy Tang]

Hi All,

On Tue, Jun 4, 2013 at 12:21 AM, Mark Mandel <mark....@gmail.com> wrote:

This sounds more like an SCM issue than an Ansible issue, no?

Some thoughts:

1) Combine the two projects into one. TBH - not sure why you have them separate, it makes this sort of thing far harder, as you can see.

2) Git Submodules (http://git-scm.com/book/en/Git-Tools-Submodules). They can be a little tricky, but they really are designed for this sort of issue.

Mark

On Tue, Jun 4, 2013 at 8:20 AM, Michael DeHaan <michael...@gmail.com> wrote:

"I want a convenient way of checking out my  infrastructure project into the same workspace as my development project."

I don't know what a workspace is.

On Mon, Jun 3, 2013 at 6:19 PM, Steve Moyer <smoy...@gmail.com> wrote:

Yeah ... and the inventory file for any of my projects is pretty short.

The real question is (perhaps) more related to managing two Git projects together.  I want a convenient way of checking out my  infrastructure project into the same workspace as my development project.

Steve

On Monday, June 3, 2013 10:08:49 AM UTC-4, Michael DeHaan wrote:

You can easily use the same ansible tree with seperate inventory files, just pass in the "-i" flag in each case.

On Mon, Jun 3, 2013 at 9:44 AM, Steve Moyer <smoy...@gmail.com> wrote:

First a little background ...

I'm building my staging and production provisioning using Ansible and organized as suggested in the Ansible best practices section of the documentation.  My Ansible playbooks are kept in an SCM (Git) repository called infrastructure.

I recently started using Vagrant on my development workstation to allow me to easily create or recreate a clean environment (as a guest VM) on my workstation.  I'm storing the required Vagrantfile in SCM (Git) along with my project's files and I want to use Ansible to also provision the Vagrant environment.  Since much of the configuration of these guest VMs matches the production environment (that's the whole point right?), I want to use the roles (and associated tasks) that are used to provision the production and staging servers when provisioning the VMs.

So the problem I have is ...

The bulk of my provisioning is organized as recommended in the Infrastructure project's repository, but I need access to those files from each of the projects that spins up guest VM(s) on my development system.  Obviously I can check out the infrastructure project and refer to it from the other project (if I'm careful to check the projects out in the same directories), but that's not at all portable.  Another alternative is to check out the Infrastructure project as one of the tasks before the Vagrant specific provisioning happens, but that may lead to having multiple copies of the same repository on my HD (not a space problem) and I'm afraid I'd miss updating all of them if the infrastructure project has changes.

Is anyone else using Ansible for both their production provisioning and with Vagrant?  What are the best strategies for coping with playbooks/plays/tasks/etc that reside in different Git (or other SCM) repositories?

Thanks,

Steve

I've been using ansible + vagrant for testing/development of playbooks before pointing them at production systems -- see these two repos https://github.com/jcftang/ansible-hydra and https://github.com/jcftang/ansible-ceph 

I have a 'hacking' directory which sets up a few paths, configs and aliases so when i run 'ansible' or 'ansible-playbook' it looks up the self contained hosts file which points to my vagrant vm's. The layout that I have means that as soon as I migrate to using roles completely, I should be able to just turn my roles directory into a git submodule which I can reuse between my projects.

Jimmy

 

--

http://www.sgenomics.org/~jtang/

[Frans Kuipers]

What Mark said.

I have a setup according to http://ansible.cc/docs/bestpractices.html 

I have 2 hosts files in this directory:

# file: hosts
[webservers]
web-1.example.com
web-2.example.com

[monitoring]
mon-1.example.com

# file: vmhosts
[webservers]
webserver.vm

[monitoring]
monitoring.vm

In my Vagrantfile I have a vm for each group:

# file: Vagrantfile

Vagrant::Config.run do |config|

  config.vm.define :test do |vm_config|
    vm_config.vm.box = "precise32"
    vm_config.vm.host_name = "webserver.vm"
    vm_config.vm.network :hostonly, "192.168.100.10"

    vm_config.vm.provision :ansible do |ansible|
      ansible.playbook = "site.yml"
      ansible.hosts = "webservers"
      ansible.options = "-vvv", "--inventory-file=vmhosts"
    end
  end

end

You can optimize this by creating multiple Vagrantfile.[ansible_group] files if you have a lot of groups and don't want to create 100 vm's every time.

-- Frans

[Mark Mandel]

Just in case you guys weren't aware. Last week(?) I contributed an external inventory script for Vagrant:

https://github.com/ansible/ansible/blob/devel/plugins/inventory/vagrant.py

Handy if you don't want to set a static ip for your VM, and you can just have ansible look up what IP and port your vms are on.

Since we were on the topic, thought I would point it out.

Mark

[Lorin Hochstein]

Cool! I wonder if we could get something like this accepted upstream into the vagrant project (although they'd probably prefer it in ruby...), so you could do something like:

    config.vm.provision :ansible do |ansible|

      ansible.playbook = "./provision/your_playbook.yml"

      ansible.use_vagrant_inventory = true

      ansible.verbose = true

    end

And then the ansible-vagrant plugin would point it at the vagrant inventory plugin file that ships with vagrant.

Lorin Hochstein

Lead Architect - Cloud Services

Nimbis Services, Inc.

www.nimbisservices.com

[Mark Mandel]

That's a great idea!

I may have a dig around in the source, see how easy that would be to add.

Mark

[Gilles Cornu]

From this thread I notice that there is at the moment two "main" ways for Ansible+Vagrant users, which is quite confusing:

• original Ansible Plugin: https://github.com/dsander/vagrant-ansible (or related forks).
• Ansible provisioner is also part of Vagrant "core" since 1.2: https://github.com/mitchellh/vagrant/pull/1465. But this provisioner does not support (yet) parameters like `ansible.options` or `ansible.hosts`

Since Vagrant now integrates Ansible in its own production line, I am more in favor to develop this second (new) way (but I have no experience with the plugin, so I am not able to compare). Do you also see that both usages should soon or later converge? I guess that maybe the Ansible Plugin has more features/activity, and thus prevent to make the shift in a convenient way... 

About the `ansible.use_vagrant_inventory` idea: I also aim at in DRYing at maximum the collaboration between Vagrant and Ansible (at least for playbook development tasks). In Vagrant-1.2 the ansible provisioner, `ansible-playbook` is called for each VM to provision (I guess it is the same with dsander-like vagrant plugin, but I haven't verified yet). In multi-machine mode, that leads vagrant to run ansible-playbook several times against the whole inventory (workflow mismatch between "vagrant vm looping" and "ansible parallelism"). I thus opened https://github.com/mitchellh/vagrant/pull/1723, which meets my needs by generating on the fly a single-host inventory file for each VM to provision. It is quite handy for playbook development (so far, I don't plan to use vagrant for staging/prod site deployments). Of course this approach does not support [host groups].

If you have time and interest on improving Vagrant default ansible provisioner, I invite you to comment in https://groups.google.com/d/topic/ansible-project/qCa7Egg2Gvk/discussion and related pull requests. I'd happy to get some more confirmation/refutation on these points... and so ensure that ansible+vagrant "standard" toolchain converges soon, making ansible community less confused about these variants.

Gilles

[Michael DeHaan]

If you are using Vagrant, you should use the new provisioner.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Peter Petrov]

On Thursday, June 6, 2013 12:51:46 PM UTC+3, Gilles Cornu wrote:

About the `ansible.use_vagrant_inventory` idea: I also aim at in DRYing at maximum the collaboration between Vagrant and Ansible (at least for playbook development tasks). In Vagrant-1.2 the ansible provisioner, `ansible-playbook` is called for each VM to provision (I guess it is the same with dsander-like vagrant plugin, but I haven't verified yet). In multi-machine mode, that leads vagrant to run ansible-playbook several times against the whole inventory (workflow mismatch between "vagrant vm looping" and "ansible parallelism"). I thus opened https://github.com/mitchellh/vagrant/pull/1723, which meets my needs by generating on the fly a single-host inventory file for each VM to provision. It is quite handy for playbook development (so far, I don't plan to use vagrant for staging/prod site deployments). Of course this approach does not support [host groups].

For me it's also a problem that the Vagrant Ansible plugin calls the provisioner separately for each VM. I'd also like it to be run once after all VMs are up, to take advantage of Ansible's parallelism. I opened an issue about this in Vagrant's tracker:

https://github.com/mitchellh/vagrant/issues/1784

However, I don't think this issue should be mixed with the auto-generation of an inventory file by Vagrant (which is a separate feature request). For example, I use the inventory file to pass some variables, so an auto-generated one is not an option for me.

[Michael DeHaan]

Yeah I don't know, is there a vagrant list to coordinate that on instead?

This is really not a part of Ansible.

I understand many people like it, but I personally like to use virtual machine snapshots instead for testing, and avoid the extra layers of abstraction that (to me) make things require more hoops to jump through.

You can then just revert those VMs back to normal if you want to do a new clean test.

Tim Gerla was doing some very nice things with "vmrun" exactly for this purpose.

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Mark Mandel]

FYI - if anyone can think of a way that the vagrant external inventory plugin can better handle multi-machines in vagrant, I'm all ears. Config file maybe?

Mark

[Frans Kuipers]

On Thursday, June 6, 2013 11:51:46 AM UTC+2, Gilles Cornu wrote:

From this thread I notice that there is at the moment two "main" ways for Ansible+Vagrant users, which is quite confusing:

• original Ansible Plugin: https://github.com/dsander/vagrant-ansible (or related forks).
• Ansible provisioner is also part of Vagrant "core" since 1.2: https://github.com/mitchellh/vagrant/pull/1465. But this provisioner does not support (yet) parameters like `ansible.options` or `ansible.hosts`

`ansible.hosts`: is `ansible.limit` now in vagrant 1.2 (see http://docs.vagrantup.com/v2/provisioning/ansible.html #ADDITIONAL OPTIONS )

`ansible.options`: there is a pull request (https://github.com/mitchellh/vagrant/pull/1697) which adds`ansible.raw_arguments` 

this same pull request also adds `ansible.tags` and a better `ansible.verbose`

-- Frans

[Anatoly Mikhailov]

I'm totaly agree with you.

[Jeff Geerling]

Nice script, I hadn't seen that until now, but will likely start using it for a few of my setups.

Just wanted to chime in and say that for 99% of the infrastructure I manage, I just have on Git repo for both dev and production (and a few that have config for dev + test + production). I usually use separate inventory files for each environment, and for environment-specific variables, I do one of two things:

  1. Store the environment config in a separate application-specific repo and use a tool like Jenkins, Ant, Phing, etc. to deploy that vars file into the proper directory, prior to running the ansible playbook.

  2. Use host_vars and/or group_vars to separate out configuration within the main ansible repository.

Either method seems reasonable, depending on how you and your team deploys servers and apps, but I like the separated approach a little better, because the particular configurations are controlled by the application, whereas role defaults are usually sane defaults that environments can fall back to if they don't care about specific configurations.

[kesten broughton]

For this particular problem (calling on every vm) i use a trick i found online: only include the ansible provision call in the last vm in the Vagrantfile in is provider section.