Running ansible in an unrelated virtualenv

[Joost Cassee]

Hi,

A Python project of mine uses virtualenv to isolate packages. I want to use Ansible to provision a Vagrant box for testing. However, when I run Vagrant it fails with this error:

[default] Running provisioner: ansible...

  File "/usr/bin/ansible-playbook", line 24, in <module>

    import ansible.playbook

ImportError: No module named ansible.playbook

If I deactivate the virtualenv it runs fine. I am now running "(deactivate; vagrant up)" but that is not ideal. Anyone know of a better solution?

Regards,

Joost

[C. Morgan Hamill]

Excerpts from Joost Cassee's message of 2013-07-16 07:20:12 -0400:

Sounds to me like you don't have ansible installed in the virtualenv.
You could either `pip install ansible` form within the virtualenv, or
create the virtualenv with access to site packages.
--
Morgan

[Joost Cassee]

2013/7/16 C. Morgan Hamill <cha...@wesleyan.edu>:

> Excerpts from Joost Cassee's message of 2013-07-16 07:20:12 -0400:

>> A Python project of mine uses virtualenv to isolate packages. I want to use
>> Ansible to provision a Vagrant box for testing. However, when I run Vagrant
>> it fails with this error:
>>
>> [default] Running provisioner: ansible...
>> File "/usr/bin/ansible-playbook", line 24, in <module>
>> import ansible.playbook
>> ImportError: No module named ansible.playbook
>>
>> If I deactivate the virtualenv it runs fine. I am now running "(deactivate;
>> vagrant up)" but that is not ideal. Anyone know of a better solution?
>
> Sounds to me like you don't have ansible installed in the virtualenv.
> You could either `pip install ansible` form within the virtualenv, or
> create the virtualenv with access to site packages.

Indeed, I installed Ansible system-wide. If I also install it in the
virtualenv, I get this error:

[default] Running provisioner: ansible...

PLAY [vagrant] ****************************************************************

GATHERING FACTS ***************************************************************
fatal: [127.0.0.1] => {'msg': "FAILED: (25, 'Inappropriate ioctl for
device')", 'failed': True}

Which is a much weirder error that I do not know how to debug any
further. It is an annoying property of virtualenv that it messes with
Python even for site-wide installed applications...

Regards,
Joost

--
Joost Cassee
http://joost.cassee.net

[Sebastian Bulzak]

Did you manage to fix this error? I don't have virtualenv, but I get the same failure during "GATHERING FACTS" before the start of any play task.

Regards,
S

[Sebastian Bulzak]

It's a private key issue on first connection. You have to call ansible with --private-key=~/.vagrant.d/insecure_private_key

[Joost Cassee]

2013/8/30 Sebastian Bulzak <s...@opus44.com>:

> It's a private key issue on first connection. You have to call ansible with
> --private-key=~/.vagrant.d/insecure_private_key

Adding the private key is something I forget once in a while too. But
I have seen the ioctl error a few times even when I specify the key.

I have had more luck with just adding ansible to my requirements.txt file.

[Joost Cassee]

Hi,

Sorry to drag this thread up again, but I now have difficulty running Ansible *in* a virtualenv... See issue 4083 [1].

My Python projects use virtualenv with to isolate the environment. I cannot run the system Ansible installation because tries to pick up packages from the virtualenv environment and I cannot use a local Ansible installation because it tries to pick up modules from the system locations.

I would really love Ansible to make a choice between being a regular Python package or being a system application. The latter for example by changing the hashbang in the main ansible script to the system Python executable (i.e. not /usr/bin/env python).

Regards,

Joost

[1] https://github.com/ansible/ansible/issues/4083

Regards,

Joost

Op dinsdag 16 juli 2013 17:38:31 UTC+2 schreef C. Morgan Hamill:

[Michael DeHaan]

Please read the web documentation on ansible_python_interpreter.

If you're having problems setting it, please share where you've actually set that information in your inventory, as well as your playbook, so we can tell what you are doing.

ansible_python_interpreter is a required mechanism because on some operating systems python may be in /usr/bin/python26 and relying on /usr/bin/env clearly doesn't work, and this allows you to manage such a system (say a modified Arch) and CentOS at the same time.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Michael DeHaan <mic...@ansibleworks.com>
CTO, AnsibleWorks, Inc.
http://www.ansibleworks.com/

[Joost Cassee]

Hi Michael,

The content of the testcase I attached to the ticket contains the following files:

hosts:

[group]

host  ansible_python_interpreter=bin/python

playbook.iml:

---

- hosts: host

  gather_facts: false

  tasks:

  - debug: msg="{{ ansible_python_interpreter }}"

  - local_action: ec2 image=X instance_type=X key_name=X

requirements.txt:

ansible

boto

I run these commands:

virtualenv .

. bin/activate

pip install -r requirements.txt

ansible -i hosts playbook.xml

Regards,

Joost

Op woensdag 11 september 2013 14:46:49 UTC+2 schreef Michael DeHaan:

[Cristobal Rosa]

Hi Joost,

I just checked what you said, and it seem to work properly:

(testansible)devel@devel:~/projects/virtualenvs/testansible$ ansible-playbook -i hostfile --connection=local playbook.yml 

PLAY [192.168.1.45] *********************************************************** 

TASK: [debug msg="{{ansible_python_interpreter}}"] **************************** 

ok: [192.168.1.45] => {"msg": "/home/devel/projects/virtualenvs/testansible/bin/python"}

TASK: [command echo "hi"] ***************************************************** 

changed: [192.168.1.45]

PLAY RECAP ******************************************************************** 

192.168.1.45               : ok=2    changed=1    unreachable=0    failed=0   

my hostfile:

(testansible)devel@devel:~/projects/virtualenvs/testansible$ cat hostfile 

xxx.xxx.xxx.xxx ansible_python_interpreter=/home/devel/projects/virtualenvs/testansible/bin/python

and the playbook:

(testansible)devel@devel:~/projects/virtualenvs/testansible$ cat playbook.yml 

---

- hosts: xxx.xxx.xxx.xxx

  gather_facts: false

  tasks:

    - debug: msg="{{ ansible_python_interpreter }}"

    - command: echo "hi" 

[Joost Cassee]

Hi Christobal,

Did you try to use a module that has an external dependency instead of the command module, like the ec2 one?

Regards,

Joost

Op woensdag 11 september 2013 16:58:06 UTC+2 schreef Cristobal Rosa:

[Cristobal Rosa]

Hi Joost,

sorry I couldn't check this case, I only tested it using the command module. 

BR,

Cristóbal 

[Michael DeHaan]

It will work the same for both.

[Joost Cassee]

What is happening is that I am confusing the code that is running the
playbook and the code that is running on the remote host. I see now
that it makes perfect sense to separate them.

The problem in my head is that some actions operate on remote systems,
but only make sense in a local_action, for example the ec2 modules. A
local_actions is still a connection to the local system, so it does
not run in the same environment as the playbook-running code.

So in my case I want to set ansible_python_interpreter for 127.0.0.1
to the Python executable from my virtual environment. I tried this
inventory file:

127.0.0.1 ansible_python_interpreter=VIRTUALENV/bin/python

[groups]
host

And that works! Thanks for your help, Cristobal and Michael!

Regards,
Joost


2013/9/12 Michael DeHaan <mic...@ansibleworks.com>:

> You received this message because you are subscribed to a topic in the

> Google Groups "Ansible Project" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/ansible-project/BT-tY3TKYlA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

> ansible-proje...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.



--

Joost Cassee
http://joost.cassee.net

[Tin Tvrtković]

Huh, wierd. If I'm understanding correctly, you have an issue on the master machine. You've installed Ansible globally, but when you have a virtualenv activated it doesn't work?

There are better ways than installing Ansible inside the virtualenv or creating a virtualenv with --system-site-packages (bad idea anyway). How exactly did you install Ansible globally?

If I misunderstood something, ignore. :) Also, as I understand it ansible_python_interpreter sets the remote Python interpreter, not the master one.

[Joost Cassee]

Hi Tin,

If I install Ansible only in a virtualenv (and not globally), the
modules are simply not found.

My problem was that local_action tasks are still "remote", delegated
to localhost. So the remote Python interpreter is used.

Regards,
Joost

2013/9/13 Tin Tvrtković <tinch...@gmail.com>:

[Michael DeHaan]

I would suggest just installing Ansible normally.

Most people use packages.

I think it's kind of a (no offense implied) symptom of Python developers to be like "This is Python, I know this!" and then make it more complicated :)

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

[Joost Cassee]

It's no problem, Morgan Hamill suggested in a reply to my original
post that installing Ansible in virtualenv was supported so that
prompted me to go down this path.

The real problem, as I see it, is that Ansible cannot run if you
happen to be inside of a virtualenv. That is unexpected for a
system-installed package. (A user should not have to understand what
language Ansible was written in.)

Regards,
Joost

2013/9/14 Michael DeHaan <mic...@ansibleworks.com>:

[Michael DeHaan]

"The real problem, as I see it, is that Ansible cannot run if you
happen to be inside of a virtualenv."

I'm not positive that's true (ansible_python_interpreter), I just believe trying is a very unnecessary complication.

"That is unexpected for a
system-installed package. (A user should not have to understand what
language Ansible was written in.)"

I think for a system installed package you'd be less likely to even look to see what language it was written in, ergo you would be unlikely to think to use virtualenv.

In other words, I think you mean "expected" vs "unexpected", or at least, that's the meaning I would take.

[Joost Cassee]

2013/9/14 Michael DeHaan <mic...@ansibleworks.com>:

> "The real problem, as I see it, is that Ansible cannot run if you
> happen to be inside of a virtualenv."
>
> I'm not positive that's true (ansible_python_interpreter), I just believe
> trying is a very unnecessary complication.

Well, Ansible does not run at all, because the hashbang on the main
ansible script says "#!/usr/bin/env python" and so it picks up the
interpreter of the virtualenv and cannot even find the ansible.runner
package.

> "That is unexpected for a
> system-installed package. (A user should not have to understand what
> language Ansible was written in.)"
>
> I think for a system installed package you'd be less likely to even look to
> see what language it was written in, ergo you would be unlikely to think to
> use virtualenv.

I agree that using virtualenv for Ansible "on purpose" is strange. But
it is perfectly reasonable to use Ansible as a tool for a project that
is developed in a virtualenv. Why would running ansible be any
different than running, say, grep?

[Michael DeHaan]

Why do you use grep in a virtualenv?

"virtualenv is a tool to create isolated Python environments."

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Michael DeHaan]

"Well, Ansible does not run at all, because the hashbang on the main

ansible script says "#!/usr/bin/env python" and so it picks up the
interpreter of the virtualenv and cannot even find the ansible.runner
package."

Note:  If I did say /usr/bin/python in the Ansible script, *then* I'd get grilled for not working inside a virtualenv and hard coding a system path.  We're totally not going there as that is a bad thing to do.

Ansible is taking the right track here by not hard coding the system path.

It seems if you want to install ansible python packages inside your virtualenv you could do that, or you could also just change the shebang line if it was causing any problems for you.

--Michael

[Joost Cassee]

2013/9/14 Michael DeHaan <mic...@ansibleworks.com>:

> Why do you use grep in a virtualenv?
>
> "virtualenv is a tool to create isolated Python environments."

I merely meant that when working on a Python project the grep command
comes in use. :-) And I would not expect it to fail just because I am
working in a virtualenv.

2013/9/14 Michael DeHaan <michael...@gmail.com>:

> "Well, Ansible does not run at all, because the hashbang on the main
> ansible script says "#!/usr/bin/env python" and so it picks up the
> interpreter of the virtualenv and cannot even find the ansible.runner
> package."
>
> Note: If I did say /usr/bin/python in the Ansible script, *then* I'd get
> grilled for not working inside a virtualenv and hard coding a system path.
> We're totally not going there as that is a bad thing to do.
>
> Ansible is taking the right track here by not hard coding the system path.

Sure, I agree. It's a dilemma. You need to work inside a virtualenv
for development.

One solution could be to use the entry_points feature of distribute
and make "pip install -e ." or "python setup.py install" set the path
to the Python interpreter. I'm not sure you would be up for that, but
it is kind of standard.

> It seems if you want to install ansible python packages inside your
> virtualenv you could do that, or you could also just change the shebang line
> if it was causing any problems for you.

Yes, I guess I'm back to my old solution: "( deactivate ;
ansible-playbook deploy.yml )"

Thanks for thinking along with me, I appreciate it.

Regards,
Joost

[Michael DeHaan]

No problem :) ... difficult balancing act!

Glad there is a solution, however involved.  

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Tin Tvrtković]

Hi again,

just for the record, which installation method is providing you with Ansible scripts with the /usr/bin/env python shebang? Because if you install it with pip, pip will fix your shebang and you won't have these issues. I just checked the EPEL repo, and their Ansible package also has a /usr/bin/python shebang fixed. Using a hard-coded Python executable instead of looking for one in the path (which is what the /usr/bin/env shebang does) will make it so you can run Ansible in any virtualenv.

Are you maybe using a git checkout and sourcing the hacking/env-setup script? Because, as I said, there are better ways.

[Michael DeHaan]

"I just checked the EPEL repo, and their Ansible package also has a /usr/bin/python shebang fixed."

That's curious.

The version in EPEL should mirror the upstream spec so there are no surprises:

https://github.com/ansible/ansible/blob/devel/packaging/rpm/ansible.spec

Perhaps this is something the EPEL build system is doing behind the scenes?

Perhaps Kevin Fenzi or someone can shine some light on this.

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Tin Tvrtković]

I opened up http://dl.fedoraproject.org/pub/epel/6/x86_64/ansible-1.2.3-2.el6.noarch.rpm and checked the ansible script. I'm not an expert on this, but having a concrete Python in the shebang here seems like the right choice for the RPM distribution. These packages aren't meant to be used in virtualenvs anyway. This way they'll work regardless of your virtualenv (or lack of).

[Michael DeHaan]

I do recall some flamewars on fedora-devel back in the day.

Really with these types of things you can't win, but /usr/bin/env is good for nearly everyone doing a checkout, especially in the age where some people have their own pythons.

It's really a question about what produces the least number of questions about it :)

Someday we can create a /usr/bin/figure-it-out that contains both behaviors and magically intuits what people want to do :)

[Kevin Fenzi]

On Sun, 15 Sep 2013 11:40:38 -0400
Michael DeHaan <mic...@ansibleworks.com> wrote:

> "I just checked the EPEL repo, and their Ansible package also has a
> /usr/bin/python shebang fixed."
>
> That's curious.
>
> The version in EPEL should mirror the upstream spec so there are no
> surprises:
>
> https://github.com/ansible/ansible/blob/devel/packaging/rpm/ansible.spec
>
> Perhaps this is something the EPEL build system is doing behind the
> scenes?
>
> Perhaps Kevin Fenzi or someone can shine some light on this.

The EPEL spec shouldn't be doing anything to scripts, it just calls
setup.py build and install.

(Here's the build output from the last epel build:
http://kojipkgs.fedoraproject.org//packages/ansible/1.3.0/1.el6/data/logs/noarch/build.log
)

kevin

[Joost Cassee]

Hi Tin,

The system version I use comes from the PPA at: ppa:rquillo/ansible
When I install Ansible using pypi-install I get a script that uses the
full /usr/bin/python hashbang. And my trouble go away... :-)

Count me firmly in the "use the full path to python" camp. The way I
see it, the ansible package depends on the python package, so it knows
where the Python interpreter lives and will have stored its packages
in that pythonpath.

Regards,
Joost

2013/9/15 Tin Tvrtković <tinch...@gmail.com>:

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Ansible Project" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/ansible-project/BT-tY3TKYlA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> ansible-proje...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Michael DeHaan]

I think we've had a long enough discussion on /usr/bin/env at this point as this is a very high traffic list.

It's going to stay as is, let's let this thread go for now.

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

[Enmanuel Cueva]

I had the same error when trying to put vagrant up and tun my ansible scripts:

GATHERING FACTS *************************************************************** 
fatal: [127.0.0.1] => {'msg': "FAILED: (25, 'Inappropriate ioctl for 
device')", 'failed': True} 

In my case the issue was that my ~/.ssh/config file had no entry for the new vagrant host VM, so I needed to add on that file the following entry

Host my-host-vm 192.168.111.11
Hostname 192.168.111.11
IdentityFile ~/.vagrant.d/insecure_private_key
User vagrant

El viernes, 30 de agosto de 2013 00:32:31 UTC+1, Sebastian Bulzak escribió:

It's a private key issue on first connection. You have to call ansible with --private-key=~/.vagrant.d/insecure_private_key

On Thursday, August 29, 2013 4:03:47 PM UTC-4, Sebastian Bulzak wrote:

Did you manage to fix this error? I don't have virtualenv, but I get the same failure during "GATHERING FACTS" before the start of any play task.

Regards,
S

On Tuesday, July 16, 2013 5:25:14 PM UTC-4, Joost Cassee wrote:

2013/7/16 C. Morgan Hamill <cha...@wesleyan.edu>:

> Excerpts from Joost Cassee's message of 2013-07-16 07:20:12 -0400:

>> A Python project of mine uses virtualenv to isolate packages. I want to use
>> Ansible to provision a Vagrant box for testing. However, when I run Vagrant
>> it fails with this error:
>>
>> [default] Running provisioner: ansible...
>>   File "/usr/bin/ansible-playbook", line 24, in <module>
>>     import ansible.playbook
>> ImportError: No module named ansible.playbook
>>
>> If I deactivate the virtualenv it runs fine. I am now running "(deactivate;
>> vagrant up)" but that is not ideal. Anyone know of a better solution?
>

> Sounds to me like you don't have ansible installed in the virtualenv.
> You could either `pip install ansible` form within the virtualenv, or
> create the virtualenv with access to site packages.

Indeed, I installed Ansible system-wide. If I also install it in the
virtualenv, I get this error:

[default] Running provisioner: ansible...

PLAY [vagrant] ****************************************************************

GATHERING FACTS ***************************************************************
fatal: [127.0.0.1] => {'msg': "FAILED: (25, 'Inappropriate ioctl for
device')", 'failed': True}

Which is a much weirder error that I do not know how to debug any
further. It is an annoying property of virtualenv that it messes with
Python even for site-wide installed applications...

Regards,
Joost