Install a pip with a fully activated virtualenv

[Will Platnick]

Hello,

I am unable to install a pip using a virtualenv. I believe the problem is that the virtualenv is not actually activating (by design?) and it's trying to create a file in /opt, which if the virtualenv is active, it would chroot within itself, rather than try to touch /opt. 

If I activate virtualenv as the depuser user, I can pip install newrelic-plugin-agent with no problem and additionally, I can even run the command it tries to run with no problem. 

What is the recommended way to install a pip that requires that a virtualenv be activated?

/home/{{ salt['pillar.get']('PYTHON_USER', 'depuser') }}/newrelic_agent:

  virtualenv.managed:

    - no_site_packages: True

    - venv_bin: /opt/python{{ salt['pillar.get']('PYTHON:VERSION', '275')}}/bin/virtualenv

    - runas: {{ salt['pillar.get']('PYTHON_USER', 'depuser') }}

    - require:

      - user: {{ salt['pillar.get']('PYTHON_USER', 'depuser') }}

      - cmd: installvirtualenv

newrelic-plugin-agent:

  pip.installed:

    - user: {{ salt['pillar.get']('PYTHON_USER', 'depuser') }}

    - bin_env: /home/{{ salt['pillar.get']('PYTHON_USER', 'depuser') }}/newrelic_agent/bin/pip

    - require:

      - virtualenv: /home/{{ salt['pillar.get']('PYTHON_USER', 'depuser') }}/newrelic_agent

When this pip tries to install, it fails with:

running install_data

creating /opt/newrelic_plugin_agent

error: could not create '/opt/newrelic_plugin_agent': Permission denied

----------------------------------------

Command /home/depuser/newrelic_agent/bin/python -c "import setuptools;__file__='/home/depuser/newrelic_agent/build/newrelic-plugin-agent/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-yPUGen-record/install-record.txt --single-version-externally-managed --install-headers /home/depuser/newrelic_agent/include/site/python2.7 failed with error code 1 in /home/depuser/newrelic_agent/build/newrelic-plugin-agent

Storing complete log in /home/depuser/.pip/pip.log

[Will Platnick]

I figured out the issue. I looked at the source of this pip, and it is doing:

base_path = '%s/opt/newrelic_plugin_agent' % os.getenv('VIRTUAL_ENV', '')

This fails because salt doesn't fully activate the virtualenv, so the environment variables aren't there.

I worked around this by writing a script that activates virtualenv and then does a pip install.

The question is, should the salt pip installation procedures have EVERYTHING that happens during a virtualenv activation, env variables and all?

[Nick Davis]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Actually, I think the problem is your bin_env statements. You should
give the path to the virtual env't directory containing 'bin/pip'. Do
not give the pull path that includes bin/pip. This is something that
tripped me up when I first started writing these states.

So instead of this:
- - bin_env: /home/{{ salt['pillar.get']('PYTHON_USER', 'depuser')
}}/newrelic_agent/bin/pip

Try this:
- - bin_env: /home/{{ salt['pillar.get']('PYTHON_USER', 'depuser')
}}/newrelic_agent


As an example, here is how I install packages in to a virtual envt:

In pillar I have these:

virt_env_dir: /opt/msvirtualenv
ms_root: /vagrant


Here are states:

virtualenv/init.sls

include:
{% if grains['os'] == 'CentOS' %}
- python27
{% endif %}
- baseline

ms-virtualenv:
virtualenv.managed:
- name: {{ pillar['virt_env_dir'] }}
- distribute: True
{% if grains['os'] == 'CentOS' %}
- venv_bin: /usr/local/bin/virtualenv-2.7
- python: /usr/local/bin/python2.7
- require:
- cmd: python-2.7-virtualenv
- cmd: baseline-dev-pkgs
{% elif grains['os'] == 'Ubuntu' %}
- require:
- cmd: baseline-dev-pkgs
{% endif %}


worker/init.sls

worker-pip-pkgs:
pip.installed:
- requirements: {{ pillar['ms_root']
}}/installation/scanworker/PACKAGES.pip
- bin_env: {{ pillar['virt_env_dir'] }}

celery-updated:
pip.installed:
- name: celery
- bin_env: {{ pillar['virt_env_dir'] }}
- upgrade: True
- require:
- pip: worker-pip-pkgs


I hope that helps,
Nick

On 08/22/2013 11:36 AM, Will Platnick wrote:
> I figured out the issue. I looked at the source of this pip, and it is
> doing:

> base_path='%s/opt/newrelic_plugin_agent'%os.getenv('VIRTUAL_ENV','')

> --
> You received this message because you are subscribed to the Google
> Groups "Salt-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to salt-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlIWROMACgkQYoNGId/piAuhPwCgiqYUDVjLIm53NFcB3VMj6H1z
Ya4An1Tp5VxxIF9LXkkFzixpJDVgnE9j
=rNio
-----END PGP SIGNATURE-----

[Will Platnick]

I've tried both ways, no difference.

[Jan Vlčinský]

Hi Will.

Just few notes (without testing your setup or diving into it).

no need to "activate". As long, as you install by means of pip, easy_install or python from your virtualenv, then there is no need to activate it. These programs have hardcoded environment in itself and install into it. It took me a while to understand, but it is true - doc is stating it and my internal test are proving it is true.

pip is installed with virtualenv (mostly, there are methods, how to force virtualenv to install without it, but you have to force it). At the same time, it is easy to upgrade pip to later version (and it will keep working well).

focus on "permission denied". What are attributes of your opt directory? Are you able manually to log into the user, who is running the pip, and create the directory?

With best regards

Jan