Ansible not working with remote Python virtual environments on Mac

[Mike Spitzer]

All actions with the pip module (other than creating the venv) apply to the user context and not the venv on a Mac. The details below are for the pip module. Running the commands when logged into the system works as expected and using those commands with both the command and shell modules results in the same outcome as the pip module. There's something different about the way it's being interpreted through Ansible. The same role works fine in Linux. Have tried this on 2 different Macs, both running Catalina.

$ ansible --version

ansible [core 2.11.6] config file = None configured module search path = ['/Users/spitzer/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /Users/spitzer/.virtualenvs/ansible460/lib/python3.8/site-packages/ansible ansible collection location = /Users/spitzer/.ansible/collections:/usr/share/ansible/collections executable location = /Users/spitzer/.virtualenvs/ansible460/bin/ansible python version = 3.8.12 (default, Sep 1 2021, 07:26:09) [Clang 11.0.3 (clang-1103.0.32.62)] jinja version = 3.0.1 libyaml = True

So, a minimal role to duplicate this is:

---

# Test pip role

- name: Create venvs root dir

  file: path: "{{ temp_dir }}/venvs"

  state: directory mode: "0755"

- name: pip install requests

  pip: name: requests

  state: latest

  virtualenv: "{{ temp_dir }}/venvs/test-env"

The pip install does create the venv where expected and all permissions look good, but during the install of 'requests' it gives me "Defaulting to user installation because normal site-packages is not writeable" and then installs it under user instead of the venv.

TASK [test_build : Create venvs dir] ******************************************************************************************************************************************

ok: [devops-mac] => {"changed": false, "gid": 20, "group": "staff", "mode": "0755", "owner": "spitzer", "path": "/Users/spitzer/ansible_tmp/venvs", "size": 96, "state": "directory", "uid": 501}

TASK [test_build : pip install requests] *******************************************************************************************************************************

ok: [devops-mac] => {"changed": false, "cmd": ["/Users/spitzer/ansible_tmp/venvs/test-env/bin/pip3", "install", "-U", "requests"], "name": ["requests"], "requirements": null, "state": "latest", "stderr": "  WARNING: The script normalizer is installed in '/Users/spitzer/Library/Python/3.8/bin' which is not on PATH.\n  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\n", "stderr_lines": ["  WARNING: The script normalizer is installed in '/Users/spitzer/Library/Python/3.8/bin' which is not on PATH.", "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location."], "stdout": "created virtual environment CPython3.8.2.final.0-64 in 398ms\n  creator CPython3macOsFramework(dest=/Users/spitzer/ansible_tmp/venvs/test-env, clear=False, no_vcs_ignore=False, global=False)\n  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/spitzer/Library/Application Support/virtualenv)\n    added seed packages: pip==21.3.1, setuptools==58.3.0, wheel==0.37.0\n  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator\nDefaulting to user installation because normal site-packages is not writeable\nRequirement already satisfied: requests in /Users/spitzer/Library/Python/3.8/lib/python/site-packages (2.26.0)\nCollecting idna<4,>=2.5\n  Using cached idna-3.3-py3-none-any.whl (61 kB)\nRequirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/spitzer/Library/Python/3.8/lib/python/site-packages (from requests) (1.26.7)\nCollecting charset-normalizer~=2.0.0\n  Using cached charset_normalizer-2.0.7-py3-none-any.whl (38 kB)\nCollecting certifi>=2017.4.17\n  Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)\nInstalling collected packages: idna, charset-normalizer, certifi\nSuccessfully installed certifi-2021.10.8 charset-normalizer-2.0.7 idna-3.3\n", "stdout_lines": ["created virtual environment CPython3.8.2.final.0-64 in 398ms", "  creator CPython3macOsFramework(dest=/Users/spitzer/ansible_tmp/venvs/test-env, clear=False, no_vcs_ignore=False, global=False)", "  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/spitzer/Library/Application Support/virtualenv)", "    added seed packages: pip==21.3.1, setuptools==58.3.0, wheel==0.37.0", "  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator", "Defaulting to user installation because normal site-packages is not writeable", "Requirement already satisfied: requests in /Users/spitzer/Library/Python/3.8/lib/python/site-packages (2.26.0)", "Collecting idna<4,>=2.5", "  Using cached idna-3.3-py3-none-any.whl (61 kB)", "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/spitzer/Library/Python/3.8/lib/python/site-packages (from requests) (1.26.7)", "Collecting charset-normalizer~=2.0.0", "  Using cached charset_normalizer-2.0.7-py3-none-any.whl (38 kB)", "Collecting certifi>=2017.4.17", "  Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)", "Installing collected packages: idna, charset-normalizer, certifi", "Successfully installed certifi-2021.10.8 charset-normalizer-2.0.7 idna-3.3"], "version": null, "virtualenv": "/Users/spitzer/ansible_tmp/venvs/test-env"}

On the same machine, doing the following by hand works:
$ cd ~/ansible_tmp
$ virtualenv test-env
$ ~/ansible_tmp/test-env/bin/pip3 install requests

Running those exact commands using the shell module still results in the user install. Have also tried with pip module, explicitly setting the pip3 binary with the same result.

Also, I completely removed the test-env directory before every test. Also, also, I've done this with the built-in Python 3.8 install, as well as the homebrew and macports 3.8 installs and forcing the correct python3 binary in the pip module. All had the same results.

Does anybody have any idea what I'm doing wrong here?

Thanks

-mS

[Dick Visser]

Drop the virtualenv command and try 'python3 -m venv' instead 

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/dd5862ed-9ed4-44f5-9edb-650c0590fbd9n%40googlegroups.com.

--

Sent from a mobile device - please excuse the brevity, spelling and punctuation.

[Mike Spitzer]

Sorry for the late response on this. Running the -m venv and then -m pip works. Still doesn't work with the built-in pip module, but at least this is a workaround.

Thanks