Jira (PUP-11681) Package provider "pip3" fails to target pip from virtual env on first run

[Dirk Heinrichs (Jira)]

Dirk Heinrichs created an issue

Puppet / PUP-11681 Package provider "pip3" fails to target pip from virtual env on first run Issue Type: Bug Affects Versions: PUP 7.20.0 Assignee: Unassigned Created: 2022/11/25 6:59 AM Environment: Linux Priority: High Reporter: Dirk Heinrichs Puppet Version: 7.20.0 Puppet Server Version: latest OS Name/Version: Debian 11 Given the following Puppet class:  # Install Python, and optionally configure different virtual # environments with a set of given modules# Parameters: #   - venvs: A list of virtual environments to be installed, #       together with a set of modules. It should have the following #       content: # python::venvs: #   - name: foo #     modules: #       - name: bar #         version: 1.2.3 #       - name: baz #         version: latest #   - name: no_modules #     modules: [] #   ... # # NOTE: Module version defaults to latest if not specifiedclass python (   venvs = [], ) {   $venv_base_path = '/opt/venvs'  package { 'python3':     ensure  => latest,   }  ['python3-pip', 'python3-venv'].each |$pkg| {     package { $pkg:       ensure  => latest,       require => Package['python3'],     }   }  file { 'venv_base':     path    => $venv_base_path,     ensure  => directory,   }  $venvs.each |$venv| {     $venv_path = "${venv_base_path}/${venv['name']}/bin"     $pip = "${venv_path}/pip"    exec { "setup_venv_${venv['name']}":       command => "python3 -m venv --upgrade ${venv_base_path}/${venv['name']}",       path    => $facts['path'],       unless  => "test -L ${venv_path}/python3",     }    $venv['modules'].each |$module| {       $mod_ver = if $module['version'] {         $module['version']       }       else {         latest       }      package { "mod_${venv['name']}_${module['name']}":         name     => $module['name'],         ensure   => $mod_ver,         provider => 'pip3',         command  => $pip,         require  => Exec["setup_venv_${venv['name']}"],       }     }   } } and the following input data:  python::venvs:   - name: foo     modules:       - name: cffi   - name: bar     modules:       - name: celery   - name: baz     modules:       - name: celery         version: 4.4.7  the pip3 package provider fails on first run. Desired Behavior: First run of the class should succeed w/o error. Actual Behavior: The class needs two runs to succeed because the first one errors out with  Error: Could not prefetch package provider 'pip3': Provider pip3 package command '/opt/venvs/bar/bin/pip' does not exist on this host The second run finally succeeds because this time '/opt/venvs/bar/bin/pip' exists. Add Comment

This message was sent by Atlassian Jira (v8.20.11#820011-sha1:0629dd8)

[Dirk Heinrichs (Jira)]

Dirk Heinrichs updated an issue

Puppet / PUP-11681 Package provider "pip3" fails to target pip from virtual env on first run

Change By: Dirk Heinrichs *Puppet Version:* 7.20.0 *Puppet Server Version:* latest *OS Name/Version:* Debian 11

Given the following Puppet class:

{code:java}

{code}

and the following input data:

{code:java}

python::venvs:   - name: foo     modules:       - name: cffi   - name: bar     modules:       - name: celery   - name: baz     modules:       - name: celery         version: 4.4.7

{code}

the pip3 package provider fails on first run.

*Desired Behavior:*

First run of the class should succeed w/o error.

*Actual Behavior:*

The class needs two runs to succeed because the first one errors out with

{code:java}

Error: Could not prefetch package provider 'pip3': Provider pip3 package command '/opt/venvs/bar/bin/pip' does not exist on this host

{code}

The second run finally succeeds because this time '/opt/venvs/bar/bin/pip' exists.

Add Comment

[Dirk Heinrichs (Jira)]

Dirk Heinrichs updated an issue

Puppet / PUP-11681 Package provider "pip3" fails to target pip from virtual env on first run Change By: Dirk Heinrichs

*Puppet Version:* 7.20.0 *Puppet Server Version:* latest *OS Name/Version:* Debian 11 Given the following Puppet class:

{code: java puppet }# Install Python, and optionally configure different virtual

# environments with a set of given modules

# Parameters: #   - venvs: A list of virtual environments to be installed, #       together with a set of modules. It should have the following #       content: # python::venvs: #   - name: foo #     modules: #       - name: bar #         version: 1.2.3 #       - name: baz #         version: latest #   - name: no_modules #     modules: [] #   ... #

# NOTE: Module version defaults to latest if not specifiedclass specified class python (

{code: java yaml }python::venvs:

- name: foo     modules:       - name: cffi   - name: bar     modules:       - name: celery   - name: baz     modules:       - name: celery         version: 4.4.7 {code}  the pip3 package provider fails on first run. *Desired Behavior:* First run of the class should succeed w/o error. *Actual Behavior:* The class needs two runs to succeed because the first one errors out with  {code:java}Error: Could not prefetch package provider 'pip3': Provider pip3 package command '/opt/venvs/bar/bin/pip' does not exist on this host {code} The second run finally succeeds because this time '/opt/venvs/bar/bin/pip' exists.

Add Comment

[Dirk Heinrichs (Jira)]

Dirk Heinrichs updated an issue

Puppet / PUP-11681 Package provider "pip3" fails to target pip from virtual env on first run Change By: Dirk Heinrichs

*Puppet Version:* 7.20.0 *Puppet Server Version:* latest *OS Name/Version:* Debian 11 Given the following Puppet class:

{code:puppet}# Install Python, and optionally configure different virtual

# environments with a set of given modules # Parameters: #   - venvs: A list of virtual environments to be installed, #       together with a set of modules. It should have the following #       content: # python::venvs: #   - name: foo #     modules: #       - name: bar #         version: 1.2.3 #       - name: baz #         version: latest #   - name: no_modules #     modules: [] #   ... #

# NOTE: Module version defaults to latest if not specified

{code:yaml}python::venvs:

- name: foo     modules:       - name: cffi   - name: bar     modules:       - name: celery   - name: baz     modules:       - name: celery         version: 4.4.7 {code}  the pip3 package provider fails on first run. *Desired Behavior:* First run of the class should succeed w/o error. *Actual Behavior:* The class needs two runs to succeed because the first one errors out with  {code:java}Error: Could not prefetch package provider 'pip3': Provider pip3 package command '/opt/venvs/bar/bin/pip' does not exist on this host {code} The second run finally succeeds because this time '/opt/venvs/bar/bin/pip' exists.

Add Comment

[Morgan Rhodes (Jira)]

Morgan Rhodes updated an issue

Puppet / PUP-11681 Package provider "pip3" fails to target pip from virtual env on first run

Change By: Morgan Rhodes Labels: needs-validation

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11681 Package provider "pip3" fails to target pip from virtual env on first run

Change By: Josh Cooper Priority: High Normal

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-11681

Re: Package provider "pip3" fails to target pip from virtual env on first run Dirk Heinrichs the problem is that the prefetch command for the pip3 provider executes before all of the virtual environments are created. So you will want to create all of the environments first before managing pip3 packages like: # setup all of the virtual environments$venvs.each |$venv| {

$venv_path = "${venv_base_path}/${venv['name']}/bin"

$pip = "${venv_path}/pip3"

exec { "setup_venv_${venv['name']}":       command => "python3 -m venv --upgrade ${venv_base_path}/${venv['name']}",       path    => $facts['path'],       unless  => "test -L ${venv_path}/python3",     }   }

# manage packages within each environment

$venvs.each |$venv| {     $venv_path = "${venv_base_path}/${venv['name']}/bin"

$pip = "${venv_path}/pip3"       $venv['modules'].each |$module| {       notice("module ${module['name']}")

$mod_ver = if $module['version'] {         $module['version']       }       else {         latest       }         package { "mod_${venv['name']}_${module['name']}":         name     => $module['name'],         ensure   => $mod_ver,         provider => 'pip3',         command  => $pip,         require  => Exec["setup_venv_${venv['name']}"],       }     }   }

Also I think you want to specify the command as $pip = "${venv_path}/pip3" so that you don't mix the "pip" command with the "pip3" provider. I can't install cffi on my system, but if I change the hiera data to "celery", then the manifest applies successfully: ---

python::venvs: - name: foo modules:

- name: celery

- name: bar modules: - name: celery - name: baz modules: - name: celery version: 4.4.7

Debug: /Stage[main]/Python/Package[python3-pip]/require: require to Package[python3] Debug: /Stage[main]/Python/Package[python3-venv]/require: require to Package[python3] Debug: /Stage[main]/Python/Package[mod_foo_celery]/require: require to Exec[setup_venv_foo] Debug: /Stage[main]/Python/Package[mod_bar_celery]/require: require to Exec[setup_venv_bar] Debug: /Stage[main]/Python/Package[mod_baz_celery]/require: require to Exec[setup_venv_baz] Debug: Prefetching apt resources for package Debug: Executing '/usr/bin/dpkg-query -W --showformat '${Status} ${Package} ${Version}\n'' Debug: Executing: '/usr/bin/apt-mark showmanual' Debug: Executing: '/usr/bin/apt-cache policy python3' Debug: Executing: '/usr/bin/apt-cache policy python3-pip' Debug: Executing: '/usr/bin/apt-cache policy python3-venv' Notice: /Stage[main]/Python/File[venv_base]/ensure: created Debug: /Stage[main]/Python/File[venv_base]: The container Class[Python] will propagate my refresh event Debug: Exec[setup_venv_foo](provider=posix): Executing check 'test -L /opt/venvs/foo/bin/python3' Debug: Executing: 'test -L /opt/venvs/foo/bin/python3' Debug: Exec[setup_venv_foo](provider=posix): Executing 'python3 -m venv --upgrade /opt/venvs/foo' Debug: Executing: 'python3 -m venv --upgrade /opt/venvs/foo' Notice: /Stage[main]/Python/Exec[setup_venv_foo]/returns: executed successfully Debug: /Stage[main]/Python/Exec[setup_venv_foo]: The container Class[Python] will propagate my refresh event Debug: Exec[setup_venv_bar](provider=posix): Executing check 'test -L /opt/venvs/bar/bin/python3' Debug: Executing: 'test -L /opt/venvs/bar/bin/python3' Debug: Exec[setup_venv_bar](provider=posix): Executing 'python3 -m venv --upgrade /opt/venvs/bar' Debug: Executing: 'python3 -m venv --upgrade /opt/venvs/bar' Notice: /Stage[main]/Python/Exec[setup_venv_bar]/returns: executed successfully Debug: /Stage[main]/Python/Exec[setup_venv_bar]: The container Class[Python] will propagate my refresh event Debug: Exec[setup_venv_baz](provider=posix): Executing check 'test -L /opt/venvs/baz/bin/python3' Debug: Executing: 'test -L /opt/venvs/baz/bin/python3' Debug: Exec[setup_venv_baz](provider=posix): Executing 'python3 -m venv --upgrade /opt/venvs/baz' Debug: Executing: 'python3 -m venv --upgrade /opt/venvs/baz' Notice: /Stage[main]/Python/Exec[setup_venv_baz]/returns: executed successfully Debug: /Stage[main]/Python/Exec[setup_venv_baz]: The container Class[Python] will propagate my refresh event Debug: Prefetching pip3 resources for package Debug: Executing '/opt/venvs/foo/bin/pip3 --version' Debug: Executing '/opt/venvs/foo/bin/pip3 freeze --all' Debug: Executing '/opt/venvs/bar/bin/pip3 --version' Debug: Executing '/opt/venvs/bar/bin/pip3 freeze --all' Debug: Executing '/opt/venvs/baz/bin/pip3 --version' Debug: Executing '/opt/venvs/baz/bin/pip3 freeze --all' Debug: Executing '/opt/venvs/foo/bin/pip3 --version' Debug: Executing '/opt/venvs/foo/bin/pip3 freeze --all' Debug: Executing: '/opt/venvs/foo/bin/pip3 install -q --upgrade celery' Notice: /Stage[main]/Python/Package[mod_foo_celery]/ensure: created Debug: /Package[mod_foo_celery]: The container Class[Python] will propagate my refresh event Debug: Executing '/opt/venvs/bar/bin/pip3 --version' Debug: Executing '/opt/venvs/bar/bin/pip3 freeze --all' Debug: Executing: '/opt/venvs/bar/bin/pip3 install -q --upgrade celery' Notice: /Stage[main]/Python/Package[mod_bar_celery]/ensure: created Debug: /Package[mod_bar_celery]: The container Class[Python] will propagate my refresh event Debug: Executing '/opt/venvs/baz/bin/pip3 --version' Debug: Executing '/opt/venvs/baz/bin/pip3 freeze --all' Debug: Executing: '/opt/venvs/baz/bin/pip3 install -q celery==4.4.7' Notice: /Stage[main]/Python/Package[mod_baz_celery]/ensure: created Debug: /Package[mod_baz_celery]: The container Class[Python] will propagate my refresh event

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-11681

Re: Package provider "pip3" fails to target pip from virtual env on first run

I don't think there's anything puppet could do differently about this, so I'm going to close.

Add Comment

[Dirk Heinrichs (Jira)]

Dirk Heinrichs commented on PUP-11681

Re: Package provider "pip3" fails to target pip from virtual env on first run

Josh Cooper , thanks a lot, that helped me get things working. For the record: I finally ended up with a defined type "python::venv" which sets up a single virtual environment together with its modules. The "exec" for creating the venv, as well as the "package" resources for each module, are now created as tagged virtual resources, which are finally realized using Exec <| tag == 'PythonVenvSetup' |> -> Package <| tag == 'PythonVenvModule' |> This allows me to create as much virtual environments as I want, even in a loop, in just one single agent run.

Add Comment