Jira (PUP-10375) Latest version ensured for python packages is incorrect

[Luchian Nemes (Jira)]

Luchian Nemes created an issue

Puppet / PUP-10375 Latest version ensured for python packages is incorrect Issue Type: Bug Assignee: Unassigned Created: 2020/03/19 8:27 AM Priority: Normal Reporter: Luchian Nemes In lib/puppet/provider/package/pip.rb pip version checks and python packages sortings/comparisons are prone to be often done incorrectly. One example would be for the 'numpy' package. Below can be seen some of this package's versions being sorted incorrectly: Correctly sorted Currently sorted 1.10.4 1.10.4 1.11.0b3 1.11.0 1.11.0rc1 1.11.0b3 1.11.0rc2 1.11.0rc1 1.11.0 1.11.0rc2 1.11.1rc1 1.11.1 1.11.1 1.11.1rc1 1.11.2rc1 1.11.2 1.11.2 1.11.2rc1   This impacts package installation, using pip, when trying to ensure the latest version for any python package.    Proposed solution: All pip version checks and python packages comparison/sortings should be done using Puppet::Util::Package::Version::Pip.compare instead of Puppet::Util::Package.versioncmp. This solution's implementation impact needs to be addressed as following: in case of invalid/unsupported versions, it shouldn't raise anything which might disrupt the rest of a manifest application/agent run would be a good idea to investigate pip's legacy versions (see pip's source code, check if this is still being used by packages/if such packages still exist and then, based on findings, maybe reconsider supporting them OR find a suitable solution for when puppet will come across one of them) some refactoring needs to be done for partially duplicated code: get all available versions for a package in only one method (at least per pip version particularities) by sorting the list of available versions and use it to find out the latest version and also use it when checking for a version range)     Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Luchian Nemes (Jira)]

Luchian Nemes updated an issue

Puppet / PUP-10375 Latest version ensured for python packages is incorrect

Change By: Luchian Nemes In [lib/puppet/provider/package/pip.rb|https://github.com/puppetlabs/puppet/blob/master/lib/puppet/provider/package/pip.rb] pip version checks and python packages sortings/comparisons are prone to be often done incorrectly. One example would be for the 'numpy' package. Below can be seen some of this package's versions being sorted incorrectly: ||{color:#00875a}Correctly sorted{color}||{color:#de350b}Currently sorted{color}|| |1.10.4|1.10.4| |1.11.0b3|{color:#de350b}1.11.0{color}| |1.11.0rc1|{color:#de350b}1.11.0b3{color}| |1.11.0rc2|{color:#de350b}1.11.0rc1{color}| |1.11.0|{color:#de350b}1.11.0rc2{color}| |1.11.1rc1|{color:#de350b}1.11.1{color}| |1.11.1|{color:#de350b}1.11.1rc1{color}| |1.11.2rc1|{color:#de350b}1.11.2{color}| |1.11.2| {color:#de350b} 1.11.2rc1 {color} |

This impacts package installation, using pip, when trying to ensure the latest version for any python package.

*{{Proposed solution:}}* All pip version checks and python packages comparison/sortings should be done using {color:#57d9a3}Puppet::Util::Package::Version::Pip.compare{color} instead of {color:#de350b}Puppet::Util::Package.versioncmp{color}.

This solution's implementation impact needs to be addressed as following:

* in case of invalid/unsupported versions, it shouldn't raise anything which might disrupt the rest of a manifest application/agent run * would be a good idea to investigate pip's legacy versions (see [pip's source code|https://github.com/pypa/pip/blob/master/src/pip/_vendor/packaging/version.py], check if this is still being used by packages/if such packages still exist and then, based on findings, maybe reconsider supporting them {color:#ff8b00}*OR*{color} find a suitable solution for when puppet will come across one of them) * some refactoring needs to be done for partially duplicated code: get all available versions for a package in only one method (at least per pip version particularities) by sorting the list of available versions and use it to find out the latest version and also use it when checking for a version range)

Add Comment

[Mihai Buzgau (Jira)]

Mihai Buzgau updated an issue

Puppet / PUP-10375 Latest version ensured for python packages is incorrect

Change By: Mihai Buzgau Story Points: 3

Add Comment

[Mihai Buzgau (Jira)]

Mihai Buzgau updated an issue

Puppet / PUP-10375 Latest version ensured for python packages is incorrect

Change By: Mihai Buzgau Sprint: PR NW - Triage 2020-04-15

Add Comment

[Luchian Nemes (Jira)]

Luchian Nemes assigned an issue to Luchian Nemes

Puppet / PUP-10375 Latest version ensured for python packages is incorrect

Change By: Luchian Nemes Assignee: Luchian Nemes

Add Comment

[Mihai Buzgau (Jira)]

Mihai Buzgau updated an issue

Puppet / PUP-10375 Latest version ensured for python packages is incorrect

Change By: Mihai Buzgau Sprint: NW - 2020-04-15 , NW - 2020-04-29

Add Comment

[Luchian Nemes (Jira)]

Luchian Nemes updated an issue

Puppet / PUP-10375 Latest version ensured for python packages is incorrect

Change By: Luchian Nemes Release Notes: Bug Fix Release Notes Summary: Ensuring latest package available using pip caused, in certain scenarios, lack of idempotency because puppet was seeing the wrong version as being latest. Comparing and sorting mechanism of versions was improved.

Add Comment

[Claire Cadman (Jira)]

Claire Cadman updated an issue

Puppet / PUP-10375 Latest version ensured for python packages is incorrect

Change By: Claire Cadman Labels: doc_reviewed

Add Comment