Determining what to retrieve from pypi and/or devpi-server

[Michael Felt aka aixtools]

Hi,

I am trying to determine where 'pip' decides what to download.

From my playing with devpi-server I expect it goes something like what the devpi(-client) list command does - to see what is available, and from that chooses something.

Does pip depend on the CPython get_platform(), or is there a "pip owned and managed" version that basically overrules the distutils (and sys.get_platform()) implementations?

Thanks.

[Chris Jerdonek]

On Fri, Aug 16, 2019 at 2:30 AM Michael Felt aka aixtools <aixt...@gmail.com> wrote:

From my playing with devpi-server I expect it goes something like what the devpi(-client) list command does - to see what is available, and from that chooses something.

Does pip depend on the CPython get_platform(), or is there a "pip owned and managed" version that basically overrules the distutils (and sys.get_platform()) implementations?

I'm sure others will chime in with more info. But quickly, on the second part, the function that pip uses to generate the list of compatible tags (for choosing among wheels) is the get_supported() function in pip's pep425tags.py: https://github.com/pypa/pip/blob/76ae377d7e38b5df8587d63b52464a08471eb1d1/src/pip/_internal/pep425tags.py#L279

The latest version of pip has a new "pip debug" command that displays, along with other info, the resulting list of tags for the environment under which the command is run (by calling that function).

--Chris 

 

Thanks.

--
You received this message because you are subscribed to the Google Groups "pypa-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pypa-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pypa-dev/826d4a1e-73a7-46cd-bed0-572b202ad4e6%40googlegroups.com.

[Chris Jerdonek]

On Fri, Aug 16, 2019 at 3:53 AM Chris Jerdonek <chris.j...@gmail.com> wrote:

On Fri, Aug 16, 2019 at 2:30 AM Michael Felt aka aixtools <aixt...@gmail.com> wrote:

From my playing with devpi-server I expect it goes something like what the devpi(-client) list command does - to see what is available, and from that chooses something.

Does pip depend on the CPython get_platform(), or is there a "pip owned and managed" version that basically overrules the distutils (and sys.get_platform()) implementations?

I'm sure others will chime in with more info. But quickly, on the second part, the function that pip uses to generate the list of compatible tags (for choosing among wheels) is the get_supported() function in pip's pep425tags.py: https://github.com/pypa/pip/blob/76ae377d7e38b5df8587d63b52464a08471eb1d1/src/pip/_internal/pep425tags.py#L279

The latest version of pip has a new "pip debug" command that displays, along with other info, the resulting list of tags for the environment under which the command is run (by calling that function).

Also, I believe the plan is for pep425tags.py to be replaced by this module (but it's not incorporated into pip yet):

https://github.com/pypa/packaging/blob/master/packaging/tags.py

--Chris

[Michael Felt]

On 16/08/2019 13:08, Chris Jerdonek wrote:
> On Fri, Aug 16, 2019 at 3:53 AM Chris Jerdonek <chris.j...@gmail.com>
> wrote:

Thanks for both replies. I'll dig further (later).

>>> <https://groups.google.com/d/msgid/pypa-dev/826d4a1e-73a7-46cd-bed0-572b202ad4e6%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>

[Michael Felt aka aixtools]

Thanks for the hints.

This is the "debug" from an unmodified pip:

root@x066:[/data/prj/python/git/pip-aix/src/pip]pip debug
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
pip version: pip 19.2.2 from /opt/lib/python3.7/site-packages/pip (python 3.7)
sys.version: 3.7.4 (default, Aug 14 2019, 12:40:12) [C]
sys.executable: /opt/bin/python3.7
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: iso8859-1
locale.getpreferredencoding: ISO8859-1
sys.platform: aix6
sys.implementation:
  name: cpython
Compatible tags: 20
  cp37-cp37m-aix_6_1
  cp37-abi3-aix_6_1
  cp37-none-aix_6_1
...

And, e.g., for cffi, it creates a wheel with this file name:

cffi-1.12.3-cp37-cp37m-aix_6_1.whl

In a modified pip I get this debug output:

pip version: pip 19.3.dev0 from /home/root/build/lib/python3.7/site-packages/pip-19.3.dev0-py3.7.egg/pip (python 3.7)
sys.version: 3.7.4 (default, Aug 14 2019, 12:40:12) [C]
sys.executable: /home/root/build/bin/python3
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: iso8859-1
locale.getpreferredencoding: ISO8859-1
sys.platform: aix6
sys.implementation:
  name: cpython
Compatible tags: 20
  cp37-cp37m-aix_6_1_7_1415_b32
  cp37-abi3-aix_6_1_7_1415_b32
  cp37-none-aix_6_1_7_1415_b32
  cp36-abi3-aix_6_1_7_1415_b32

...

Sadly, the name of the wheels it creates have not changed. I have not had any success (or luck) finding the logic that generates the filename for a new wheel.

Some pointers would be greatly appreciated.

Michael

On Friday, August 16, 2019 at 4:06:35 PM UTC+1, Michael Felt aka aixtools wrote:

On 16/08/2019 13:08, Chris Jerdonek wrote:
> On Fri, Aug 16, 2019 at 3:53 AM Chris Jerdonek

> wrote:
Thanks for both replies. I'll dig further (later).
>> On Fri, Aug 16, 2019 at 2:30 AM Michael Felt aka aixtools <

>>> email to pypa-dev+unsubscribe@googlegroups.com.

[Chris Jerdonek]

On Wed, Aug 21, 2019 at 3:36 PM Michael Felt aka aixtools <aixt...@gmail.com> wrote:

And, e.g., for cffi, it creates a wheel with this file name:

cffi-1.12.3-cp37-cp37m-aix_6_1.whl

...

Sadly, the name of the wheels it creates have not changed. I have not had any success (or luck) finding the logic that generates the filename for a new wheel.

How are you generating your wheels? You might want to read this:

https://pip.pypa.io/en/stable/user_guide/#installing-from-wheels

I'm not 100% sure, but the code for that might be here: https://github.com/pypa/wheel

That project also has a copy of pip's pep425tags.py file.

--Chris

 

>>> email to pypa-dev+u...@googlegroups.com.

>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/pypa-dev/826d4a1e-73a7-46cd-bed0-572b202ad4e6%40googlegroups.com
>>> <https://groups.google.com/d/msgid/pypa-dev/826d4a1e-73a7-46cd-bed0-572b202ad4e6%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>

--
You received this message because you are subscribed to the Google Groups "pypa-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pypa-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pypa-dev/66e12809-de41-4a41-ab9d-51c259987594%40googlegroups.com.

[Chris Jerdonek]

On Thu, Aug 22, 2019 at 9:23 AM Michael Felt <aixt...@gmail.com> wrote:

  ...

  creating build/bdist.aix-6.1/wheel/cffi-1.12.3.dist-info/WHEEL
  creating '/tmp/pip-wheel-u4xy3jit/cffi-1.12.3-cp37-cp37m-aix_6_1.whl'

It looks like this is where the wheel package creates the file name:
https://github.com/pypa/wheel/blob/b227ddd5beaba49294017d061d501f6d433393b0/wheel/bdist_wheel.py#L230-L231
(by calling the bdist_wheel.get_tag() method)

--Chris

[Michael Felt]

Many thanks. (and need to recall to reply to all)!

I hope I can find where it looks / but first I need to "make" one with
the correct tag info.

Looks like it may all be resolvable without involving a PR in CPython.

Michael

[Michael Felt]

So, it seems I'll need to modify get_platform in the pep425tags.py
file, also in wheel - BUT - I am unsure if

from .pep425tags import ....

is actually importing from wheel/pep425tags.py

Shall look further tomorrow. My changes do not look like they are
getting picked up.

FYI: what I also notice: when I rebuilt pip yesterday it gave itself a
"dev0" tag, while wheel did not.

[Michael Felt aka aixtools]

Ok - update on current status.

I have submitted three PR - one to pip - https://github.com/pypa/pip/issues/6922; one to wheel - https://github.com/pypa/wheel/issues/307, and per request in the pip issue, one to packaging - https://github.com/pypa/packaging/issues/188.

I have tested the PR's of pip and wheel using devpi-server (to see that a wheel can be downloaded from a server and installed - so again, thanks Chris for your pointers.

Sincerely,

Michael

On Thursday, August 22, 2019 at 10:49:31 PM UTC+2, Michael Felt aka aixtools wrote:

So, it seems I'll need to modify get_platform in the pep425tags.py
file, also in wheel - BUT - I am unsure if

from .pep425tags import ....

is actually importing from wheel/pep425tags.py

Shall look further tomorrow. My changes do not look like they are
getting picked up.

FYI: what I also notice: when I rebuilt pip yesterday it gave itself a
"dev0" tag, while wheel did not.

On Thu, Aug 22, 2019 at 8:58 PM Michael Felt <aixtools at gmail.com> wrote:
>
> Many thanks. (and need to recall to reply to all)!
>
> I hope I can find where it looks / but first I need to "make" one with
> the correct tag info.
>
> Looks like it may all be resolvable without involving a PR in CPython.
>
> Michael

> On Thu, Aug 22, 2019 at 8:37 PM Chris Jerdonek <chris.jerdonek at gmail.com> wrote:

[Michael Felt]

Just wanted to thank everyone for their replies and feedback.

I have a PR for CPython (https://bugs.python.org/issue38021 and
https://github.com/python/cpython/pull/17303) as well as
pypa/packaging - https://github.com/pypa/packaging/pull/190.

CPython is asking for feedback from pypa on this.

Please make sure aixtools @ felt . demon . nl is added to the recipients.

Sincerely,
Michael

> --
> You received this message because you are subscribed to the Google Groups "pypa-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pypa-dev+u...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/pypa-dev/451b5f7a-4d1e-480b-9397-5560900d5f4a%40googlegroups.com.

[Paul Moore]

I replied on the bpo issue.

> To view this discussion on the web visit https://groups.google.com/d/msgid/pypa-dev/CANvxniUyP6TrfsHnbaZfaE9-5gTAXtaDpLtfm4mcgSgiJCcHow%40mail.gmail.com.

[Michael Felt]

Thx for the reply. I cannot participate on googlegroups. FF seems to
be having issues:

Firefox detected a potential security threat and did not continue to
www.groups.google.com because this website requires a secure
connection.

What can you do about it?

www.groups.google.com has a security policy called HTTP Strict
Transport Security (HSTS), which means that Firefox can only connect
to it securely. You can’t add an exception to visit this site.

The issue is most likely with the website, and there is nothing you
can do to resolve it. You can notify the website’s administrator about
the problem.

I would have thought it would be okay - from this text also provided by FF:

Firefox does not trust this site because it uses a certificate that is
not valid for www.groups.google.com. The certificate is only valid for
the following names: *.google.com, *.android.com,
*.appengine.google.com,

So, back to real comment - Paul, where can I best answer, deeper, your
comment re:

"I thought that was something that had been discussed on the Pip
tracker, but maybe the implications weren't clear (I don't really
understand the AIX scheme, so I'm relying on someone else, probably
Michael, to propose something that could be added to PEP 425 and to
lead/guide the discussion)."

A discussion was started on pypa/pip, but it stalled, as it did in
pypa/wheel and never started in pypa/packaging. In other words, I have
been looking for the correct place to start.

In the meantime, I'll be working on ``something`` to be added -
wherever is deemed best, even if that be a new ( I hope short) PEP.

> To view this discussion on the web visit https://groups.google.com/d/msgid/pypa-dev/CACac1F_C6Z5XNQK5FqmGA4Xc1CahYu-d_YxDWMZMs-KTc_PAXw%40mail.gmail.com.

[Paul Moore]

On Fri, 22 Nov 2019 at 18:12, Michael Felt <aixt...@gmail.com> wrote:
> So, back to real comment - Paul, where can I best answer, deeper, your
> comment re:
>
> "I thought that was something that had been discussed on the Pip
> tracker, but maybe the implications weren't clear (I don't really
> understand the AIX scheme, so I'm relying on someone else, probably
> Michael, to propose something that could be added to PEP 425 and to
> lead/guide the discussion)."
>
> A discussion was started on pypa/pip, but it stalled, as it did in
> pypa/wheel and never started in pypa/packaging. In other words, I have
> been looking for the correct place to start.
>
> In the meantime, I'll be working on ``something`` to be added -
> wherever is deemed best, even if that be a new ( I hope short) PEP.

I'd strongly recommend trying to write up your proposed solution, in
the form of some sort of PEP or spec revision, if only to help to
express your proposal clearly and in a way that non-experts can
understand and review. At the moment, it feels very much as if you're
the only person qualified to judge your proposal, and no-one else
really has the time to invest or the inclination to research the
problem sufficiently to have an opinion. So we get stuck in the
situation of either having to take your word that your solution is the
best approach, or reject your ideas without giving them sufficient
consideration. (I should say that this is *not* a problem solely
because AIX is a niche platform - binary compatibility on *all*
platforms is a complex topic that most people have little
understanding of, myself included). The process of writing a PEP can
help a lot in refining and clarifying the explanation of the problem
and its solution.

It may well be that you could do with someone to help you through the
process of developing such a PEP - if for no other reason than the
fact that the PEP process itself requires that the author get a
sponsor who can assist in navigating the process (although a sponsor
would probably be helpful to you in any case). I'm sorry, but I'm
afraid I can't offer to do that for you, I simply don't have the spare
time at the moment.

Paul