[Python-Dev] It's now time to deprecate the stdlib urllib module

[Victor Stinner]

Hi,

I propose to deprecate the urllib module in Python 3.11. It would emit
a DeprecationWarning which warn users, so users should consider better
alternatives like urllib3 or httpx: well known modules, better
maintained, more secure, support HTTP/2 (httpx), etc.

I don't propose to schedule its removal. Let's discuss the removal in
1 or 2 years.

--

urllib has many abstraction to support a wide range of protocols with
"handlers": HTTP, HTTPS, FTP, "local file", proxy, HTTP
authentication, HTTP Cookie, etc. A simple HTTP request using Basic
Authentication requires 10-20 lines of code, whereas it should be a
single line.

Users (me included) don't like urllib API which was too complicated
for common tasks.

--

Unhappy users created multiple better alternatives to the stdlib urllib module.

In 2008, the "urllib3" module was created to provide an API designed
to be as simple as possible for the most common HTTP and HTTPS
requests. Example:

req = http.request('GET', 'http://httpbin.org/robots.txt').

In 2011, the "requests" module based on urllib3 was created.

In 2013, the "aiohttp" module based on asyncio was created.

In 2015, new "httpx" module was created:

req = httpx.get('https://www.example.org/')

Not only httpx has a regular "synchronous" API (blocking function
calls), but it also has an asynchronous API!

Sadly, while HTTP/3 is being developed, it seems like in this list,
httpx is the only HTTP client library supporting HTTP/2 currently :-(

For HTTP/2, I also found the "httplib2" module.

For HTTP/3, I found the "http3" and "aioquic" modules.

--

Let's come back to urllib:

* It's API is too complicated
* It doesn't support HTTP/2 nor HTTP/3
* It's barely maintained: there are 121 open issues including 3 security issues!

The 3 open security issues:

* bpo-33661 open 2018;
* bpo-36338 open in 2019;
* bpo-45795 open in 2021.

Usually, it's bad when you refer to an open security issue by its
creation year :-(

The urllib module has long history of security vulnerabilities. List
of *fixed* vulnerabilities:

* 2011 (bpo-11662):
https://python-security.readthedocs.io/vuln/urllib-redirect.html
* 2017 (bpo-30119):
https://python-security.readthedocs.io/vuln/urllib-ftp-stream-injection.html
* 2017 (bpo-30500):
https://python-security.readthedocs.io/vuln/urllib-connects-wrong-host.html
* 2019 (bpo-35907):
https://python-security.readthedocs.io/vuln/urllib-local-file-scheme.html
* 2019 (bpo-38826):
https://python-security.readthedocs.io/vuln/urllib-basic-auth-regex.html
* 2021 (bpo-42967):
https://python-security.readthedocs.io/vuln/urllib-query-string-semicolon-separator.html
* 2021 (bpo-43075):
https://python-security.readthedocs.io/vuln/urllib-basic-auth-regex2.html
* 2021 (bpo-44022):
https://python-security.readthedocs.io/vuln/urllib-100-continue-loop.html

urllib is a package made of 4 parts:

* urllib.request for opening and reading URLs
* urllib.error containing the exceptions raised by urllib.request
* urllib.parse for parsing URLs
* urllib.robotparser for parsing robots.txt files

I propose to deprecate all of them. Maybe the deprecation can be
different for each sub-module?

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/EZ6O7MOPZ4GA75MKTDO7LAELKXUHK2QS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Dong-hee Na]

I am not an expert about pip, 

but it will be not a problem about installing the pip module once CPython removes urllib module from stdlib?

Warm regards,

Dong-hee

2022년 2월 6일 (일) 오후 11:13, Victor Stinner <vsti...@python.org>님이 작성:

[Paul Moore]

Strong -1 from me.

urllib.request may not be "best practice", but it's still extremely
useful for simple situations, and urllib.parse is useful for basic
handling of URLs.Yes, the more complex aspects of urllib are better
handled by external packages, but that's not sufficient argument for
removing the package altogether. There are many situations where
external dependencies are unsuitable. Also, there's quite a lot of
usage of urllib in the stdlib itself - how would you propose to
replace that?

In addition, pip relies pretty heavily on urllib (parse and request),
and pip has a bootstrapping issue, so using 3rd party libraries is
non-trivial. Also, of pip's existing vendored dependencies,
webencodings, urllib3, requests, pkg_resources, packaging, html5lib,
distlib and cachecontrol all import urllib. So this would be *hugely*
disruptive to the whole packaging ecosystem (which is under-resourced
at the best of times, so this would put a lot of strain on us).

In any case, why is this being proposed as a simple posting on
python-dev? There's already PEP 594 for removals from the stdlib. If
you have a case for removing urllib, I suggest you get it added to PEP
594, so it can be discussed and agreed properly, along with the other
removals (none of which is remotely as controversial as urllib, so
there's absolutely no doubt in my mind that this would need a PEP
however it was proposed).

Paul

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/YQPV5MQXOCFZNWZMG22CZKQT33IHYBNP/

[Damian Shaw]

Speaking from anecdotal experience, "urllib.parse" is a very popular and highly depended on module, I would be shocked if removing it wouldn't be very disruptive.

In fact a quick search of the replacement modules you mention see that they all rely it on it, here is an example from each:

* requests: https://github.com/psf/requests/blob/99b3b492418d0751ca960178d274f89805095e4c/requests/sessions.py#L121

* aiohttp: https://github.com/aio-libs/aiohttp/blob/7d78fd01dbe983d119141d7f2775aefd42494f99/aiohttp/formdata.py#L129

* httpx: https://github.com/encode/httpx/blob/b7dc0c3df68279ce89f016a69a41b27a2346d54d/httpx/_content.py#L144

As for "urllib.request" I know that the philosophy of Python being a "batteries included language" is going away, but having no way to make any http call without importing Python definitely has a lot of situations where it makes Python more difficult to use. Could it not always emit a warning that this library should not be used in a production environment? Much in the same way that Flask's default web server does.

Damian (he/hm)

[Damian Shaw]

Pip vendors requests for network calls: https://github.com/pypa/pip/tree/main/src/pip/_vendor/requests

But still does depend on functions from urllib.parse and urllib.request in many places: https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/urls.py

Damian (he/him)

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/E6GN2THYCNQ2Q3CGMSH7GRCDFOOFDDCQ/

[Victor Stinner]

On Sun, Feb 6, 2022 at 3:48 PM Damian Shaw <damian.p...@gmail.com> wrote:
>
> Pip vendors requests for network calls: https://github.com/pypa/pip/tree/main/src/pip/_vendor/requests
>
> But still does depend on functions from urllib.parse and urllib.request in many places: https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/urls.py

Aha, it doesn't use urllib.request to open a HTTP connection, it only
uses pathname2url() and url2pathname() functions of urllib.request.
Maybe we can keep these functions. I'm not sure why they don't belong
to urllib.parse.

If urllib.parse is widely used, maybe we can keep this module.

Victor

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/PDFGPDGESBLSBHVLINCPAFEOHXQWFIRI/

[Senthil Kumaran]

On Sun, Feb 06, 2022 at 03:08:40PM +0100, Victor Stinner wrote:

> I propose to deprecate the urllib module in Python 3.11. It would emit
> a DeprecationWarning which warn users, so users should consider better
> alternatives like urllib3 or httpx: well known modules, better
> maintained, more secure, support HTTP/2 (httpx), etc.
>
> I don't propose to schedule its removal. Let's discuss the removal in
> 1 or 2 years.

I am not certain if we can deprecate/remove the whole 'urllib' module without any good plan for replacement
of its facilities within the stdlib. There is heavy usage of urllib.parse in multiple projects (including in urllib3),
and parse is semi-maintained.

> Let's come back to urllib:

> * It's API is too complicated
> * It doesn't support HTTP/2 nor HTTP/3
> * It's barely maintained: there are 121 open issues including 3 security issues!

I agree with all of these.
I think that removing the old cruft code, might lead to us to closing a number of open issues.

> The 3 open security issues:

Just because if something marked 'security' doesn't make it actionable too.
For instance the last one asks for urllib to maintain client state to be safe against a scenario, which it never did.

I don't think it is time to deprecate the urllib module. It will be too disruptive IMO. SO, -1.

Right now, I don't have a solution.
My suggestion will be we close old bugs, and remove old code (aka maintain a bit, and it falls on me too).
Then we can probably chart out a deprecation / replacement path in a non-disruptive manner.


--
Senthil

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/ORQEJXJTZDYYV53MHKXTJ3Q6W72AUSGA/

[Damian Shaw]

That was just one example, here are others in the pip code base that urllib.request is used for more than the pathname functions, they are all vendored or tests but would still be disruptive to remove:

https://github.com/pypa/pip/blob/main/tests/lib/local_repos.py

https://github.com/pypa/pip/blob/main/src/pip/_vendor/webencodings/mklabels.py

https://github.com/pypa/pip/blob/main/src/pip/_vendor/requests/compat.py

https://github.com/pypa/pip/blob/main/src/pip/_vendor/distlib/compat.py

In particular the vendored library, and replacement you suggest, "requests" is very dependent on the proxy functions such as "getproxies" that are currently in urllib.requests. More than once I've had to go down the rabbit hole of seeing where those functions get that info for each platform.

Damian (he/him)

[Christian Heimes]

On 06/02/2022 15.08, Victor Stinner wrote:
> Hi,
>
> I propose to deprecate the urllib module in Python 3.11. It would emit
> a DeprecationWarning which warn users, so users should consider better
> alternatives like urllib3 or httpx: well known modules, better
> maintained, more secure, support HTTP/2 (httpx), etc.
>
> I don't propose to schedule its removal. Let's discuss the removal in
> 1 or 2 years.
>
> --
>
> urllib has many abstraction to support a wide range of protocols with
> "handlers": HTTP, HTTPS, FTP, "local file", proxy, HTTP
> authentication, HTTP Cookie, etc. A simple HTTP request using Basic
> Authentication requires 10-20 lines of code, whereas it should be a
> single line.
>
> Users (me included) don't like urllib API which was too complicated
> for common tasks.
>
> --

[...]

>
> urllib is a package made of 4 parts:
>
> * urllib.request for opening and reading URLs
> * urllib.error containing the exceptions raised by urllib.request
> * urllib.parse for parsing URLs
> * urllib.robotparser for parsing robots.txt files
>
> I propose to deprecate all of them. Maybe the deprecation can be
> different for each sub-module?

Thanks for bringing this topic forward, Victor!

Disclaimer: I proposed the removal of urllib today in Python core's
internal chat.

The urllib package -- and to some degree also the http package -- are
constant source of security bugs. The code is old and the parsers for
HTTP and URLs don't handle edge cases well. Python core lacks a true
maintainer of the code. To be honest, we have to admit defeat and be up
front that urllib is not up to the task for this decade. It was designed
written during a more friendly, less scary time on the internet.

If I had the power and time, then I would replace urllib with a simpler,
reduced HTTP client that uses platform's HTTP library under the hood
(WinHTTP on Windows, NSURLSession (?) on macOS, Web API for Emscripten,
maybe curl on Linux/BSD). For non-trivial HTTP requests, httpx or
aiohttp are much better suited than urllib.

The second best option is to reduce the feature set of urllib to core
HTTP (no ftp, proxy, HTTP auth) and a partial rewrite with stricter,
more standard conform parsers for urls, query strings, and RFC 2822
instead of RFC 822 for headers.

Christian

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/WYVETVHMGRS4CI47GTFY6W7B43YLSJH2/

[Paul Moore]

On Sun, 6 Feb 2022 at 14:15, Victor Stinner <vsti...@python.org> wrote:

> I propose to deprecate the urllib module in Python 3.11. It would emit
> a DeprecationWarning which warn users, so users should consider better
> alternatives like urllib3 or httpx: well known modules, better
> maintained, more secure, support HTTP/2 (httpx), etc.

Also, I'm -1 on deprecating as a way of saying we *might* remove the
module, but haven't decided yet. That isn't (IMO) what deprecation is
for, and it doesn't give users a clear message, as maybe they'll be
fine continuing to rely on urllib. The net result would likely to be
for people to simply become more inclined to ignore deprecation
warnings.

Conversely, if the idea is to deprecate, and then in a couple of years
say "well, it's been deprecated for a while now, so let's remove it"
then that seems to me to be a rather cynical way of deflecting
arguments, as we can say now "well, it's only deprecation", in spite
of the fact that the real intention is to remove.

Paul

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/ICHMNBE7PMOHCGXLT4REP2HJZAGSOCHJ/

[Paul Moore]

On Sun, 6 Feb 2022 at 16:51, Christian Heimes <chri...@python.org> wrote:

> The urllib package -- and to some degree also the http package -- are
> constant source of security bugs. The code is old and the parsers for
> HTTP and URLs don't handle edge cases well. Python core lacks a true
> maintainer of the code. To be honest, we have to admit defeat and be up
> front that urllib is not up to the task for this decade. It was designed
> written during a more friendly, less scary time on the internet.
>
> If I had the power and time, then I would replace urllib with a simpler,
> reduced HTTP client that uses platform's HTTP library under the hood
> (WinHTTP on Windows, NSURLSession (?) on macOS, Web API for Emscripten,
> maybe curl on Linux/BSD). For non-trivial HTTP requests, httpx or
> aiohttp are much better suited than urllib.
>
> The second best option is to reduce the feature set of urllib to core
> HTTP (no ftp, proxy, HTTP auth) and a partial rewrite with stricter,
> more standard conform parsers for urls, query strings, and RFC 2822
> instead of RFC 822 for headers.

I'd likely be fine with either of these two options. I'm not worried
about supporting "advanced" uses. But having no way of getting a file
from the internet without relying on 3rd party packages seems like a
huge gap in functionality for a modern language. And having to use a
3rd party library to parse URLs will simply push more people to use
home-grown regexes rather than something safe and correct. Remember
that a lot of Python users are not professional software developers,
but scientists, data analysts, and occasional users, for whom the
existence of something in the stdlib is the *only* reason they have
any idea that URLs need specialised parsing in the first place.

And while we all like to say 3rd party modules are great, the reality
is that they provide a genuine problem for many of these
non-specialist users - and I say that as a packaging specialist and
pip maintainer. The packaging ecosystem is *not* newcomer-friendly in
the way that core Python is, much as we're trying to improve that
situation.

I've said it previously, but I'll reiterate - IMO this *must* have a
PEP, and that PEP must be clear that the intention is to *remove*
urllib, not simply to "deprecate and then think about it". That could
be making it part of PEP 594, or a separate PEP, but one way or
another it needs a PEP.

Paul

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/KT6TGUBTLLETHES2OVVGZWSGYC5JCEKC/

[Jelle Zijlstra]

PEP 594 is meant to be a set of uncontroversial removals of mostly unused modules. Removing urllib is obviously not going to be uncontroversial, so it should be discussed in a separate PEP.

[Victor Stinner]

On Sun, Feb 6, 2022 at 3:35 PM Paul Moore <p.f....@gmail.com> wrote:
> urllib.request may not be "best practice", but it's still extremely
> useful for simple situations, and urllib.parse is useful for basic
> handling of URLs.Yes, the more complex aspects of urllib are better
> handled by external packages, but that's not sufficient argument for
> removing the package altogether. There are many situations where
> external dependencies are unsuitable. Also, there's quite a lot of
> usage of urllib in the stdlib itself - how would you propose to
> replace that?

> (...)

> In addition, pip relies pretty heavily on urllib (parse and request),
> and pip has a bootstrapping issue, so using 3rd party libraries is
> non-trivial.

If a project like urllib3 uses it, urllib can be copied there and its
maintenance will continue there. Or maybe the maintenance can be moved
into a new project on PyPI like "legacy_urllib".

It's situation similar to the distutils deprecation: setuptools
decided to include a hidden copy of the distutils in its source, and
the distutils maintenance moved there. IMO it's a great move.
setuptools is a better place than Python to maintain this code:
setuptools release cycle is faster and is related to pip. Python
release cycle is slow and the distutils API was too big. Since the
distutils API is now hidden, setuptools can freely drop code and
changing APIs without affecting the public setuptools API.

I'm well aware that moving distutils into setuptools caused troubles.
IMO it is worth it and we have to go trough these issues once for a
better maintenance burden in the long term.

> In any case, why is this being proposed as a simple posting on
> python-dev? There's already PEP 594 for removals from the stdlib.

urllib is bigger than modules proposed for deprecation in PEP 594.
Also, I expect that deprecating urllib is more controversial.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/UR6BT5S2S4WGEI62MRWHCRAPZNTQXTVT/

[sethmich...@gmail.com]

Chiming in to say that whichever way this goes urllib3 would be okay. We can always vendor the small amount of http.client logic we actually depend on for HTTP connections. I do agree that the future of HTTP clearly lies outside the standard library, our team is already thinking about ways to integrate non-http.client HTTP implementations (like HTTP/2).

My feeling is that it will be difficult to remove urllib.parse, however urllib.request is much less depended on and more likely to be deprecated and removed.

Also clarifying that httplib2 doesn't support HTTP/2, the HTTP/2 package of interest is usually h2: https://pypi.org/project/h2. "http3" also doesn't implement HTTP/3 (bad name), this was one of the potential names for the HTTPX project.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/AW3JP6DHEAKME5FTFNRHV3EJMPJQEDME/

[Ethan Furman]

On 2/6/22 6:08 AM, Victor Stinner wrote:

> I propose to deprecate the urllib module in Python 3.11. It would emit
> a DeprecationWarning which warn users, so users should consider better
> alternatives like urllib3 or httpx: well known modules, better
> maintained, more secure, support HTTP/2 (httpx), etc.

Besides the needs of pip, round-up, etc., I think we should keep whatever parts of urllib, cgi, cgitb, http, etc., are
necessary for basic serving/consuming of web pages for the same reason we ended up keeping the wave module -- it's fun
and engaging for a younger audience. Having one computer get information from another is pretty cool.

If we need to do some trimming and rearranging of the above modules, that's fine, but I think losing all the
functionality would be a mistake.

--
~Ethan~

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/TGENXEKPFCIZUQD63ROCIK2WGAN3F7XL/

[Gregory P. Smith]

This would need to be it's own PEP.  urllib et. al. are used by virtually everybody.  They're highly used batteries.

I'm -1 on deprecating it for that reason alone.

Christian proposes that having a simpler scope rewrite of it might be nice, but I think disruption to the world and loss of trust in Python would be similar either way.

-gps

[Edwin Zimmerman]

> Christian proposes that having a simpler scope rewrite of it might be nice, but I think disruption to the world and loss of trust in Python would be similar either way.

Please don't remove urllib.  There are mountains of code that rely on it.  A much better idea, IMO, would be to add a new modern API to http.client, where http functionality properly belongs.  Maybe a function signature like this: 
http.client.get(url, user_agent = None, basic_auth=(None, None),  custom_headers=None).  That would one line to cover many use basic use cases, including user agent and basic auth.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/EV7R35OMQ7QWY7Y744FX7Y7VI7AO5CWX/

[Steve Dower]

On 2/6/2022 4:44 PM, Christian Heimes wrote:
> If I had the power and time, then I would replace urllib with a simpler,
> reduced HTTP client that uses platform's HTTP library under the hood
> (WinHTTP on Windows, NSURLSession (?) on macOS, Web API for Emscripten,
> maybe curl on Linux/BSD). For non-trivial HTTP requests, httpx or
> aiohttp are much better suited than urllib.

I'm +1 on this, though I think it would have to be in place before the
"two releases until removal" kicked in for urllib.request.

The stdlib can't get by without at least the basic functionality of curl
built in natively. But we can do this on most platforms without
vendoring OpenSSL, which is a HUGE win. Then our default behaviour could
correctly use proxies (including auto-config), CA certificate bundles,
integrated authentication, and other OS features that are currently
ignored by our core.

Chances are we could keep simple urlopen() calls in place, and use the
deprecation as a "potential change of behaviour" without necessarily
having to break the API. I'm yet to come across a case where making a
trivial urlopen() call _better_ would break things (the cases I've seen
that would break are things like "using an OpenSSL environment variable
to configure something that I wish had been automatic").

The nature of network/internet access is that we have to break things
periodically anyway, because all the code that was written over the last
30+ years is eventually going to be found to be exploitable. I'd be
quite happy to say "Python gives you what your OS gives you; update the
OS for fixes".

Cheers,
Steve

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/2P3RL7PAOZZFZ7PRGO6FJRMKR6MM2VXH/

[Stéfane Fermigier]

My two cents:

1. I’ve grepped ("ag-ed" acually) through all my code bases, and as noted by others, urllib.parse is used in many places. urllib.request never is, as we've been using requests or httpx instead.

IMHO and for the context I'm using it (YMMV), urllib.parse is useful and should be kept (could be replaced by third-party libs like furl or yarl, but probably not a great idea).

2. Obviously bootstrapping pip or similar package management tools should be a concern.

3. Overall, I think the days were "battery included" was a positive argument are over. I'd rather make the standard library leaner, and focussed on core language constructs. The only advantage that I can see is that having stuff in the standard lib can reduce fragmentation, at least initially, and ensure a very high level of quality and support, but at some point in the future history has shown us that usually a better alternative ends up emerging.

4. When deprecating and removing stuff from the stdlib and if there are no dependency issues, it should be possible to more the components to their own dedicated packages, maybe under an "extra" or "legacy" organisation. The question of support stays open, though.

Regards,

  S.

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/QMSFZBQJFWKFFE3LFQLQE2AT6WKMLPGL/

Code of Conduct: http://python.org/psf/codeofconduct/

--

Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier
Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/

Co-Founder & Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/

Co-Founder & Chairman, Association Professionnelle Européenne du Logiciel Libre (APELL) - https://www.apell.info/

Co-Founder & Spokesperson, European Cloud Industrial Alliance (EUCLIDIA) - https://www.euclidia.eu/ 

Founder, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/

[Brett Cannon]

On Mon, Feb 7, 2022 at 4:56 AM Steve Dower <steve...@python.org> wrote:

On 2/6/2022 4:44 PM, Christian Heimes wrote:
> If I had the power and time, then I would replace urllib with a simpler,
> reduced HTTP client that uses platform's HTTP library under the hood
> (WinHTTP on Windows, NSURLSession (?) on macOS, Web API for Emscripten,
> maybe curl on Linux/BSD). For non-trivial HTTP requests, httpx or
> aiohttp are much better suited than urllib.

I'm +1 on this, though I think it would have to be in place before the
"two releases until removal" kicked in for urllib.request.

Yes, we definitely couldn't deprecate anything regarding downloading over HTTP w/o having a replacement in place.

I am not even considering deprecating urllib.parse.

 

The stdlib can't get by without at least the basic functionality of curl
built in natively. But we can do this on most platforms without
vendoring OpenSSL, which is a HUGE win. Then our default behaviour could
correctly use proxies (including auto-config), CA certificate bundles,
integrated authentication, and other OS features that are currently
ignored by our core.

I also agree this is the best of the 2 options, although I would also accept Christian's other option of a more targeted, tight, standards-compliant solution if that would somehow lead to less maintenance overhead. And when I say "less maintenance overhead," I really mean it: I would question whether following redirects as an option is worth the overhead in this scenario. I'm very much thinking of this from a bootstrap/script/learning scenario and pushing people towards e.g. httpx for anything fancier.

 

Chances are we could keep simple urlopen() calls in place, and use the
deprecation as a "potential change of behaviour" without necessarily
having to break the API. I'm yet to come across a case where making a
trivial urlopen() call _better_ would break things (the cases I've seen
that would break are things like "using an OpenSSL environment variable
to configure something that I wish had been automatic").

We could try to get fancy and only raise DeprecationWarning in cases where things won't work to extend when we consider pushing people to the better API.

 

The nature of network/internet access is that we have to break things
periodically anyway, because all the code that was written over the last
30+ years is eventually going to be found to be exploitable. I'd be
quite happy to say "Python gives you what your OS gives you; update the
OS for fixes".

Exactly. My guideline for this whole idea would be that if it doesn't make sense in a beginner course that says to "download an HTML page and count all the anchor tags," then it's too fancy for the stdlib. And that should be enough to bootstrap installers which then get you httpx. Otherwise the networking stack moves too fast (from a security POV) and requires unique knowledge to get right that we have simply not kept up as much as we would like. I think it's okay to admit it might be time to trim with part of the stdlib down to something that we can manage easily (but we cannot drop the ability to download something over HTTPS).

[Jim J. Jewett]

There are problems with urllib. With hindsight, it would have been nice to do a few things differently. But that doesn't make migrating away from it any easier.

This thread has mentioned several "better" alternatives -- but with the exception of 3rd party Requests, the docs don't even mention them.

Saying "You can do better, but we won't tell you how" is pretty rude to beginners, and we should not do it.

Delegating to the operating system may be sensible for a production system, and there is nothing wrong with saying so in the docs, and it would be great if we made that easy. But it is absolutely not a reasonable replacement for a straightforward (possibly inefficient and non-scalable) implementation written in python that people can read and use for reference. urllib shouldn't be deprecated until we have a better solution to *that* use case that is also in the stdlib. (That might well be worth doing, but it should happen before the deprecation.)

-jJ

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/JI5CFS3WYXQEXKSEZH2ZTE3JJJ7AUAMW/

[Marc-Andre Lemburg]

... and there are also plenty examples out there of using http.server as
a quick HTTP server for trying out new things, testing and teaching.

FWIW: I find this discussion a bit strange. Python's stdlib is
supposed to provide basic tooling for a breadth of use cases,
with emphasis on "basic" and "breadth".

urllib is such a basic library and covers one of the main
use cases for Python we have. It would be pretty much beside
the point of the stdlib to remove such basic functionality
and make Python less attractive and less useful for beginners.

Anything more complex can be dealt with on PyPI, as it is already
happening.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 08 2022)
>>> Python Projects, Coaching and Support ... https://www.egenix.com/
>>> Python Product Development ... https://consulting.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
https://www.egenix.com/company/contact/
https://www.malemburg.com/

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/AKWL7QKOZJDODM3LIX4BBYZ4HMXZC3CZ/

[Christopher Barker]

On Tue, Feb 8, 2022 at 1:31 AM Marc-Andre Lemburg <m...@egenix.com> wrote:

FWIW: I find this discussion a bit strange. Python's stdlib is
supposed to provide basic tooling for a breadth of use cases,
with emphasis on "basic" and "breadth".

urllib is such a basic library and covers one of the main
use cases for Python we have.

Exactly. However, it is also a bit of an "attractive nuisance". For example, there is a lot of code in some of my major projects that use urllib for more complex cases, where we'd be much better off with requests, or ... 

Yes, that's mainly the result of our team's atrocious lack of code review, but this code was written by smart productive people.

The fact is that the stdlib is the first place folks look for stuff, and if what you are looking for is there, then many people won't think: "maybe there's a better, and well supported, package on PyPi for this"

So my thoughts:

Rather than deprecate urllib, we refactor it a bit (and maybe deprecate parts of it), so that it:

1) contains the core building blocks: e.g. urllib.parse with which to build "better" libraries,

2) make the "easy stuff easy" -- e.g. a basic http: request.

- For instance, I'd like to see an API that's kind of "requests-lite" 

And much better docs explain when you should use it, and when you might want to look for another library (even if it's the stdlib http.client)

I note that I don't see any discussion of that in urllib dics, whereas http.client does have the suggestion front and center that you might want to use requests.

Yes, the web moves fast, but it's also pretty backward compatible - folks keep old browsers around for an astonishingly long time! So I'd think cPython release Cycle shold be able to keep up with all but the very latest.

> make Python less attractive and less useful for beginners.

On this point, I'm not so sure -- the first thing I do for beginners is to point them to requests, as it's easier to use :-) -- but see my point above, that's why it would be good to put an easy-to-use-for-the-basics API in the stdlib

-CHB

--

Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython

[Chris Angelico]

On Wed, 9 Feb 2022 at 04:50, Christopher Barker <pyth...@gmail.com> wrote:
> So my thoughts:
>
> Rather than deprecate urllib, we refactor it a bit (and maybe deprecate parts of it), so that it:
>
> 1) contains the core building blocks: e.g. urllib.parse with which to build "better" libraries,
>
> 2) make the "easy stuff easy" -- e.g. a basic http: request.
> - For instance, I'd like to see an API that's kind of "requests-lite"
>
> And much better docs explain when you should use it, and when you might want to look for another library (even if it's the stdlib http.client)

This sounds like a decent plan. I'd like to add my voice to the appeal
to keep urllib.parse; in fact, of all the places where I've used
anything from urllib, only two of them are anything other than
urllib.parse. (One is an old script that I specifically wanted to be
as shareable as possible, so I restricted it to the stdlib; the other
catches urllib.error.URLError thrown by a third-party library.) If
there are security issues with urllib.request, I wouldn't shed many
tears about its deprecation.

A "requests-lite" module would certainly be handy, but it's hard to
judge how much wants to be in the stdlib and how much can be pushed
off to a pip-installable module:

> the first thing I do for beginners is to point them to requests, as it's easier to use :-)

Exactly my thoughts :) But a very very simple HTTP/HTTPS GET request
endpoint would be a great bootstrapping aid. Consider: with nothing
but the stdlib, you could fetch a file from some server, unzip it
(zipfile module), and import it. For building dirt-simple install
scripts, this kind of thing is really REALLY handy, and I'd rather not
have to use plain TCP sockets to do it :)

ChrisA

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/VVGKL2TA3UXDD3RDASIYSGEOLMTKPOSH/

[Brett Cannon]

On Mon, Feb 7, 2022 at 5:51 PM Jim J. Jewett <jimjj...@gmail.com> wrote:

There are problems with urllib.  With hindsight, it would have been nice to do a few things differently.  But that doesn't make migrating away from it any easier.

This thread has mentioned several "better" alternatives -- but with the exception of 3rd party Requests, the docs don't even mention them.

And as soon as httpx hits 1.0 I plan to update the docs to point at it. But until that occurs I personally do not want to have a debate about whether httpx's 0.N version number means it shouldn't be recommended.

 

Saying "You can do better, but we won't tell you how" is pretty rude to beginners, and we should not do it.

Delegating to the operating system may be sensible for a production system, and there is nothing wrong with saying so in the docs, and it would be great if we made that easy.  But it is absolutely not a reasonable replacement for a straightforward (possibly inefficient and non-scalable) implementation written in python that people can read and use for reference.  urllib shouldn't be deprecated until we have a better solution to *that* use case that is also in the stdlib.  (That might well be worth doing, but it should happen before the deprecation.)

Why do you think the stdlib must provide an example implementation for this specific scenario? Is there something unique to HTTP request handling that you feel is important to demonstrate?

[Jim J. Jewett]

> Why do you think the stdlib *must *provide an example implementation

> for this specific scenario? Is there something unique to HTTP request
> handling that you feel is important to demonstrate?

*must* is too strong, but I would use a very strong *should*.

I think the stdlib should provide simple source-included examples of most things. I think the case is even stronger when it is:

(1) a fairly simple protocol (such as version 1 of http was) -- QUIC wouldn't count for a simple demonstration.
(2) something new users are likely to find motivating. Short of "here is a way to do IO", and maybe "write a simple game", "get something from the web" is probably the most obvious case.
(3) something where bootstrapping might be an issue (network protocols, particularly web downloads). Network access is not an always-available resource. Even when it is available, there is sometimes a barrier between "available in python" and "I could read it on my phone, but can't get it open in python".
(4) something where a a beginner is likely to be overwhelmed by choices if we just say "use a 3rd party module".
(5) something with a backwards-compatibility story in the stdlib already.

As a side note, are there concerns about urllib.robotparser being broken or obsolete, or was that part of the deprecation proposal just contagion from urllib.request?

-jJ

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/HF5V6SFWV4BZUAOJTSEBD6DSZWSJONAM/

[Ethan Furman]

[apologies for the late post, just found this in my drafts folder]

On 2/7/22 12:49 AM, Stéfane Fermigier wrote:

> 3. Overall, I think the days where "battery included" was a positive argument are over

I strongly disagree. Being able to download something and immediately get something to work and see results is hugely
rewarding; on the other hand, having to research, find, compare & contrast available third-party modules (especially for
new-comers) can be extremely discouraging.

--
~Ethan~

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/N2F3AC6BFFQ63L3EFJVCQPBBV4MDBNSK/

[Paul Moore]

On Sun, 27 Mar 2022 at 00:52, Ethan Furman <et...@stoneleaf.us> wrote:
>
> [apologies for the late post, just found this in my drafts folder]
>
> On 2/7/22 12:49 AM, Stéfane Fermigier wrote:
>
> > 3. Overall, I think the days where "battery included" was a positive argument are over
>
> I strongly disagree. Being able to download something and immediately get something to work and see results is hugely
> rewarding; on the other hand, having to research, find, compare & contrast available third-party modules (especially for
> new-comers) can be extremely discouraging.

100% agreed.

I think people massively underestimate how frustrating it can be to
have to hunt out some piece of functionality that *doesn't* come with
the base download. Or worse, to find 5 different implementations, and
have no idea which one you "should" use.

Paul

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/2BIMFGNREXPOAFKGNVC7FZMT5LNVNVAY/

[Christopher Barker]

On Sun, Mar 27, 2022 at 3:08 AM Paul Moore <p.f....@gmail.com> wrote:

> > 3. Overall, I think the days where "battery included" was a positive argument are over
>
> I strongly disagree.  Being able to download something and immediately get something to work and see results is hugely
> rewarding; on the other hand, having to research, find, compare & contrast available third-party modules (especially for
> new-comers) can be extremely discouraging.

exactly - let's say someone needs to write some JSON for the first time. 

With the json package included, all they need to do is `import json`. If that wasn't there, they's look in PyPi for a JSON implementation, and find an absolutely astonishing number of options. I just did a search for "JSON"

 on PYPI, and it's HUGE -- most of them are for specialized JSON-using protocols of some sort. I was actually really surprised that couple I know about of the top of my head (ujson, orjson) are actually hard to find.

"You can just pip install it" is really not a good solution.

In fact, this is an example, I think, of where we should put some effort into making the included batteries better -- it's great to have a JSON lib built in, but it's too bad that it's not best-of-bread by pretty much any definition (speed, memory performance, flexibility) -- there are quite a few feature requests open for it -- it would be nice to actually implement some of those. (but yes, that's a lot of work that someone(s) would have to do)

Back to the topic at hand, rather than remove urllib, maybe it could be made better -- an as-easy-to-use-as-requests package in the stdlib would be really great.

-CHB

[lincoln auster [they/them]]

> Back to the topic at hand, rather than remove urllib, maybe it could
> be made better -- an as-easy-to-use-as-requests package in the stdlib
> would be really great.

Agreed. I think one thing that's easy to forget is that Python is useful
not just for very large applications but also for your standard tiny
one-off scripts (i.e., cron jobs and the like). The big standard library
is indispensible for writing clean code in those somewhat inherently
hacky situations. Note that this isn't really unique to Python. If we
look across the landscape for scripting lanugages useful in the same
areas, Bash stands out as being a bit of an odd language on its own, but
one that pairs well with all your traditional UNIX tools. These tools
make for a decently capable 'standard library', and Bash a surprisingly
versatile tool. My point obviously isn't that Python is or should be
Bash, but that Python gets a tangible benefit from its batteries in much
the same way as Bash gets a benefit from sed, awk, &c. Namely, these
languages are both made so much more /versatile/ by their respective
included tools. Being able to count on some common problems *just being
solved already* is a highly appreciated language feature, and something
I'd love to see stable and supported going forward.

So, let's not remove urllib. The fact that we're having this discussion
indicates that the scope of urllib is somewhat undefined, so perhaps it
would benefit from a firm and semi-formal (re)definition of scope
(hopefully in a manner motivated by real code). After that, it would
probably benefit from patching up all the rough spots in the
implementation side of things. To keep the metaphor going, let's
recharge our dead batteries.

--
lincoln auster
they/them


As an aside, I've opened PR#30520 on the CPython repo, which makes
urllib.urlparse less dependent on the hard-coded list of schemes it uses
to determine parsing behavior. In the interest of recharging batteries,
would anyone be willing to review that? @orsenthil volunteered a while
back, but with all of my first-PR issues, it's sort of stalled.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/FR4AGGHFUQ3XUHUG3Q3QF7PAO6HJ3CJI/

[Paul Moore]

On Sun, 27 Mar 2022 at 17:11, Christopher Barker <pyth...@gmail.com> wrote:
>
> With the json package included, all they need to do is `import json`. If that wasn't there, they's look in PyPi for a JSON implementation, and find an absolutely astonishing number of options. I just did a search for "JSON"
> on PYPI, and it's HUGE -- most of them are for specialized JSON-using protocols of some sort. I was actually really surprised that couple I know about of the top of my head (ujson, orjson) are actually hard to find.
>
> "You can just pip install it" is really not a good solution.
>
> In fact, this is an example, I think, of where we should put some effort into making the included batteries better -- it's great to have a JSON lib built in, but it's too bad that it's not best-of-bread by pretty much any definition (speed, memory performance, flexibility) -- there are quite a few feature requests open for it -- it would be nice to actually implement some of those. (but yes, that's a lot of work that someone(s) would have to do)
>
> Back to the topic at hand, rather than remove urllib, maybe it could be made better -- an as-easy-to-use-as-requests package in the stdlib would be really great.

I think that's where the mistake happens, though. Someone who needs
"best of breed" is motivated (and likely knowledgeable enough) to make
informed decisions about what's on PyPI. But someone who just wants to
get the job done probably doesn't - and that's the audience for the
stdlib. A stdlib module needs to be a good, reliable set of basic
functionality that non-experts can use successfully. There can be
better libraries on PyPI, but that doesn't mean the stdlib module is
unnecessary, nor does it mean that the stdlib has to match the PyPI
library feature for feature.

So here, specifically, I'd rather see urlllib be the best urlllib it
can be, and not demand that it turn into requests. Requests is there
if people need/want it (as is httpx, and urllib3, and aiohttp). But
urllib is for people who want to get a file from the web, and *not*
have to deal with dependencies, 3rd party libraries, etc.

The "batteries included" standard library and PyPI complement each
other. Neither is redundant, and neither implies the other is
unnecessary.

Paul
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/E2FGXTYOEYLC6GSJGMWBQKHOO62LZPHQ/

[Steve Holden]

Time for a __legacy__ package?

Kind regards,

Steve

[Dan Stromberg]

On Sat, Mar 26, 2022 at 5:58 PM Ethan Furman <et...@stoneleaf.us> wrote:

[apologies for the late post, just found this in my drafts folder]

On 2/7/22 12:49 AM, Stéfane Fermigier wrote:

> 3. Overall, I think the days where "battery included" was a positive argument are over

I strongly disagree.  Being able to download something and immediately get something to work and see results is hugely
rewarding; on the other hand, having to research, find, compare & contrast available third-party modules (especially for
new-comers) can be extremely discouraging.

It might make sense to have CPython's release cadence decoupled from the Standard Library's release cadence.  That is, maybe they should be separate downloads.

[MRAB]

On the other hand, it's nice to get it all in one go. Perhaps the
installer could gain controls to download optional parts of the standard
library or update those parts that have been installed.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/73DJBBJS7CF5CGSSVRPITQR6NU54LXJV/

[lincoln auster [they/them]]

> On the other hand, it's nice to get it all in one go. Perhaps the
> installer could gain controls to download optional parts of the
> standard library or update those parts that have been installed.

I'm not sure the proposed is really a standard library if it's
distributed like that. If a given script cannot rely on the presence of
some given module --- one which may need to be installed as though it
were just another 3rd-party component --- then it's not really all that
standard, is it?

--
Lincoln Auster
they/them

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/OZSDKTLIGHSNQY32ZIX3B2ALSB4CARAM/

[Barry Warsaw]

On Mar 27, 2022, at 11:52, Dan Stromberg <drsa...@gmail.com> wrote:
>
> It might make sense to have CPython's release cadence decoupled from the Standard Library's release cadence. That is, maybe they should be separate downloads.

While I don’t underestimate the work and complexity, we can do both. I.e. separate the stdlib development cycle from the interpreter (for all but a handful of required packages perhaps). We could still distribute “sumo” releases which include all the batteries, but develop and maintain them outside the cpython repo, and even release them separately on PyPI. It’s *possible* but I don’t know if it’s *practical*.

-Barry

[Stephen J. Turnbull]

Barry Warsaw writes:

> While I don’t underestimate the work and complexity, we can do
> both. I.e. separate the stdlib development cycle from the
> interpreter (for all but a handful of required packages perhaps).
> We could still distribute “sumo” releases which include all the
> batteries, but develop and maintain them outside the cpython repo,
> and even release them separately on PyPI. It’s *possible* but I
> don’t know if it’s *practical*.

After a couple decades maintaining XEmacs, I'd have to say that one of
the things that made our sumos (you heard that word here first,
folks!) practical was that about 80% of the code was developed in GNU
Emacs. That is, developed in concert with core Emacs. This meant
that dependencies (think "hybrid of hydra, kitsune-yokai, and
ouroboros" :-) evolve together, as they do in Python. This is
probably more important in Emacs because of the big-ball-of-string
nature of big Lisp systems (at least with a relatively undisciplined
dev process), but I'd still say there would be *substantial*
coordination costs to be paid if you split the repos.

Also, the only feature in Emacs Lisp that really changed the way
people code since 1990 (and likely earlier) is lexical scope.
Everything else is merely a new callable for practical purposes, and
almost all of it can be implemented in Lisp either as a function or a
macro (although usually at great expense in performance). But in
Python, every new release makes the Mailman crew want to stop
supporting all previous releases of Python because there's some
feature that can't be emulated that we really love: genexps or async
or walrus operator or .... It was certainly quite costly in XEmacs
trying to most packages usable in 21.4, mostly in frustration about
not being able to use 21.5 features. ;-) But Python has like 4 or 5
generations of stdlib in some form of development or maintenance at
any given time. It's really not attractive to think either about
creating a separate Sumo Dev Core to maintain several generations of
stdlib corresponding to in-maintenance versions of Python, or about a
lowest-common-denominator stdlib.

Until XEmacs split off the stdlib into packages, Emacs didn't have
anything like a package repository, certainly nothing like PyPI. I
don't have experience with the Emacs package repos ELPA and MELPA for,
uh, hysterical raisins, so season the next comment to taste. My
feeling is that both splitting out the stdlib and increasing the rate
at which new packages are added are excessively expensive non-
solutions to the problem that Paul has described, that finding the
right package on PyPI is pretty hard, especially for people new to a
technology. The extra effort that would be needed to achieve either,
as well as the increased on-going maintenance burden on the core team,
would not really resolve the problem, given that there are now whole
large ecosystems like Django and SciPy outside of Python core
development.

Although it's a different kind of effort from package incorporation
and maintenance, to the extent that effort is fungible, I think it
would be better devoted to creating an annotated catalog of PyPI.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/ZKODGJ3I6XQNX7ISE4EVCI5RVRCTTOA4/

[Skip Montanaro]

Barry writes (in part):

> We could still distribute “sumo” releases which include all the
> batteries, but develop and maintain them outside the cpython repo,
> and even release them separately on PyPI. It’s *possible* but I
> don’t know if it’s *practical*.

to which Stephen responds (in part):

> [Emacs really didn't change much in 20 years.] .... [I]n

> Python, every new release makes the Mailman crew want to stop
> supporting all previous releases of Python because there's some
> feature that can't be emulated that we really love: genexps or async
> or walrus operator or ....

It seems to me that what we have is the possibility that, say, package
P migrates to PyPI in concert with the release of Python version X
where maintenance can be picked up by the broader Python community.
Whether or not it would proceed to track ongoing changes to the
language isn't clear, and it's not obvious to me that less effort
would be required other than perhaps by core devs (though some of them
would likely pitch in to maintain packages with which they are
currently involved). If you go with the "discard the batteries"
approach, I think it would at least be worthwhile distributing a
requirements.txt file ("sumo.txt" or "batteries.txt"?) which would
tell someone installing Python how to reclaim the batteries of their
Python youth, and for other Python implementations to track a set of
packages which would make them plausibly compatible with CPython. CI
could still rely on that to provide as much test coverage as current
today (I think). If nothing else, it might alert the core devs to the
potential for breakage of presumably widely used packages by changes
to CPython.

What happens to P when Python version Y grows a new syntactic feature?
Do P's maintainers fork to both continue feature growth as well as
syntactic modernization? If something like batteries.txt is created to
tie a slimmed-down CPython distribution to the batteries it once
contained, would the core group exert any control over unit testing,
documentation, package variations and such?

Just thinking out loud...

Skip

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/76ZTGO2PD6Q5NZOO5JWJRTTR2E2JY2H6/

[Coyot Linden (Glenn Glazer)]

On Sun, Mar 27, 2022 at 3:08 AM Paul Moore p.f....@gmail.com wrote:

...

exactly - let's say someone needs to write some JSON for the first time.

With the json package included, all they need to do is `import json`. If that wasn't there, they's look in PyPi for a JSON implementation, and find an absolutely astonishing number of options. I just did a search for "JSON" on PYPI, and it's HUGE -- most of them are for specialized JSON-using protocols of some sort. I was actually really surprised that couple I know about of the top of my head (ujson, orjson) are actually hard to find.

"You can just pip install it" is really not a good solution.

In fact, this is an example, I think, of where we should put some effort into making the included batteries better -- it's great to have a JSON lib built in, but it's too bad that it's not best-of-bread by pretty much any definition (speed, memory performance, flexibility) -- there are quite a few feature requests open for it -- it would be nice to actually implement some of those. (but yes, that's a lot of work that someone(s) would have to do)

Back to the topic at hand, rather than remove urllib, maybe it could be made better -- an as-easy-to-use-as-requests package in the stdlib would be really great.

-CHB

Strong plus one to this. Part of Python's huge attraction and widespread adoption is that one doesn't need a lot of formal training or engineering experience to just dive right in. Having a rich set of libraries only helps this, forcing newbies to pip/PYPI is a great way to lose them.

I also completely agree that the right answer is to make the standard libraries better, not throw them away.

And for those that don't use them, ignore them.

Best,

coyot
P.S. There are uses for urllib outside of standard web programming. I recently needed to handle strings in query parameter format (`?...&...&...`) and found urllib parse() the easiest way of doing that even though I wasn't taking them in as a webservice. Deleting the library would break this code. :(

[Petr Viktorin]

What happens when the new maintainer puts malware in the next release of
a package in sumo.txt?
Will core devs be blamed for listing it?
As a user, how do I determine if I can trust the packages there? (This
is easily the hardest part of finding and installing a package from
PyPI, though many people seem to skip it.)

> What happens to P when Python version Y grows a new syntactic feature?
> Do P's maintainers fork to both continue feature growth as well as
> syntactic modernization? If something like batteries.txt is created to
> tie a slimmed-down CPython distribution to the batteries it once
> contained, would the core group exert any control over unit testing,
> documentation, package variations and such?

If they do exert control, why not keep it in stdlib?

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/YLK5UMSWYXX6JT2PL23F26DGQ5XSOVV2/

[Steve Dower]

I think to most people "batteries included" doesn't necessarily mean
"standard library," with all that implies. It just means "it came with
the first thing that I installed" (or alternatively, "at no additional
charge").

Given there are *plenty* of existing distros out there that install more
than just the standard library for the benefit of their users, and yet
the people asking for a fatter stdlib don't seem to want them, perhaps
we just need to officially endorse one?

I've got no issue preinstalling a curated sumo.txt into the Windows
installers, provided we're clear about support (i.e. none from us) and
someone besides me is tracking everything in that curated list. That
probably requires new volunteers who are somehow already trustworthy,
but that should be easier than people to actually write and maintain new
functionality on stdlib timelines.

We would still have to have the "light" distribution, as many of our
users wouldn't be able to use a "this is full of unsupported stuff"
package, but at least most users could be pointed to the bigger install
by default and not even care that the batteries just came in the same
package and aren't actually an integral part of the core product.

Cheers,
Steve

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/TC5EEDFNSZJBR22GQDJ32UWVTQNILU2P/

[lincoln auster [they/them]]

"Coyot Linden (Glenn Glazer)" <co...@lindenlab.com> writes:
> P.S. There are uses for urllib outside of standard web programming. I
> recently needed to handle strings in query parameter format
> (`?...&...&...`) and found urllib parse() the easiest way of doing
> that even though I wasn't taking them in as a webservice. Deleting the
> library would break this code. :(

+1. If urllib is removed, I would very much like to preserve at least
the functionality of urlparse /somewhere/.

--
Lincoln Auster
they/them

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/C3JZDCWLCMED32Q4MFHMITI2QQGK2ME5/

[Damian Shaw]

> If urllib is removed, I would very much like to preserve at least the functionality of urlparse /somewhere/.

Given every alternative library to urllib relies on urllib.parse (and some rely on urllib.request), as well as popular libraries like pip, in this hypothetical it would definitely need to be maintained somewhere.

Damian (he/him)

[Paul Moore]

On Mon, 28 Mar 2022 at 17:45, Steve Dower <steve...@python.org> wrote:
>
> I think to most people "batteries included" doesn't necessarily mean
> "standard library," with all that implies. It just means "it came with
> the first thing that I installed" (or alternatively, "at no additional
> charge").

I think that for users it means "came as part of the official
installer" where "official" means "whoever I chose to trust" - which
in turn often means "python.org".

For package maintainers it means "the bunch of stuff I can assume
exists when someone says they have python" - which is why distros that
debundle parts of the stdlib get so much grief. Also note that "just
add a dependency" is not a simple request for a library maintainer -
if I depend on requests, and someone using my library uses httpx, then
I'm imposing a potential transitive dependency conflict on them. So
using stdlib functionality is a non-trivial advantage for library
maintainers (of course, sometimes the benefits of using a 3rd party
library outweigh this, but not always).

> Given there are *plenty* of existing distros out there that install more
> than just the standard library for the benefit of their users, and yet
> the people asking for a fatter stdlib don't seem to want them, perhaps
> we just need to officially endorse one?

But people don't use those existing options presumably for a reason.
Having the core devs say "distro X is approved" won't remove those
reasons. And removing the python.org distro because X is now approved
will create a problem for those users.

And none of this alters the fact that "stdlib" means "can be assumed
to be present in all Python installations" for some use cases.

> I've got no issue preinstalling a curated sumo.txt into the Windows
> installers, provided we're clear about support (i.e. none from us) and
> someone besides me is tracking everything in that curated list. That
> probably requires new volunteers who are somehow already trustworthy,
> but that should be easier than people to actually write and maintain new
> functionality on stdlib timelines.

"Support" is a somewhat ambiguous matter in the context of
volunteer-supported software. The point of having a stdlib is that
people know where to report issues, not that we're necessarily any
more responsive than a 3rd party maintainer. There's a value in
knowing that if you find an issue with "anything you installed as part
of the basic Python install", you report it in the same place. And
having the documentation all in one place. Etc.

Yes, people come along and make a big fuss about how they expect more
in the way of support ("this bug is urgent, my production system is
failing because of it") and yes, people with that attitude have a
negative effect on core developer morale. But we don't promise fixes,
or solutions. At best there's an implication that the core devs care
about the stdlib in some sort of vague, generalised sense. (I care
about the health of the stdlib, but I'm not likely to try to fix a bug
in ossaudiodev, for example).

If we did try to mandate something more concrete, then maybe the
conversation would be different. But I'm not aware of anyone proposing
anything like that.

> We would still have to have the "light" distribution, as many of our
> users wouldn't be able to use a "this is full of unsupported stuff"
> package, but at least most users could be pointed to the bigger install
> by default and not even care that the batteries just came in the same
> package and aren't actually an integral part of the core product.

To be honest, I feel like I'm just reiterating stuff I've said before
here, and I think the same is true of the points I'm responding to. Is
there actually any new development here, or is it just a repeat of the
same positions people have expressed in the past, based on the same
underlying realities? (I'm not *against* going over the debate again,
it helps make sure people haven't changed their minds, but it's
important to be clear that none of the practical facts have changed,
if that is the case).

I should also point out (as someone heavily involved in Python's
packaging ecosystem) that I don't believe that the existing packaging
infrastructure is sufficiently robust to support a world where the
Python stdlib was all delivered as external packages. And even if it
were, bootstrapping something like pip would be a nightmare - for a
start, we don't vendor libraries that include C extensions, so we
could only handle pure Python stdlib modules - and even vendoring all
of them (and all of our vendored tools' stdlib dependencies) would be
a nightmare.

Paul

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/SKIKUC4U3L74G43BNC5SDCXFYUY3G447/

[Christopher Barker]

On Sun, Mar 27, 2022 at 11:04 AM Paul Moore <p.f....@gmail.com> wrote:

> In fact, this is an example, I think, of where we should put some effort into making the included batteries better -- it's great to have a JSON lib built in, but it's too bad that it's not best-of-bread by pretty much any definition (speed, memory performance, flexibility) -- there are quite a few feature requests open for it -- it would be nice to actually implement some of those. (but yes, that's a lot of work that someone(s) would have to do)
>
> Back to the topic at hand, rather than remove urllib, maybe it could be made better -- an as-easy-to-use-as-requests package in the stdlib would be really great.

I think that's where the mistake happens, though. Someone who needs
"best of breed" is motivated (and likely knowledgeable enough) to make
informed decisions about what's on PyPI. But someone who just wants to
get the job done probably doesn't - and that's the audience for the
stdlib. A stdlib module needs to be a good, reliable set of basic
functionality that non-experts can use successfully. There can be
better libraries on PyPI, but that doesn't mean the stdlib module is
unnecessary, nor does it mean that the stdlib has to match the PyPI
library feature for feature.

So here, specifically, I'd rather see urlllib be the best urlllib it
can be, and not demand that it turn into requests. Requests is there

if people need/want it.

Actually, I think this is very hard to talk about in generalities. If. you look at the two examples I used, json and urllib, we have two different "problems".

I would like to see a requests-like interface in the stdlib not because it's "best of breed", and has, e.g. more features or better performance, or ...., I would like to see it because it has a cleaner, simpler interface (the easy things should be easy) -- so I'm not necessarily advocating that the entirely of requests be brought in to the stdlib, or that request be made obsolete, but that we borrow a bit of its API. Which would also make the transition from stdlib to requests easier -- write you scripts with the stdlib, and when you find you need some more "advanced" features, you simpy drop requests in and move along.

json is actually the opposite- most of the third-party json libs mimic (and maybe extend) the json API -- so you can, in most cases, start out with the stdlib, and drop in another app when you need more performance, or an extra feature.

The issue I have with the stdlib json lib is that  I think it could have a somewhat better API, and there's a few features I'd like to see it grow, so that third party libs are less necessary. But it seems there's not much motivation to do so, because "folks can always use a third party lib". Why is this an problem? two reasons:

1) There are quite a few json libs out there, and they tend to be a fourth party dependency (I just coined that term for a dependency of a dependency) . I literally have some of my code requiring four different json libs to run. It works, so maybe not a huge deal, but it would be really nice to simplify that!

2) I don't think any of the third party libs provide a full menu -- JSON5 support, performance, etc, so sometimes there's no single best of breed for even a particular application. Can/should that be solved by the stdlib? maybe not, but it would be nice to see it addressed somewhere -- a grand unification of JSON libs.

In short: I think I agree with most folks here that we should still include the batteries, and they should be updated / maintained to some extent. What exactly could/should be done is going to have to be worked on on a case by case basis.

-CHB

[Christopher Barker]

On Mon, Mar 28, 2022 at 11:29 AM Paul Moore <p.f....@gmail.com> wrote:

To be honest, I feel like I'm just reiterating stuff I've said before
here, and I think the same is true of the points I'm responding to

 ...

 (I'm not *against* going over the debate again,
it helps make sure people haven't changed their minds, but it's
important to be clear that none of the practical facts have changed,
if that is the case).

Maybe there's a way to make this discussion (it feels more like a discussion than debate at the moment) more productive by writing some things down. I'm not sure it's a PEP, but some kind of:

"policy for the stdlib" document in which we could capture the primary points of view, places where there's consensus, etc. would be helpful to keep us retreading this over and over again.

I suggest this without the bandwidth to actually shepherd the project, but if someone wants to, I think it would be a great idea.

-CHB

[Steve Dower]

On 3/28/2022 7:26 PM, Paul Moore wrote:
> To be honest, I feel like I'm just reiterating stuff I've said before
> here, and I think the same is true of the points I'm responding to. Is
> there actually any new development here, or is it just a repeat of the
> same positions people have expressed in the past, based on the same
> underlying realities? (I'm not *against* going over the debate again,
> it helps make sure people haven't changed their minds, but it's
> important to be clear that none of the practical facts have changed,
> if that is the case).

Yep, same. Nothing much has changed, but if those of us who know that
nothing has changed fail to reiterate those points, everyone new to the
discussion will assume that everyone agrees with them.

Just deleted a long rant to the discussion in general (not directed at
Paul at all ;) ) that would have achieved nothing, but I'll summarise to
this: email doesn't fix bugs; maintainers fix bugs. Please let us know
*publicly* if you want to become the maintainer for a stdlib module and
then we can support them, but if nobody is willing/able/ready to care
for them it's irresponsible for us to keep shipping them to users.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/I37FCHGNGCKACI4C4S3OA6MO4MZLRABY/

[Skip Montanaro]

> What happens when the new maintainer puts malware in the next release of
> a package in sumo.txt?
> Will core devs be blamed for listing it?
> As a user, how do I determine if I can trust the packages there? (This
> is easily the hardest part of finding and installing a package from
> PyPI, though many people seem to skip it.)

I will point out that you quoted my entire post except for the most important line, the one which reads:

> Just thinking out loud...

I am still thinking out loud, so keep that in mind. (Perhaps this belongs on one of the two ideas groups, but this is where things started, so I will continue here for now.)

Consider a hypothetical situation.

• Suppose Python v. X is the last version delivered with batteries, so v. Y lacks a json module (assuming that's not determined to be so important that it must be delivered as part of the CPython installer proper).
• Downstream, some Linux distro realizes their users still want batteries (as do their Python admin tools), so recreates a Python package with them. Their package of v. Y *does* include (at least) the json battery.
• One of the maintainers of the now externally maintained json battery takes it upon themselves to protest an incursion into East WhatsItStan by West WhatsItStan in the worst manner possible and inserts code to scrub disks of all West WhatsItStan-ian homed computers (as the primary maintainer of some widely installed (Node? JS?) package package did).
• Some poor West WhatsItStan-ian upgrades their Linux distro, only to find when they next run their favorite Python application that their disk drive is wiped clean.

Note that I haven't postulated the existence of a sumo.txt file. Whether the CPython distribution contains batteries or not, someone will recreate the batteries, if not in toto, at least piecemeal (some application or package depends on json and will blindly pull it in). Who gets the blame here? The core Python devs (because json "always came with Python")? The maintainers of the Linux distro for recreating the batteries but not having any West WhatsItStanian-based testers? The new maintainers of the now external (and corrupted) json package? Maybe the several levels of indirection would serve to insulate the Python devs, but maybe not. 

> If they do exert control, why not keep it in stdlib?

Perhaps that is the best route. Someone here (Stéfane Fermigier, it seems) questioned whether batteries included are such a good idea. It's quite possible that enough core devs disagree with that sentiment that the batteries will stay (and perhaps grow in number, though more slowly than in the past). That said, if enough devs agree with Stéfane, then what's the best route forward? Discard all batteries and let the chips fall where they may? Extract the dead batteries (and presumably their docs and test bits) into separate github.com/python repositories? Ask others to extract them into their own non-github.com/python repositories?

One of the common reasons old platform support is dropped from CPython (OS/2 or AmigaOS anyone?) is that the maintenance load for the core devs is too high relative to the community benefit derived from supporting small minority platforms. The unlucky modules named by PEP 594 are about to suffer the same fate. (I'm not arguing that they shouldn't be deleted.)

Once PEP 594 has been implemented, all the low-hanging fruit will have been picked. At that point, it's keep everything or keep (almost) nothing. I think that will largely depend on the willingness of the core devs to keep maintaining modules they may well never use in their own work (like getopt and wave). One thing I think is obvious. If you remove all the batteries not deemed absolutely essential to build and test CPython, I think you have to somehow symbolically say, "these correspond to the various batteries which used to be in the CPython distribution." Maybe it's text in a README file, a new PEP, a sumo.txt file, or a Linux distro's packaging.

Skip

P.S. Personally, I was never fond of shutil or the logging package (as two examples — though for different reasons). We might have something better in both realms by now if they hadn't been added to the stdlib long ago. This ability of presence in the stdlib to forestall further development might well be the strongest argument to remove most batteries. On the other hand, we seem to have had little trouble cycling through several completely different regular expression modules. Maybe I'm just imagining a barrier where none exists.

[Paul Moore]

On Mon, 28 Mar 2022 at 20:37, Steve Dower <steve...@python.org> wrote:

> Please let us know
> *publicly* if you want to become the maintainer for a stdlib module and
> then we can support them, but if nobody is willing/able/ready to care
> for them it's irresponsible for us to keep shipping them to users.

I'm sorry for picking on a single point you made and continuing to
simply state my contrary view, but I think it's important.

IMO, it's irresponsible of us to *remove* functionality if there's no
suitable alternative (and remember, my definition of "suitable"
includes "appropriate for cases where a PyPI library isn't acceptable
for some reason"). Of course, it's not OK for us to promise a level of
support that we can't or won't provide, but *we don't provide such
promises*. We say that we support the stdlib, but there's no actual
statement of what that means - so it's not clear to me how we're
failing to deliver "enough". Nor is it at all clear to me that dumping
the responsibility on an external maintainer will be any better - most
likely it just lets *us* feel better about the situation while not
actually being better for the end user.

If we were to define some sort of actual support guarantees, we might
be having a different conversation. But I'm unclear how a volunteer
organisation can provide meaningful promises that we wouldn't be able
to deliver for the stdlib as it stands now.

Paul

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/OSYFNLIULB56EVSG57QXKIW4NRWI4CNZ/

[Brett Cannon]

On Mon, Mar 28, 2022 at 11:52 AM Christopher Barker <pyth...@gmail.com> wrote:

On Mon, Mar 28, 2022 at 11:29 AM Paul Moore <p.f....@gmail.com> wrote:

To be honest, I feel like I'm just reiterating stuff I've said before
here, and I think the same is true of the points I'm responding to

 ...

 (I'm not *against* going over the debate again,
it helps make sure people haven't changed their minds, but it's
important to be clear that none of the practical facts have changed,
if that is the case).

Maybe there's a way to make this discussion (it feels more like a discussion than debate at the moment) more productive by writing some things down. I'm not sure it's a PEP, but some kind of:

"policy for the stdlib" document in which we could capture the primary points of view, places where there's consensus, etc. would be helpful to keep us retreading this over and over again.

I suggest this without the bandwidth to actually shepherd the project, but if someone wants to, I think it would be a great idea.

Once https://mail.python.org/archives/list/python-c...@python.org/thread/5EUZLT5PNA4HT42NGB5WVN5YWW5ASTT5/ is considered resolved, the next part of my "what is the stdlib" plan is to finally try to suss all of this out and more-or-less write a stdlib policy PEP so we stop talking about this. My guess it will be more of guidance about what we want the stdlib to be and thus guide what things belong in it. No ETA on that work since I also have four other big Python projects on the go right now whose work I am constantly alternating between.

-Brett

 

-CHB

--

Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/GAZAFRFVJVQZMEIHTQUJASP7VRAKA5RR/

[Toshio Kuratomi]

On Sun, Mar 27, 2022, 11:07 AM Paul Moore <p.f....@gmail.com> wrote:

On Sun, 27 Mar 2022 at 17:11, Christopher Barker <pyth...@gmail.com> wrote:
> Back to the topic at hand, rather than remove urllib, maybe it could be made better -- an as-easy-to-use-as-requests package in the stdlib would be really great.

I think that's where the mistake happens, though. Someone who needs
"best of breed" is motivated (and likely knowledgeable enough) to make
informed decisions about what's on PyPI. But someone who just wants to
get the job done probably doesn't - and that's the audience for the
stdlib. A stdlib module needs to be a good, reliable set of basic
functionality that non-experts can use successfully. There can be
better libraries on PyPI, but that doesn't mean the stdlib module is
unnecessary, nor does it mean that the stdlib has to match the PyPI
library feature for feature.

So here, specifically, I'd rather see urlllib be the best urlllib it
can be, and not demand that it turn into requests. Requests is there
if people need/want it (as is httpx, and urllib3, and aiohttp). But
urllib is for people who want to get a file from the web, and *not*
have to deal with dependencies, 3rd party libraries, etc.

One thing about talking about "make urllib more like requests" that is different than any of the other libs, though, is that requests aims to be easier to use than anything else (which I note Chris Barker called out as why he wanted urllib to be more like it).  I think that's important to think about because i think ease of use is also the number one thing that the audience you talk about is also looking for.

Of course, figuring out whether an api like request's is actually easier to use than urllib or merely more featureful is open to debate. 

-toshio

[Paul Moore]

On Tue, 29 Mar 2022 at 00:37, Toshio Kuratomi <a.ba...@gmail.com> wrote:

> One thing about talking about "make urllib more like requests" that is different than any of the other libs, though, is that requests aims to be easier to use than anything else (which I note Chris Barker called out as why he wanted urllib to be more like it). I think that's important to think about because i think ease of use is also the number one thing that the audience you talk about is also looking for.

Agreed, but one thing that I would note is that the more people ask
for changes of this sort of scope, the more this fuels the argument
that maintaining the stdlib is a lot of work. It's fairly obvious that
none of the core developers have the time or energy to invest in the
sort of urllib API change you're suggesting here, so realistically
this simply gives the impression "people don't want urllib, they want
requests". The net result is not that urllib becomes more like
requests, but rather that there's extra pressure to *drop* urllib, and
point people to requests.

Personally, I want *both* urllib and requests. I want urllib for when
I don't want a 3rd party dependency, and requests (or httpx, or
urllib3, ...) for when I can afford one. And if the cost of having
that is that urllib has a less user-friendly API, then I'm willing to
live with that.

Paul
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/77D3WHKPB2FS2JKAC3EPTPDH6UDQIAFD/

[Sebastian Rittau]

Am 27.03.22 um 18:11 schrieb Christopher Barker:

On Sun, Mar 27, 2022 at 3:08 AM Paul Moore <p.f....@gmail.com> wrote:

> > 3. Overall, I think the days where "battery included" was a positive argument are over
>
> I strongly disagree.  Being able to download something and immediately get something to work and see results is hugely
> rewarding; on the other hand, having to research, find, compare & contrast available third-party modules (especially for
> new-comers) can be extremely discouraging.

exactly - let's say someone needs to write some JSON for the first time.

I think the truth is somewhere in the middle. Python should include support for commonly needed functionality. This includes all kinds of language support (functools, collections, asyncio, wsgiref, re etc.), common file formats like XML, JSON, tar, zip etc., OS support (os, sys, stat etc.), basic networking (TCP/IP, making HTTP requests, sending mail), but also basic math modules like decimal, fractions, or statistics. I don't think it should support esoteric, niche, or long obsolete file formats or networking protocols like FTP, sunau etc. I also don't think that Python should come with "basic" servers that can't be used for anything useful. The same is true for domain-specific applications. For example, I wouldn't want a complex numerics package as part of Python (although having some more basic data types would make sense).

Another problem are "optional" modules like sqlite or tk/tcl. Because they are optional, you can't depend on them being present, but you also can't ensure their presence by "pip install"ing them.

In fact, this is an example, I think, of where we should put some effort into making the included batteries better -- it's great to have a JSON lib built in, but it's too bad that it's not best-of-bread by pretty much any definition (speed, memory performance, flexibility) -- there are quite a few feature requests open for it -- it would be nice to actually implement some of those. (but yes, that's a lot of work that someone(s) would have to do)

Back to the topic at hand, rather than remove urllib, maybe it could be made better -- an as-easy-to-use-as-requests package in the stdlib would be really great.

I agree 100%. On the one hand the stdlib is missing some functionality I would consider "basic" (for example, async file and HTTP fetching support), on the other hand some of the existing modules would benefit from a more modern API that makes the common case easy and the uncommon case possible.

 - Sebastian

[lincoln auster [they/them]]

> email doesn't fix bugs; maintainers fix bugs. Please let us
> know *publicly* if you want to become the maintainer for a stdlib
> module and then we can support them, but if nobody is
> willing/able/ready to care for them it's irresponsible for us to keep
> shipping them to users.

At the moment, I don't have the time, experience, skill, nor involvement
with the Python project to volunteer to maintain urllib, but I would
like to take a more active role in its maintenance as a contributor such
that we continue to see it in the stdlib. This may be the second time
I've mentioned this on this thread, but please see PR#30520 on the
CPython repo (BPO-46337), which attempts to patch up a rough edge in
urlparse's scheme-based API. Feedback/review/etc on that would be
greatly appreciated, and so would some pointers on what next needs the
most work with regards to urllib as a whole. Thanks so much!

--
Lincoln Auster
they/them

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/HD46JG3JT525KEGYXETX2OCR63MWIPXU/

[Ronald Oussoren via Python-Dev]

On 29 Mar 2022, at 00:34, Brett Cannon <br...@python.org> wrote:

On Mon, Mar 28, 2022 at 11:52 AM Christopher Barker <pyth...@gmail.com> wrote:

On Mon, Mar 28, 2022 at 11:29 AM Paul Moore <p.f....@gmail.com> wrote:

To be honest, I feel like I'm just reiterating stuff I've said before
here, and I think the same is true of the points I'm responding to

 ...

 (I'm not *against* going over the debate again,
it helps make sure people haven't changed their minds, but it's
important to be clear that none of the practical facts have changed,
if that is the case).

Maybe there's a way to make this discussion (it feels more like a discussion than debate at the moment) more productive by writing some things down. I'm not sure it's a PEP, but some kind of:

"policy for the stdlib" document in which we could capture the primary points of view, places where there's consensus, etc. would be helpful to keep us retreading this over and over again.

I suggest this without the bandwidth to actually shepherd the project, but if someone wants to, I think it would be a great idea.

Once https://mail.python.org/archives/list/python-c...@python.org/thread/5EUZLT5PNA4HT42NGB5WVN5YWW5ASTT5/ is considered resolved, the next part of my "what is the stdlib" plan is to finally try to suss all of this out and more-or-less write a stdlib policy PEP so we stop talking about this. My guess it will be more of guidance about what we want the stdlib to be and thus guide what things belong in it. No ETA on that work since I also have four other big Python projects on the go right now whose work I am constantly alternating between.

Having such a policy is a good thing and helps in evolving the stdlib, but I wonder if the lack of such a document is the real problem.   IMHO the main problem is that the CPython team is very small and therefore has little bandwidth for maintaining, let alone evolving, large parts of the stdlib.  In that it doesn’t help that some parts of the stdlib have APIs that make it hard to make modifications (such as distutils where effectively everything is part of the public API).  Shrinking the stdlib helps in the maintenance burden, but feels as a partial solution. 

That said, I have no ideas on how a better stdlib development  proces would look like, let alone on how to get there.

Ronald

-Brett

 

-CHB

-- 

Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/GAZAFRFVJVQZMEIHTQUJASP7VRAKA5RR/
Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/PC67DOLDEQXIAGXEB2QXCGS3C4B6PTCY/

Code of Conduct: http://python.org/psf/codeofconduct/

—

Twitter / micro.blog: @ronaldoussoren

Blog: https://blog.ronaldoussoren.net/

[Brett Cannon]

On Tue, Mar 29, 2022 at 8:58 AM Ronald Oussoren <ronaldo...@mac.com> wrote:

On 29 Mar 2022, at 00:34, Brett Cannon <br...@python.org> wrote:

On Mon, Mar 28, 2022 at 11:52 AM Christopher Barker <pyth...@gmail.com> wrote:

On Mon, Mar 28, 2022 at 11:29 AM Paul Moore <p.f....@gmail.com> wrote:

To be honest, I feel like I'm just reiterating stuff I've said before
here, and I think the same is true of the points I'm responding to

 ...

 (I'm not *against* going over the debate again,
it helps make sure people haven't changed their minds, but it's
important to be clear that none of the practical facts have changed,
if that is the case).

Maybe there's a way to make this discussion (it feels more like a discussion than debate at the moment) more productive by writing some things down. I'm not sure it's a PEP, but some kind of:

"policy for the stdlib" document in which we could capture the primary points of view, places where there's consensus, etc. would be helpful to keep us retreading this over and over again.

I suggest this without the bandwidth to actually shepherd the project, but if someone wants to, I think it would be a great idea.

Once https://mail.python.org/archives/list/python-c...@python.org/thread/5EUZLT5PNA4HT42NGB5WVN5YWW5ASTT5/ is considered resolved, the next part of my "what is the stdlib" plan is to finally try to suss all of this out and more-or-less write a stdlib policy PEP so we stop talking about this. My guess it will be more of guidance about what we want the stdlib to be and thus guide what things belong in it. No ETA on that work since I also have four other big Python projects on the go right now whose work I am constantly alternating between.

Having such a policy is a good thing and helps in evolving the stdlib, but I wonder if the lack of such a document is the real problem.   IMHO the main problem is that the CPython team is very small and therefore has little bandwidth for maintaining, let alone evolving, large parts of the stdlib.  In that it doesn’t help that some parts of the stdlib have APIs that make it hard to make modifications (such as distutils where effectively everything is part of the public API).  Shrinking the stdlib helps in the maintenance burden, but feels as a partial solution. 

You're right that is the fundamental problem. But for me this somewhat stems from the fact that we don't have a shared understanding of what the stdlib is,  and so the stdlib is a bit unbounded in its size and scope. That leads to a stdlib which is hard to maintain. It's just like dealing with any scarce resource: you try to cut back on your overall use as best as you can and then become more efficient with what you must still consume; I personally think we don't have an answer to the "must consume" part of that sentence that leads us to "cut back" to a size we can actually keep maintained so we don't have 1.6K open PRs.

That said, I have no ideas on how a better stdlib development  proces would look like, let alone on how to get there.

I did what I could when I helped get us over to GitHub and what Mariatta helped bring to our workflow.

[Skip Montanaro]

I was trying to think through how a "remote" stdlib might work. In the
process, I got to wondering if there are known "specialists" for
various current modules. Every now and then I still get assigned (or
at least made nosy) about something to do with the csv module. Is
there an official module-by-module list of maintainers?

I was thinking about this in the context of the urllib discussion.
Whether or not it might be a candidate to keep long term (as one of
the handful of modules required to build and test CPython), if there
are known maintainers of specific modules or packages, I think it
might be worthwhile to give them the chance to chime in.

Skip

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/NHUDBM2NBUIRSQOYSPBCOJIW45FCY2TY/

[Eric V. Smith]

On 3/29/2022 4:55 PM, Skip Montanaro wrote:
> I was trying to think through how a "remote" stdlib might work. In the
> process, I got to wondering if there are known "specialists" for
> various current modules. Every now and then I still get assigned (or
> at least made nosy) about something to do with the csv module. Is
> there an official module-by-module list of maintainers?

There's the CODEOWNERS file:
https://github.com/python/cpython/blob/main/.github/CODEOWNERS

Eric

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/EKNOIU54LPVM3HZJN6WZCTVQIZHOLSYL/

[Skip Montanaro]

There's the CODEOWNERS file:
https://github.com/python/cpython/blob/main/.github/CODEOWNERS

Thanks. Never would have thought there was such a thing. I was looking for files with "maintain" in them. Skimming it, it would seem that most of the stuff in Lib or Modules isn't really associated with a particular person or small group.

Skip

[Terry Reedy]

On 3/29/2022 4:55 PM, Skip Montanaro wrote:

> I was trying to think through how a "remote" stdlib might work. In the
> process, I got to wondering if there are known "specialists" for
> various current modules. Every now and then I still get assigned (or
> at least made nosy) about something to do with the csv module. Is
> there an official module-by-module list of maintainers?

https://devguide.python.org/experts/

> I was thinking about this in the context of the urllib discussion.
> Whether or not it might be a candidate to keep long term (as one of
> the handful of modules required to build and test CPython), if there
> are known maintainers of specific modules or packages, I think it
> might be worthwhile to give them the chance to chime in.

I believe someone can be listed without being a coredev with merge
privileges.

--
Terry Jan Reedy

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/NZXZMEX3RTJBYITYFQLDPDUTCS3EXQC6/

[Alex Waygood]

There's also the "Experts index" in the devguide: https://devguide.python.org/experts/#experts

Best, 

Alex

[Damian Shaw]

I'm probably overly stressing a well understood point here that urllib.parse is incredibly widely used and critical to many foundational libraries in Python. But I just came across this today that:

conda is migrating from urllib3 for parsing to urllib.parse: https://github.com/conda/conda/pull/11373

And not to mention urllib3 itself somewhat depends on urllib.parse: https://github.com/urllib3/urllib3/search?q=urllib.parse

Damian (he/him)

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/CN7SXNP2D4ML4U2OBIUC4DGQOKEYNSKE/

[Barney Gale]

On Mon, 28 Mar 2022 at 20:37, Steve Dower <steve...@python.org> wrote:

email doesn't fix bugs; maintainers fix bugs. Please let us know
*publicly* if you want to become the maintainer for a stdlib module and
then we can support them, but if nobody is willing/able/ready to care
for them it's irresponsible for us to keep shipping them to users. 

I'd like to become a maintainer for the pathlib module, if possible. I know the code and tests inside-out, and I've been wrestling the internals for past few Python releases. I check the bugs/PRs at least every week and help wherever I can.

I can also look after urllib if it would help. I have much less experience with that module vs pathlib, so I'll need to spend some time familiarizing myself with the code and the outstanding bug reports. But I do have plenty of experience with networking, HTTP, URLs, etc :-). I'll spend some time reading the bug tracker to start.

[Steve Dower]

On 30Mar2022 1124, Barney Gale wrote:
> I'd like to become a maintainer for the pathlib module, if possible. I
> know the code and tests inside-out, and I've been wrestling the
> internals for past few Python releases. I check the bugs/PRs at least
> every week and help wherever I can.

Antoine is still active, so if he's fine with that, I'm also supportive.
You're certainly familiar enough with that module.

> I can also look after urllib if it would help. I have much less
> experience with that module vs pathlib, so I'll need to spend some time
> familiarizing myself with the code and the outstanding bug reports. But
> I do have plenty of experience with networking, HTTP, URLs, etc :-).
> I'll spend some time reading the bug tracker to start.

Please! I skim the open issues occasionally, but definitely feel out of
my depth to decide on issues (e.g. should
urlparse("http://[::1]spam:80") raise? - I have literally no idea :) ),
but feel free to nosy me on issues you think you've figured out and I
can help confirm your logic and do merges. Or if another core dev has
been helping you, keep nosying them.

Cheers,
Steve

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/DRHN22EUHGYCEUHJ3GSMEYMM2BQCS3QR/

[Baptiste Carvello]

Le 28/03/2022 à 18:44, Steve Dower a écrit :
> I think to most people "batteries included" doesn't necessarily mean
> "standard library," with all that implies. It just means "it came with
> the first thing that I installed" (or alternatively, "at no additional
> charge").

A point I have not seen made, is that some uses really need *standard*
batteries.

In scientific research, you'll sometimes share a data-processing script
among a large group of non-computer-specialists, who hopefully all have
*some* Python+NumPy installed: it may be a full up-to-date Conda, or it
may be "a very old version that a former colleague installed for me
years ago and now I won't update it for fear I break it". You have to
cater to all versions.

It may be more glamorous to compete with Rust and JS for "best modern
language", than with Excel for "lowest common denominator for
calculation". Still, that's a use case where Python has historically
been strong, and it's one of the reasons it works so well in the
research world.

Hopefully this use case is not forgotten, even if those
non-computer-specialists users are less likely to be involved here in
core development.

Cheers,
Baptiste

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/V4QX4ZSHSUUMVQR3CRAEMBBR6N3WGVLM/

[Antoine Pitrou]

On Wed, 30 Mar 2022 12:03:58 +0100
Steve Dower <steve...@python.org> wrote:
> On 30Mar2022 1124, Barney Gale wrote:
> > I'd like to become a maintainer for the pathlib module, if possible. I
> > know the code and tests inside-out, and I've been wrestling the
> > internals for past few Python releases. I check the bugs/PRs at least
> > every week and help wherever I can.
>
> Antoine is still active, so if he's fine with that, I'm also supportive.
> You're certainly familiar enough with that module.

I'm entirely supportive. Huge thank you for working on pathlib!

Regards

Antoine.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/6N7YKZVCV6TLFIMWIUVRKFMJY34JUHYJ/

[Steve Dower]

On 30Mar2022 1132, Baptiste Carvello wrote:
> Le 28/03/2022 à 18:44, Steve Dower a écrit :
>> I think to most people "batteries included" doesn't necessarily mean
>> "standard library," with all that implies. It just means "it came with
>> the first thing that I installed" (or alternatively, "at no additional
>> charge").
>
> A point I have not seen made, is that some uses really need *standard*
> batteries.

I've certainly not forgotten it, it's just every time I try to make the
point it doesn't seem to help. So until I find a clear way to argue
that, yes, a standard "datetime" value makes sense, but no, a standard
"HTTP headers dictionary" is unnecessary, I'm avoiding bringing it up
again ;)

Cheers,
Steve

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/OCY7VGDIXJAQ55XAHOCYFZOBF6O4P4I4/

[Barney Gale]

On Wed, 30 Mar 2022 at 12:20, Antoine Pitrou <ant...@python.org> wrote:

On Wed, 30 Mar 2022 12:03:58 +0100
Steve Dower <steve...@python.org> wrote:
> On 30Mar2022 1124, Barney Gale wrote:
> > I'd like to become a maintainer for the pathlib module, if possible. I
> > know the code and tests inside-out, and I've been wrestling the
> > internals for past few Python releases. I check the bugs/PRs at least
> > every week and help wherever I can. 
>
> Antoine is still active, so if he's fine with that, I'm also supportive.
> You're certainly familiar enough with that module.

I'm entirely supportive. Huge thank you for working on pathlib!

Thank you Antoine, Steve. It's humbling to stand on your shoulders and thrilling to give back to Python. If I'm successful in my application I'll do my best to take good care of pathlib and hopefully urllib too.

These batteries ain't dead yet! :D

Best,

Barney

[Paul Moore]

On Wed, 30 Mar 2022 at 12:28, Steve Dower <steve...@python.org> wrote:
>
> On 30Mar2022 1132, Baptiste Carvello wrote:
> > Le 28/03/2022 à 18:44, Steve Dower a écrit :
> >> I think to most people "batteries included" doesn't necessarily mean
> >> "standard library," with all that implies. It just means "it came with
> >> the first thing that I installed" (or alternatively, "at no additional
> >> charge").
> >
> > A point I have not seen made, is that some uses really need *standard*
> > batteries.
>
> I've certainly not forgotten it, it's just every time I try to make the
> point it doesn't seem to help. So until I find a clear way to argue
> that, yes, a standard "datetime" value makes sense, but no, a standard
> "HTTP headers dictionary" is unnecessary, I'm avoiding bringing it up
> again ;)

I also strongly agree with this position, and do my best to argue it.
But as you say, it is hard to draw the line.

Here's some examples, which while not exhaustive, might give a flavour
of my views:

"Obviously" things that should be in the stdlib (basic data types or
algorithms):
re, datetime, collections, enum, dataclasses, decimal, fractions
(maybe borderline), statistics, itertools, functools, pathlib

Obvious stuff that's OS related:
os, io, threading, multiprocessing, concurrent.futures,
subprocess, asyncio, urllib.request

Core services:
sys, venv, sysconfig, warnings, atexit, gc, zipimport, runpy,
importlib, marshal, pickle

Standard development tools:
argparse, pdb, cprofile, typing (arguable in some ways)

Handling of key file and data formats:
struct, csv, json, tomllib, zlib, gzip, bz2 (borderline), zipfile,
tarfile, base64

Arguable cases of file formats, but with important use cases
xml (you may not like it, but it's still used a lot)
configparser (similarly, ini-style files are used a lot, for
better or worse)
email (this is an outlier, but the packaging metadata file format
uses it, so there's a bootstrapping issue if it's not in the stdlib)

I've missed a lot out, and I'm sure people will argue most of these,
but that's sort of the point - it's *hard* to come up with a blanket
statement on how we should approach "the stdlib", because there's so
much diversity in there. That's why my starting position is that we
should retain the stdlib - sure, we can remove (and add!) stuff, but
we should do so on a case by case basis, not based on some blanket
"minimise it because we don't have the manpower or we don't want to
support it" basis.

If writing and sharing small, one-off scripts that depended on
non-stdlib packages were *significantly* easier, I might revise my
position on some of the arguable cases. PEP 582 (Python local packages
directory) might make a big difference here. But at the moment, adding
a 3rd party dependency effectively alters what you're doing from
"writing a script I can share" to "developing an application" - and
that's a *big* step for many people. Quoting Nathaniel Smith from
https://mail.python.org/archives/list/distut...@python.org/message/YFJITQB37MZOPOFJJF3OAQOY4TOAFXYM/
(albeit in a slightly different context):

"""
This last point is important: as projects grow in complexity, and are
used by a wider audience, they often end up with fairly complex tool
specifications that have to be shared among a team.
"""

My argument is that relying only on the stdlib allows people putting
off the need to take that step up in complexity, and we shouldn't
dismiss how significant that is.

Paul

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/LYKBB6YVZYQTJHPVRR5YMWE5D6YNSENY/

[Nick Coghlan]

On Tue, 29 Mar 2022, 9:44 am Toshio Kuratomi, <a.ba...@gmail.com> wrote:

One thing about talking about "make urllib more like requests" that is different than any of the other libs, though, is that requests aims to be easier to use than anything else (which I note Chris Barker called out as why he wanted urllib to be more like it).  I think that's important to think about because i think ease of use is also the number one thing that the audience you talk about is also looking for.

Of course, figuring out whether an api like request's is actually easier to use than urllib or merely more featureful is open to debate. 

There's a concrete scope difference between the two:

* requests was written as a way to make HTTP(S) requests from Python, including authenticated JSON based API requests

* urllib was designed to work with a variety of URL schemas which may or may not support authentication and encryption (before the rise of the web and restrictive corporate firewalls helped HTTP become the de facto standard for data exchange over the public internet)

I wrote more on that topic back in 2016 [1], and the gist of that article still holds true: urllib is OK for the basic HTTP GET use case, but having to tell urllib that you want to use HTTP-only features exposes a lot of low level plumbing that end users ideally wouldn't have to care about.

Unfortunately, it isn't clear where a useful subset of a full-fledged HTTP client library lies. "Make it easy to send a TLS protected JSON API POST request with Basic authentication and check the status code of the response" *might* qualify, but even that isn't particularly easy to define or maintain (What about proxies? What about redirects? What about other authentication methods?).

Thus the status quo persists - adding a fully featured HTTP client library to the existing stdlib maintenance burden doesn't (or at least shouldn't) strike *anyone* as a good idea, but it also isn't clear how much of the ease of use of the existing 3rd party libraries comes from the sheer amount of functionality they have built-in so users don't have to worry about it themselves, which means defining something "lighter and easier to maintain than a fully featured HTTP client library, but still easier to use than calling in to urllib directly" isn't a viable option either.

At least with urllib present in the stdlib, it's *possible* to share "No dependency" recipes for crafting particular HTTP requests, even though the default recommendation remains to grab a higher level 3rd party library instead of rolling your own solution.

Cheers,

Nick.

[1] https://www.curiousefficiency.org/posts/2016/08/what-problem-does-it-solve.html

-toshio

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/4ZQC4H7HD3UXFT3CONU64YPOQBSPUTVY/

[Gyro Funch]

On 3/30/2022 12:32 PM, Baptiste Carvello wrote:
> Le 28/03/2022 à 18:44, Steve Dower a écrit :
>> I think to most people "batteries included" doesn't necessarily mean
>> "standard library," with all that implies. It just means "it came with
>> the first thing that I installed" (or alternatively, "at no additional
>> charge").
>
> A point I have not seen made, is that some uses really need *standard*
> batteries.
>
> In scientific research, you'll sometimes share a data-processing script
> among a large group of non-computer-specialists, who hopefully all have
> *some* Python+NumPy installed: it may be a full up-to-date Conda, or it
> may be "a very old version that a former colleague installed for me
> years ago and now I won't update it for fear I break it". You have to
> cater to all versions.
>
> It may be more glamorous to compete with Rust and JS for "best modern
> language", than with Excel for "lowest common denominator for
> calculation". Still, that's a use case where Python has historically
> been strong, and it's one of the reasons it works so well in the
> research world.
>
> Hopefully this use case is not forgotten, even if those
> non-computer-specialists users are less likely to be involved here in
> core development.
>
> Cheers,
> Baptiste

This is a use case in which I am very interested. Reproducibility is
extremely important in my field as well. However, I don't think that any
solution offered by the standard python distribution will serve this
need, while still allowing the language to evolve.
The conda virtual environment mechanism, with carefully specified
package versions is a useful approach, but does not cover differences in
results owing to the user's OS, architecture, libraries, etc.
Among the alternatives, solutions based on containers are probably the
best bet when reproducibility is essential.

-gyro

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/2DGEWKHKKSHSYYFJBRJUO7VLUMOW456Z/

[Toshio Kuratomi]

On Tue, Mar 29, 2022, 10:55 AM Brett Cannon <br...@python.org> wrote:

On Tue, Mar 29, 2022 at 8:58 AM Ronald Oussoren <ronaldo...@mac.com> wrote:

On 29 Mar 2022, at 00:34, Brett Cannon <br...@python.org> wrote:

On Mon, Mar 28, 2022 at 11:52 AM Christopher Barker <pyth...@gmail.com> wrote:

On Mon, Mar 28, 2022 at 11:29 AM Paul Moore <p.f....@gmail.com> wrote:

Having such a policy is a good thing and helps in evolving the stdlib, but I wonder if the lack of such a document is the real problem.   IMHO the main problem is that the CPython team is very small and therefore has little bandwidth for maintaining, let alone evolving, large parts of the stdlib.  In that it doesn’t help that some parts of the stdlib have APIs that make it hard to make modifications (such as distutils where effectively everything is part of the public API).  Shrinking the stdlib helps in the maintenance burden, but feels as a partial solution. 

You're right that is the fundamental problem. But for me this somewhat stems from the fact that we don't have a shared understanding of what the stdlib is,  and so the stdlib is a bit unbounded in its size and scope. That leads to a stdlib which is hard to maintain. It's just like dealing with any scarce resource: you try to cut back on your overall use as best as you can and then become more efficient with what you must still consume; I personally think we don't have an answer to the "must consume" part of that sentence that leads us to "cut back" to a size we can actually keep maintained so we don't have 1.6K open PRs.

One of the things that's often missed in discussions is that a *good* policy document can also help grow the number of maintainers.

As just one example, i found two interesting items in the discussion started by Skip about determining what modules don't have maintainers just downstream if this. (1) There's a file which matches maintainers to modules in the stdlib (this is documented but i only found out about it a few years ago and Skip, who's been around even longer than me didn't know about it either... So this means something about how our policy docs are currently structured could be improved).  (2) Terry brought up that you don't have to be a core maintainer in order to take up ownership of something in the stdlib. That's awesome!  But this is definitely something i didn't know. I've been "focusing"[1] on  becoming a core maintainer because i thought that was a prerequisite to getting anything done in the stdlib. Knowing that getting involved with stdlib maintenance is different could be vastly helpful.

[1] focusing is the wrong word... It expresses the feeling of "directed action" correctly but doesn't convey the lack of activity that sprinkles my attempts.  Nor does it account for discouragement, helplessness, and imposter-y feelings which are the reasons for that lack.

-toshio

[Brett Cannon]

The first step is to get triage permissions: https://devguide.python.org/triaging/?highlight=triage#python-triage-team . We can't add anyone to the CODEOWNERS file unless you have write permissions, which means becoming a core dev. I have added myself to CODEOWNERS for pathlib and I will try to remember to @ mention you on things, Barney (https://github.com/python/cpython/pull/32202).

Antoine, I left you out of my PR for CODEOWNERS just because I thought it was presumptuous to add anyone to more GitHub notifications without their permission.

 

These batteries ain't dead yet! :D

Definitely not all of them. 😉

-Brett

 

Best,

Barney

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/RWVLBPUYNCLCJUYKFR2GWU3V5IT55UCG/

[Sasha Kacanski]

---------- Forwarded message ---------
From: Sasha Kacanski <skac...@gmail.com>
Date: Wed, Mar 30, 2022, 2:55 PM
Subject: Re: [Python-Dev] Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module]
To: Paul Moore <p.f....@gmail.com>
Cc: Christopher Barker <pyth...@gmail.com>, Ethan Furman <et...@stoneleaf.us>, Python Dev <pytho...@python.org>

Agree with Paul, 

Best of python always was, you can do almost everything with standard libs, but then novice becomes better developer and starts exploring libs and tools that wrap stuff or perform better or has functionality that stdlib does not support right of the bet they start using variety of libs that suits their needs.

Regards,

Sasha

On Sun, Mar 27, 2022, 8:06 PM Paul Moore <p.f....@gmail.com> wrote:

On Sun, 27 Mar 2022 at 17:11, Christopher Barker <pyth...@gmail.com> wrote:
>

> With the json package included, all they need to do is `import json`. If that wasn't there, they's look in PyPi for a JSON implementation, and find an absolutely astonishing number of options. I just did a search for "JSON"
>  on PYPI, and it's HUGE -- most of them are for specialized JSON-using protocols of some sort. I was actually really surprised that couple I know about of the top of my head (ujson, orjson) are actually hard to find.
>
> "You can just pip install it" is really not a good solution.

>
> In fact, this is an example, I think, of where we should put some effort into making the included batteries better -- it's great to have a JSON lib built in, but it's too bad that it's not best-of-bread by pretty much any definition (speed, memory performance, flexibility) -- there are quite a few feature requests open for it -- it would be nice to actually implement some of those. (but yes, that's a lot of work that someone(s) would have to do)
>
> Back to the topic at hand, rather than remove urllib, maybe it could be made better -- an as-easy-to-use-as-requests package in the stdlib would be really great.

I think that's where the mistake happens, though. Someone who needs
"best of breed" is motivated (and likely knowledgeable enough) to make
informed decisions about what's on PyPI. But someone who just wants to
get the job done probably doesn't - and that's the audience for the
stdlib. A stdlib module needs to be a good, reliable set of basic
functionality that non-experts can use successfully. There can be
better libraries on PyPI, but that doesn't mean the stdlib module is
unnecessary, nor does it mean that the stdlib has to match the PyPI
library feature for feature.

So here, specifically, I'd rather see urlllib be the best urlllib it
can be, and not demand that it turn into requests. Requests is there
if people need/want it (as is httpx, and urllib3, and aiohttp). But
urllib is for people who want to get a file from the web, and *not*
have to deal with dependencies, 3rd party libraries, etc.

The "batteries included" standard library and PyPI complement each
other. Neither is redundant, and neither implies the other is
unnecessary.

Paul

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/E2FGXTYOEYLC6GSJGMWBQKHOO62LZPHQ/

[Skip Montanaro]

On Wed, Mar 30, 2022, 12:02 PM Toshio Kuratomi <a.ba...@gmail.com> wrote:

As just one example, i found two interesting items in the discussion started by Skip about determining what modules don't have maintainers just downstream if this.

Age in snake years doesn't necessarily correlate well with one's desire to take a deep dive into the documentation. Just sayin'... <wink> These answers might have been there waiting for a more diligent search on my part.

Skip

[Jeremy Kloth]

On Tue, Mar 29, 2022 at 11:55 AM Brett Cannon <br...@python.org> wrote:
> You're right that is the fundamental problem. But for me this somewhat stems from the fact that we don't have a shared understanding of what the stdlib is, and so the stdlib is a bit unbounded in its size and scope. That leads to a stdlib which is hard to maintain. It's just like dealing with any scarce resource: you try to cut back on your overall use as best as you can and then become more efficient with what you must still consume; I personally think we don't have an answer to the "must consume" part of that sentence that leads us to "cut back" to a size we can actually keep maintained so we don't have 1.6K open PRs.

This sent me down a rabbit hole of trying to analyze the PRs to see
where things are failing. Using a couple libraries from PyPI
(pygithub and gitignore_parser) I've mined out some interesting
things. Some of the lines in CODEOWNERS are not matching what people
think they are matching so notifications are being missed. The
"partial matches" are PRs which have at least one matching file in
CODEOWNERS so I think that those people should have been notified as
well as those where the entire changeset matches via CODEOWNERS.

I'm still fine tuning the script, so if more detail is desired (or
some other ignorable entries like /Misc/NEWS.d/ or /**/clinic/*.h)
please give feedback.

So with that, here are some results:

== PR Totals ========
core developers: 281
triage team: 89
draft PRs: 37
CODEOWNERS: 139
partial matches: 653
orphans: 396
-----
Total PRs: 1595

Of the files changed in all of the PRs, the following that do not
match via CODEOWNERS:
-- Core (/Grammar/, /Include/, /Objects/, /Python/) that should always
be maintained
Include/Python.h
Include/bltinmodule.h
Include/boolobject.h
Include/buffer.h
Include/bytes_methods.h
Include/ceval.h
Include/cpython/code.h
Include/cpython/dictobject.h
Include/cpython/fileutils.h
Include/cpython/object.h
Include/cpython/pyerrors.h
Include/cpython/pystate.h
Include/cpython/pythonrun.h
Include/cpython/sysmodule.h
Include/descrobject.h
Include/exports.h
Include/internal/pycore_atomic.h
Include/internal/pycore_bitutils.h
Include/internal/pycore_dict.h
Include/internal/pycore_format.h
Include/internal/pycore_global_strings.h
Include/internal/pycore_interp.h
Include/internal/pycore_object.h
Include/internal/pycore_pathconfig.h
Include/internal/pycore_pylifecycle.h
Include/internal/pycore_runtime.h
Include/internal/pycore_runtime_init.h
Include/internal/pycore_sysmodule.h
Include/methodobject.h
Include/modsupport.h
Include/moduleobject.h
Include/object.h
Include/obmalloc.h
Include/opcode.h
Include/opcode_names.h
Include/py_curses.h
Include/pydtrace.h
Include/pyerrors.h
Include/pymath.h
Include/pyport.h
Include/pyportosf.h
Include/pystate.h
Include/pystrtod.h
Include/pythread.h
Include/structmember.h
Objects/abstract.c
Objects/boolobject.c
Objects/bytearrayobject.c
Objects/bytes_methods.c
Objects/bytesobject.c
Objects/classobject.c
Objects/complexobject.c
Objects/descrobject.c
Objects/fileobject.c
Objects/funcobject.c
Objects/genericaliasobject.c
Objects/listobject.c
Objects/longobject.c
Objects/memoryobject.c
Objects/methodobject.c
Objects/moduleobject.c
Objects/namespaceobject.c
Objects/object.c
Objects/obmalloc.c
Objects/picklebufobject.c
Objects/rangeobject.c
Objects/sliceobject.c
Objects/stringlib/split.h
Objects/structseq.c
Objects/tupleobject.c
Objects/unicodectype.c
Objects/unicodeobject.c
Objects/unicodetype_db.h
Objects/unionobject.c
Objects/weakrefobject.c
Python/Python-ast.c
Python/_warnings.c
Python/bltinmodule.c
Python/ceval_gil.h
Python/codecs.c
Python/condvar.h
Python/dtoa.c
Python/errors.c
Python/fileutils.c
Python/formatter_unicode.c
Python/getargs.c
Python/hashtable.c
Python/importlib.h
Python/importlib_external.h
Python/initconfig.c
Python/makeopcodetargets.py
Python/modsupport.c
Python/opcode_targets.h
Python/pathconfig.c
Python/preconfig.c
Python/pylifecycle.c
Python/pystate.c
Python/pystrtod.c
Python/structmember.c
Python/symtable.c
Python/sysmodule.c
Python/thread_nt.h
Python/thread_pthread.h

-- Truly Orphaned
.azure-pipelines/ci.yml
.azure-pipelines/libffi-build.yml
.azure-pipelines/pr
.azure-pipelines/pr.yml
.azure-pipelines/windows-steps.yml
.github/workflows/build.yml
.github/workflows/build_msi.yml
.github/workflows/deploy-previews.yml
.github/workflows/doc.yml
.github/workflows/verify-bundled-wheels.yml
Doc/c-api/arg.rst
Doc/c-api/dict.rst
Doc/c-api/init.rst
Doc/c-api/intro.rst
Doc/c-api/long.rst
Doc/c-api/object.rst
Doc/c-api/structures.rst
Doc/c-api/type.rst
Doc/c-api/typeobj.rst
Doc/c-api/unicode.rst
Doc/conf.py
Doc/data/refcounts.dat
Doc/extending/embedding.rst
Doc/extending/extending.rst
Doc/extending/newtypes.rst
Doc/extending/newtypes_tutorial.rst
Doc/extending/windows.rst
Doc/glossary.rst
Doc/howto/functional.rst
Doc/includes/custom.c
Doc/includes/custom2.c
Doc/includes/custom3.c
Doc/includes/custom4.c
Doc/includes/sqlite3/blob.py
Doc/includes/sqlite3/blob_with.py
Doc/includes/sublist.c
Doc/library/2to3.rst
Doc/library/__main__.rst
Doc/library/abc.rst
Doc/library/aifc.rst
Doc/library/argparse.rst
Doc/library/array.rst
Doc/library/asyncio-tutorial/async-functions.rst
Doc/library/asyncio-tutorial/case-study-chat-client-cli.rst
Doc/library/asyncio-tutorial/case-study-chat-server.rst
Doc/library/asyncio-tutorial/client05.py
Doc/library/asyncio-tutorial/index.rst
Doc/library/asyncio-tutorial/pttest.py
Doc/library/asyncio-tutorial/running-async-functions.rst
Doc/library/asyncio-tutorial/server01.py
Doc/library/asyncio-tutorial/server02.py
Doc/library/asyncio-tutorial/server03.py
Doc/library/asyncio-tutorial/server04.py
Doc/library/asyncio-tutorial/server05.py
Doc/library/asyncio-tutorial/server20.py
Doc/library/asyncio-tutorial/utils01.py
Doc/library/asyncio-tutorial/utils20.py
Doc/library/base64.rst
Doc/library/bdb.rst
Doc/library/cmd.rst
Doc/library/code.rst
Doc/library/codecs.rst
Doc/library/concurrent.futures.rst
Doc/library/configparser.rst
Doc/library/crypt.rst
Doc/library/csv.rst
Doc/library/ctypes.rst
Doc/library/curses.ascii.rst
Doc/library/curses.panel.rst
Doc/library/curses.rst
Doc/library/difflib.rst
Doc/library/dis.rst
Doc/library/distutils.rst
Doc/library/doctest.rst
Doc/library/ensurepip.rst
Doc/library/errno.rst
Doc/library/exceptions.rst
Doc/library/fileinput.rst
Doc/library/formatter.rst
Doc/library/fractions.rst
Doc/library/ftplib.rst
Doc/library/functions.rst
Doc/library/getpass.rst
Doc/library/gettext.rst
Doc/library/gzip.rst
Doc/library/http.client.rst
Doc/library/http.cookiejar.rst
Doc/library/http.cookies.rst
Doc/library/http.rst
Doc/library/http.server.rst
Doc/library/idle.rst
Doc/library/importlib.metadata.rst
Doc/library/importlib.rst
Doc/library/inspect.rst
Doc/library/io.rst
Doc/library/ipaddress.rst
Doc/library/locale.rst
Doc/library/math.rst
Doc/library/mmap.rst
Doc/library/multiprocessing.rst
Doc/library/netrc.rst
Doc/library/numbers.rst
Doc/library/operator.rst
Doc/library/os.path.rst
Doc/library/os.rst
Doc/library/pathlib.rst
Doc/library/pdb.rst
Doc/library/pickle.rst
Doc/library/pkgutil.rst
Doc/library/platform.rst
Doc/library/plistlib.rst
Doc/library/profile.rst
Doc/library/pyexpat.rst
Doc/library/re.rst
Doc/library/reprlib.rst
Doc/library/rlcompleter.rst
Doc/library/runpy.rst
Doc/library/sched.rst
Doc/library/secrets.rst
Doc/library/selectors.rst
Doc/library/shlex.rst
Doc/library/shutil.rst
Doc/library/signal.rst
Doc/library/site.rst
Doc/library/socket.rst
Doc/library/socketserver.rst
Doc/library/ssl.rst
Doc/library/stat.rst
Doc/library/stdtypes.rst
Doc/library/string.rst
Doc/library/struct.rst
Doc/library/subprocess.rst
Doc/library/sunau.rst
Doc/library/sys.rst
Doc/library/telnetlib.rst
Doc/library/tempfile.rst
Doc/library/test.rst
Doc/library/textwrap.rst
Doc/library/timeit.rst
Doc/library/tkinter.dnd.rst
Doc/library/tkinter.font.rst
Doc/library/tkinter.rst
Doc/library/trace.rst
Doc/library/traceback.rst
Doc/library/tty.rst
Doc/library/unicodedata.rst
Doc/library/unittest.mock-examples.rst
Doc/library/unittest.mock.rst
Doc/library/unittest.rst
Doc/library/urllib.parse.rst
Doc/library/urllib.request.rst
Doc/library/warnings.rst
Doc/library/wave.rst
Doc/library/weakref.rst
Doc/library/winreg.rst
Doc/library/wsgiref.rst
Doc/library/xml.dom.pulldom.rst
Doc/library/xml.etree.elementtree.rst
Doc/library/xml.sax.handler.rst
Doc/library/zipfile.rst
Doc/library/zlib.rst
Doc/license.rst
Doc/reference/compound_stmts.rst
Doc/reference/datamodel.rst
Doc/reference/executionmodel.rst
Doc/reference/expressions.rst
Doc/reference/import.rst
Doc/reference/lexical_analysis.rst
Doc/reference/simple_stmts.rst
Doc/requirements.txt
Doc/tools/extensions/pyspecific.py
Doc/tools/extensions/unidata_version.py
Doc/tutorial/appendix.rst
Doc/tutorial/errors.rst
Doc/tutorial/inputoutput.rst
Doc/tutorial/interactive.rst
Doc/tutorial/modules.rst
Doc/tutorial/stdlib.rst
Doc/using/cmdline.rst
Doc/using/windows.rst
Lib/__init__.py
Lib/_dummy_thread.py
Lib/_py_abc.py
Lib/_pyio.py
Lib/_weakrefset.py
Lib/abc.py
Lib/aifc.py
Lib/argparse.py
Lib/asyncio/__main__.py
Lib/asyncio/base_events.py
Lib/asyncio/base_subprocess.py
Lib/asyncio/events.py
Lib/asyncio/futures.py
Lib/asyncio/locks.py
Lib/asyncio/proactor_events.py
Lib/asyncio/runners.py
Lib/asyncio/selector_events.py
Lib/asyncio/sslproto.py
Lib/asyncio/streams.py
Lib/asyncio/subprocess.py
Lib/asyncio/tasks.py
Lib/asyncio/unix_events.py
Lib/asyncio/windows_events.py
Lib/asyncore.py
Lib/base64.py
Lib/bdb.py
Lib/cProfile.py
Lib/cmd.py
Lib/collections/__init__.py
Lib/compileall.py
Lib/concurrent/futures/_base.py
Lib/concurrent/futures/process.py
Lib/concurrent/futures/thread.py
Lib/configparser.py
Lib/crypt.py
Lib/csv.py
Lib/ctypes/__init__.py
Lib/ctypes/_aix.py
Lib/ctypes/macholib/dyld.py
Lib/ctypes/macholib/dylib.py
Lib/ctypes/macholib/framework.py
Lib/ctypes/test/test_callbacks.py
Lib/ctypes/test/test_find.py
Lib/ctypes/test/test_functions.py
Lib/ctypes/test/test_loading.py
Lib/ctypes/test/test_macholib.py
Lib/ctypes/test/test_pep3118.py
Lib/ctypes/test/test_python_api.py
Lib/ctypes/test/test_refcounts.py
Lib/ctypes/test/test_structures.py
Lib/ctypes/test/test_win32.py
Lib/ctypes/util.py
Lib/curses/textpad.py
Lib/difflib.py
Lib/dis.py
Lib/distutils/_msvccompiler.py
Lib/distutils/ccompiler.py
Lib/distutils/command/build_ext.py
Lib/distutils/tests/test_build_ext.py
Lib/distutils/tests/test_check.py
Lib/distutils/tests/test_dist.py
Lib/distutils/tests/test_sdist.py
Lib/distutils/tests/test_sysconfig.py
Lib/distutils/version.py
Lib/doctest.py
Lib/email/_header_value_parser.py
Lib/email/contentmanager.py
Lib/email/feedparser.py
Lib/email/generator.py
Lib/email/header.py
Lib/email/message.py
Lib/email/mime/signed.py
Lib/email/mime/text.py
Lib/email/parser.py
Lib/email/policy.py
Lib/email/utils.py
Lib/encodings/aliases.py
Lib/encodings/cp1250.py
Lib/encodings/cp1251.py
Lib/encodings/cp1252.py
Lib/encodings/cp1253.py
Lib/encodings/cp1254.py
Lib/encodings/cp1255.py
Lib/encodings/cp1256.py
Lib/encodings/cp1257.py
Lib/encodings/cp1258.py
Lib/encodings/cp874.py
Lib/encodings/idna.py
Lib/encodings/punycode.py
Lib/encodings/undefined.py
Lib/encodings/utf_16.py
Lib/encodings/utf_32.py
Lib/ensurepip/__init__.py
Lib/ensurepip/_bundled/.gitignore
Lib/ensurepip/_bundled/pip-19.0.3-py2.py3-none-any.whl
Lib/ensurepip/_bundled/setuptools-40.8.0-py2.py3-none-any.whl
Lib/ensurepip/_bundler.py
Lib/ensurepip/bundle.py
Lib/fileinput.py
Lib/fnmatch.py
Lib/fractions.py
Lib/ftplib.py
Lib/genericpath.py
Lib/getpass.py
Lib/gettext.py
Lib/gzip.py
Lib/http/__init__.py
Lib/http/client.py
Lib/http/cookiejar.py
Lib/http/cookies.py
Lib/http/server.py
Lib/idlelib/Icons/debug_current.gif
Lib/idlelib/Icons/debug_go.gif
Lib/idlelib/Icons/debug_go_disabled.gif
Lib/idlelib/Icons/debug_line.gif
Lib/idlelib/Icons/debug_out.gif
Lib/idlelib/Icons/debug_out_disabled.gif
Lib/idlelib/Icons/debug_over.gif
Lib/idlelib/Icons/debug_over_disabled.gif
Lib/idlelib/Icons/debug_prefs.gif
Lib/idlelib/Icons/debug_prefs_disabled.gif
Lib/idlelib/Icons/debug_step.gif
Lib/idlelib/Icons/debug_step_disabled.gif
Lib/idlelib/Icons/debug_stop.gif
Lib/idlelib/Icons/debug_stop_disabled.gif
Lib/idlelib/ParenClose.py
Lib/idlelib/README.txt
Lib/idlelib/config-extensions.def
Lib/idlelib/config-keys.def
Lib/idlelib/config-main.def
Lib/idlelib/config.py
Lib/idlelib/configdialog.py
Lib/idlelib/debugger.py
Lib/idlelib/debugger_r.py
Lib/idlelib/dynoption.py
Lib/idlelib/editor.py
Lib/idlelib/filelist.py
Lib/idlelib/help.html
Lib/idlelib/idle_test/htest.py
Lib/idlelib/idle_test/test_calltip.py
Lib/idlelib/idle_test/test_config.py
Lib/idlelib/idle_test/test_configdialog.py
Lib/idlelib/idle_test/test_debugger.py
Lib/idlelib/idle_test/test_editor.py
Lib/idlelib/idle_test/test_parenclose.py
Lib/idlelib/idle_test/test_query.py
Lib/idlelib/idle_test/test_scrolledlist.py
Lib/idlelib/idle_test/test_textview.py
Lib/idlelib/iomenu.py
Lib/idlelib/mainmenu.py
Lib/idlelib/navigatebar.py
Lib/idlelib/pyshell.py
Lib/idlelib/query.py
Lib/idlelib/replace.py
Lib/idlelib/run.py
Lib/idlelib/scrolledlist.py
Lib/idlelib/search.py
Lib/idlelib/textview.py
Lib/imghdr.py
Lib/importlib/_bootstrap_external.py
Lib/importlib/abc.py
Lib/inspect.py
Lib/ipaddress.py
Lib/json/__init__.py
Lib/json/decoder.py
Lib/json/encoder.py
Lib/json/scanner.py
Lib/json/tool.py
Lib/lib2to3/tests/support.py
Lib/lib2to3/tests/test_parser.py
Lib/linecache.py
Lib/locale.py
Lib/logging/__init__.py
Lib/logging/config.py
Lib/logging/handlers.py
Lib/modulefinder.py
Lib/multiprocessing/__init__.py
Lib/multiprocessing/connection.py
Lib/multiprocessing/dummy/__init__.py
Lib/multiprocessing/forkserver.py
Lib/multiprocessing/heap.py
Lib/multiprocessing/managers.py
Lib/multiprocessing/pool.py
Lib/multiprocessing/popen_fork.py
Lib/multiprocessing/popen_forkserver.py
Lib/multiprocessing/popen_spawn_posix.py
Lib/multiprocessing/popen_spawn_win32.py
Lib/multiprocessing/process.py
Lib/multiprocessing/reduction.py
Lib/multiprocessing/resource_tracker.py
Lib/multiprocessing/spawn.py
Lib/multiprocessing/synchronize.py
Lib/multiprocessing/util.py
Lib/netrc.py
Lib/ntpath.py
Lib/numbers.py
Lib/opcode.py
Lib/os.py
Lib/pathlib.py
Lib/pdb.py
Lib/pickle.py
Lib/pickletools.py
Lib/pkgutil.py
Lib/platform.py
Lib/plistlib.py
Lib/pprint.py
Lib/profile.py
Lib/pstats.py
Lib/pty.py
Lib/pyclbr.py
Lib/pydoc.py
Lib/pydoc_data/topics.py
Lib/quopri.py
Lib/re.py
Lib/reprlib.py
Lib/rlcompleter.py
Lib/runpy.py
Lib/sched.py
Lib/selectors.py
Lib/shelve.py
Lib/shlex.py
Lib/shutil.py
Lib/site.py
Lib/sndhdr.py
Lib/socket.py
Lib/socketserver.py
Lib/sqlite3/dbapi2.py
Lib/sqlite3/dump.py
Lib/sqlite3/test/dbapi.py
Lib/sqlite3/test/regression.py
Lib/sqlite3/test/test_dump.py
Lib/sqlite3/test/test_namedrow.py
Lib/sre_compile.py
Lib/sre_constants.py
Lib/ssl.py
Lib/stat.py
Lib/string.py
Lib/sunau.py
Lib/sysconfig.py
Lib/tabnanny.py
Lib/telnetlib.py
Lib/tempfile.py
Lib/test/_test_multiprocessing.py
Lib/test/_typed_dict_helper.py
Lib/test/ann_module.py
Lib/test/audiodata/pluck-pcm24-ext.wav
Lib/test/audiotests.py
Lib/test/audit-tests.py
Lib/test/clinic.test
Lib/test/dataclass_forward_ref.py
Lib/test/dataclass_forward_ref_child.py
Lib/test/dataclass_forward_ref_str.py
Lib/test/final_a.py
Lib/test/final_b.py
Lib/test/imghdrdata/python.jpg
Lib/test/imghdrdata/python2.jpeg
Lib/test/inspect_fodder.py
Lib/test/libregrtest/refleak.py
Lib/test/libregrtest/runtest.py
Lib/test/libregrtest/win_utils.py
Lib/test/list_tests.py
Lib/test/mod_generics_cache.py
Lib/test/pickletester.py
Lib/test/sndhdrdata/input1.bad
Lib/test/sndhdrdata/input2.bad
Lib/test/string_tests.py
Lib/test/support/__init__.py
Lib/test/test__opcode.py
Lib/test/test__xxsubinterpreters.py
Lib/test/test_abc.py
Lib/test/test_aifc.py
Lib/test/test_argparse.py
Lib/test/test_array.py
Lib/test/test_ast.py
Lib/test/test_asyncgen.py
Lib/test/test_asyncio/test_base_events.py
Lib/test/test_asyncio/test_events.py
Lib/test/test_asyncio/test_futures.py
Lib/test/test_asyncio/test_futures2.py
Lib/test/test_asyncio/test_proactor_events.py
Lib/test/test_asyncio/test_selector_events.py
Lib/test/test_asyncio/test_sslproto.py
Lib/test/test_asyncio/test_streams.py
Lib/test/test_asyncio/test_subprocess.py
Lib/test/test_asyncio/test_tasks.py
Lib/test/test_asyncio/test_unix_events.py
Lib/test/test_asyncio/test_unix_pipes.py
Lib/test/test_asyncio/utils.py
Lib/test/test_audioop.py
Lib/test/test_audit.py
Lib/test/test_base64.py
Lib/test/test_bdb.py
Lib/test/test_buffer.py
Lib/test/test_builtin.py
Lib/test/test_bytes.py
Lib/test/test_bz2.py
Lib/test/test_c_locale_coercion.py
Lib/test/test_calendar.py
Lib/test/test_capi.py
Lib/test/test_cmd.py
Lib/test/test_cmd_line.py
Lib/test/test_cmd_line_script.py
Lib/test/test_codecs.py
Lib/test/test_compileall.py
Lib/test/test_concurrent_futures.py
Lib/test/test_configparser.py
Lib/test/test_coroutines.py
Lib/test/test_cprofile.py
Lib/test/test_crypt.py
Lib/test/test_csv.py
Lib/test/test_curses.py
Lib/test/test_decorators.py
Lib/test/test_descr.py
Lib/test/test_dis.py
Lib/test/test_doctest.py
Lib/test/test_dynamicclassattribute.py
Lib/test/test_email/test__header_value_parser.py
Lib/test/test_email/test_contentmanager.py
Lib/test/test_email/test_defect_handling.py
Lib/test/test_email/test_generator.py
Lib/test/test_email/test_headerregistry.py
Lib/test/test_email/test_message.py
Lib/test/test_email/test_multipart_signed.py
Lib/test/test_email/test_parser.py
Lib/test/test_email/test_pickleable.py
Lib/test/test_email/test_policy.py
Lib/test/test_embed.py
Lib/test/test_ensurepip.py
Lib/test/test_faulthandler.py
Lib/test/test_fileinput.py
Lib/test/test_fileio.py
Lib/test/test_float.py
Lib/test/test_fnmatch.py
Lib/test/test_format.py
Lib/test/test_fractions.py
Lib/test/test_future.py
Lib/test/test_gc.py
Lib/test/test_gdb.py
Lib/test/test_genericalias.py
Lib/test/test_genericpath.py
Lib/test/test_getargs2.py
Lib/test/test_getpass.py
Lib/test/test_gettext.py
Lib/test/test_grp.py
Lib/test/test_gzip.py
Lib/test/test_hmac.py
Lib/test/test_http/test_http.py
Lib/test/test_http_cookiejar.py
Lib/test/test_http_cookies.py
Lib/test/test_httplib.py
Lib/test/test_httpservers.py
Lib/test/test_imghdr.py
Lib/test/test_import/__init__.py
Lib/test/test_importlib/extension/test_loader.py
Lib/test/test_importlib/fixtures.py
Lib/test/test_importlib/import_/test_path.py
Lib/test/test_importlib/test_abc.py
Lib/test/test_importlib/test_main.py
Lib/test/test_importlib/test_metadata_api.py
Lib/test/test_importlib/test_util.py
Lib/test/test_inspect.py
Lib/test/test_io.py
Lib/test/test_ipaddress.py
Lib/test/test_json/test_decode.py
Lib/test/test_json/test_dump.py
Lib/test/test_json/test_indent.py
Lib/test/test_json/test_separators.py
Lib/test/test_json/test_speedups.py
Lib/test/test_json/test_tool.py
Lib/test/test_linecache.py
Lib/test/test_list.py
Lib/test/test_lltrace.py
Lib/test/test_locale.py
Lib/test/test_lzma.py
Lib/test/test_memoryio.py
Lib/test/test_memoryview.py
Lib/test/test_minidom.py
Lib/test/test_mmap.py
Lib/test/test_module.py
Lib/test/test_modulefinder.py
Lib/test/test_multiprocessing_dummy.py
Lib/test/test_netrc.py
Lib/test/test_nntplib.py
Lib/test/test_ntpath.py
Lib/test/test_numeric_tower.py
Lib/test/test_os.py
Lib/test/test_pathlib.py
Lib/test/test_pdb.py
Lib/test/test_peg_parser.py
Lib/test/test_pkg.py
Lib/test/test_pkgutil.py
Lib/test/test_plistlib.py
Lib/test/test_posix.py
Lib/test/test_pprint.py
Lib/test/test_profile.py
Lib/test/test_property.py
Lib/test/test_pstats.py
Lib/test/test_pulldom.py
Lib/test/test_pyclbr.py
Lib/test/test_pydoc.py
Lib/test/test_pyexpat.py
Lib/test/test_range.py
Lib/test/test_re.py
Lib/test/test_regrtest.py
Lib/test/test_repl.py
Lib/test/test_reprlib.py
Lib/test/test_rlcompleter.py
Lib/test/test_selectors.py
Lib/test/test_set.py
Lib/test/test_shlex.py
Lib/test/test_shutil.py
Lib/test/test_signal.py
Lib/test/test_site.py
Lib/test/test_sndhdr.py
Lib/test/test_socket.py
Lib/test/test_socketserver.py
Lib/test/test_source_encoding.py
Lib/test/test_ssl.py
Lib/test/test_stat.py
Lib/test/test_str.py
Lib/test/test_string.py
Lib/test/test_struct.py
Lib/test/test_sunau.py
Lib/test/test_super.py
Lib/test/test_support.py
Lib/test/test_syntax.py
Lib/test/test_sys.py
Lib/test/test_sysconfig.py
Lib/test/test_tabnanny.py
Lib/test/test_telnetlib.py
Lib/test/test_tempfile.py
Lib/test/test_textwrap.py
Lib/test/test_thread.py
Lib/test/test_threading.py
Lib/test/test_timeit.py
Lib/test/test_tk.py
Lib/test/test_tokenize.py
Lib/test/test_tools/test_c_analyzer/test_cpython/test_supported.py
Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py
Lib/test/test_tools/test_c_analyzer/test_parser/test_preprocessor.py
Lib/test/test_tools/test_c_analyzer/test_variables/test_known.py
Lib/test/test_tools/test_i18n.py
Lib/test/test_tools/test_pindent.py
Lib/test/test_trace.py
Lib/test/test_ttk_guionly.py
Lib/test/test_ttk_textonly.py
Lib/test/test_tuple.py
Lib/test/test_types.py
Lib/test/test_unicode.py
Lib/test/test_unicodedata.py
Lib/test/test_unparse.py
Lib/test/test_urllib.py
Lib/test/test_urllib2.py
Lib/test/test_urllib2net.py
Lib/test/test_urlparse.py
Lib/test/test_userstring.py
Lib/test/test_utf8_mode.py
Lib/test/test_warnings/__init__.py
Lib/test/test_wave.py
Lib/test/test_weakref.py
Lib/test/test_winconsoleio.py
Lib/test/test_winreg.py
Lib/test/test_wsgiref.py
Lib/test/test_xml_etree.py
Lib/test/test_zipapp.py
Lib/test/test_zipfile.py
Lib/test/test_zlib.py
Lib/test/test_zoneinfo/test_zoneinfo.py
Lib/test/tkinter_test/README
Lib/test/tkinter_test/__init__.py
Lib/test/tkinter_test/runtktests.py
Lib/test/tkinter_test/support.py
Lib/test/tkinter_test/test_tkinter/__init__.py
Lib/test/tkinter_test/test_tkinter/test_font.py
Lib/test/tkinter_test/test_tkinter/test_geometry_managers.py
Lib/test/tkinter_test/test_tkinter/test_images.py
Lib/test/tkinter_test/test_tkinter/test_loadtk.py
Lib/test/tkinter_test/test_tkinter/test_misc.py
Lib/test/tkinter_test/test_tkinter/test_text.py
Lib/test/tkinter_test/test_tkinter/test_variables.py
Lib/test/tkinter_test/test_tkinter/test_widgets.py
Lib/test/tkinter_test/test_ttk/__init__.py
Lib/test/tkinter_test/test_ttk/test_extensions.py
Lib/test/tkinter_test/test_ttk/test_functions.py
Lib/test/tkinter_test/test_ttk/test_style.py
Lib/test/tkinter_test/test_ttk/test_widgets.py
Lib/test/tkinter_test/widget_tests.py
Lib/textwrap.py
Lib/threading.py
Lib/timeit.py
Lib/tkinter/__init__.py
Lib/tkinter/constants.py
Lib/tkinter/dnd.py
Lib/tkinter/simpledialog.py
Lib/tkinter/test/test_tkinter/test_event.py
Lib/tkinter/test/test_tkinter/test_misc.py
Lib/tkinter/test/test_tkinter/test_threads.py
Lib/tkinter/test/test_tkinter/test_tkdnd.py
Lib/tokenize.py
Lib/trace.py
Lib/tty.py
Lib/turtle.py
Lib/types.py
Lib/unittest/case.py
Lib/unittest/loader.py
Lib/unittest/main.py
Lib/unittest/result.py
Lib/unittest/runner.py
Lib/unittest/signals.py
Lib/unittest/suite.py
Lib/unittest/test/support.py
Lib/unittest/test/test_assertions.py
Lib/unittest/test/test_break.py
Lib/unittest/test/test_case.py
Lib/unittest/test/test_discovery.py
Lib/unittest/test/test_loader.py
Lib/unittest/test/test_program.py
Lib/unittest/test/test_result.py
Lib/unittest/test/test_runner.py
Lib/unittest/test/test_setups.py
Lib/unittest/test/test_suite.py
Lib/urllib/parse.py
Lib/urllib/request.py
Lib/uuid.py
Lib/venv/__init__.py
Lib/venv/scripts/common/Activate.ps1
Lib/venv/scripts/common/activate
Lib/venv/scripts/posix/activate.csh
Lib/venv/scripts/posix/activate.fish
Lib/warnings.py
Lib/wave.py
Lib/weakref.py
Lib/webbrowser.py
Lib/wsgiref/handlers.py
Lib/wsgiref/headers.py
Lib/wsgiref/simple_server.py
Lib/wsgiref/validate.py
Lib/xdrlib.py
Lib/xml/dom/expatbuilder.py
Lib/xml/dom/minidom.py
Lib/xml/dom/pulldom.py
Lib/xml/etree/ElementPath.py
Lib/xml/etree/ElementTree.py
Lib/xml/sax/xmlreader.py
Lib/xmlrpc/client.py
Lib/xmlrpc/server.py
Lib/zipapp.py
Lib/zipfile.py
Lib/zoneinfo/_common.py
Makefile.pre.in
Misc/ACKS
Misc/NEWS
Misc/README.valgrind
Misc/python-config.in
Misc/python-config.sh.in
Misc/python.man
Misc/python.pc.in
Misc/valgrind-python.supp
Misc/verify-bundled-wheels.sh
Modules/Makefile
Modules/Setup
Modules/_abc.c
Modules/_bz2module.c
Modules/_csv.c
Modules/_ctypes/_ctypes.c
Modules/_ctypes/_ctypes_test.c
Modules/_ctypes/callbacks.c
Modules/_ctypes/callproc.c
Modules/_ctypes/cfield.c
Modules/_ctypes/ctypes.h
Modules/_ctypes/stgdict.c
Modules/_cursesmodule.c
Modules/_decimal/libmpdec/io.c
Modules/_decimal/tests/formathelper.py
Modules/_elementtree.c
Modules/_hashopenssl.c
Modules/_io/_iomodule.c
Modules/_io/bufferedio.c
Modules/_io/bytesio.c
Modules/_io/fileio.c
Modules/_io/iobase.c
Modules/_io/textio.c
Modules/_io/winconsoleio.c
Modules/_json.c
Modules/_localemodule.c
Modules/_lsprof.c
Modules/_lzmamodule.c
Modules/_multiprocessing/clinic/posixshmem.c.h
Modules/_multiprocessing/multiprocessing.h
Modules/_multiprocessing/posixshmem.c
Modules/_multiprocessing/semaphore.c
Modules/_operator.c
Modules/_pickle.c
Modules/_sqlite/blob.c
Modules/_sqlite/blob.h
Modules/_sqlite/connection.c
Modules/_sqlite/connection.h
Modules/_sqlite/cursor.c
Modules/_sqlite/module.c
Modules/_sqlite/named_row.c
Modules/_sqlite/named_row.h
Modules/_sre.c
Modules/_ssl.c
Modules/_ssl_data.h
Modules/_stat.c
Modules/_struct.c
Modules/_testcapimodule.c
Modules/_threadmodule.c
Modules/_tkinter.c
Modules/_tracemalloc.c
Modules/_uuidmodule.c
Modules/_winapi.c
Modules/_xxsubinterpretersmodule.c
Modules/_zoneinfo.c
Modules/arraymodule.c
Modules/audioop.c
Modules/cjkcodecs/clinic/multibytecodec.c.h
Modules/cjkcodecs/multibytecodec.c
Modules/errnomodule.c
Modules/fcntlmodule.c
Modules/getpath.c
Modules/grapheme_cluster_break_automaton.h
Modules/main.c
Modules/makesetup
Modules/mmapmodule.c
Modules/nismodule.c
Modules/ossaudiodev.c
Modules/overlapped.c
Modules/posixmodule.c
Modules/pyexpat.c
Modules/selectmodule.c
Modules/signalmodule.c
Modules/socketmodule.c
Modules/socketmodule.h
Modules/sre.h
Modules/sre_constants.h
Modules/sre_lib.h
Modules/unicodedata.c
Modules/unicodedata_db.h
Modules/unicodename_db.h
Modules/xxsubtype.c
Modules/zlibmodule.c
Programs/_testembed.c
Tools/c-analyzer/c_parser/_state_machine.py
Tools/c-analyzer/c_parser/preprocessor/__init__.py
Tools/c-analyzer/cpython/_parser.py
Tools/ccbench/ccbench.py
Tools/clinic/clinic.py
Tools/gdb/libpython.py
Tools/i18n/msgfmt.py
Tools/scripts/README
Tools/scripts/deepfreeze.py
Tools/scripts/eptags.py
Tools/scripts/generate_global_objects.py
Tools/scripts/generate_opcode_h.py
Tools/scripts/ptags.py
Tools/scripts/win_add2path.py
Tools/ssl/multissltests.py
Tools/unicode/Makefile
Tools/unicode/makeunicodedata.py
abc_bench.py
aclocal.m4
configure
configure.ac
m4/ax_pthread.m4
pyconfig.h.in
setup.py
test.py

--
Jeremy Kloth

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/GIJ4SWF6ATKPZ6VOAOPKPKS7SMAONBCU/

[Ronald Oussoren via Python-Dev]

On 29 Mar 2022, at 19:51, Brett Cannon <br...@python.org> wrote:

On Tue, Mar 29, 2022 at 8:58 AM Ronald Oussoren <ronaldo...@mac.com> wrote:

On 29 Mar 2022, at 00:34, Brett Cannon <br...@python.org> wrote:

Once https://mail.python.org/archives/list/python-c...@python.org/thread/5EUZLT5PNA4HT42NGB5WVN5YWW5ASTT5/ is considered resolved, the next part of my "what is the stdlib" plan is to finally try to suss all of this out and more-or-less write a stdlib policy PEP so we stop talking about this. My guess it will be more of guidance about what we want the stdlib to be and thus guide what things belong in it. No ETA on that work since I also have four other big Python projects on the go right now whose work I am constantly alternating between.

Having such a policy is a good thing and helps in evolving the stdlib, but I wonder if the lack of such a document is the real problem.   IMHO the main problem is that the CPython team is very small and therefore has little bandwidth for maintaining, let alone evolving, large parts of the stdlib.  In that it doesn’t help that some parts of the stdlib have APIs that make it hard to make modifications (such as distutils where effectively everything is part of the public API).  Shrinking the stdlib helps in the maintenance burden, but feels as a partial solution. 

You're right that is the fundamental problem. But for me this somewhat stems from the fact that we don't have a shared understanding of what the stdlib is,  and so the stdlib is a bit unbounded in its size and scope. That leads to a stdlib which is hard to maintain.

That (the hard to maintain part) is not necessarily true, if we had enough resources…. In theory the stdlib could be split into logical parts with teams that feel responsible for those parts (similar to having maintainers sign up for various platforms).  That doesn’t work because of the small team, and partially also due to necessarily having very strict rules w.r.t. backward compatibility. 

It's just like dealing with any scarce resource: you try to cut back on your overall use as best as you can and then become more efficient with what you must still consume; I personally think we don't have an answer to the "must consume" part of that sentence that leads us to "cut back" to a size we can actually keep maintained so we don't have 1.6K open PRs.

I agree, but this is something to state explicitly when describing what should and should not be in scope for the stdlib.   I’m a fan of a batteries included stdlib, but with our current resources we cannot afford to have some bits in the stdlib that would “obviously” be a candidate for a modern batteries included stdlib, such as a decent HTTP stack with support for HTTP/1, /2 and /3. 

Ronald