[Django] #28905: Overhaul extra_requires to include more optional dependencies

Django

unread,

Dec 7, 2017, 7:42:45 AM12/7/17

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------
Reporter: Jaap Roes | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: | Version: master
Uncategorized | Keywords: setup optional
Severity: Normal | dependencies packages requirements
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Django has a number of optional features that require the installation of
additional Python packages.

Right now it's possible to `pip install Django[bcrypt,argon2]` to install
Django and the optional packages (at the correct minimum version) required
for bcrypt/argon2 support.

However this is not possible for other optional features, like
database/cache backend, gis, certain image related features etc.

I've created a pull request adding the optional dependencies found in the
test requirements (initially only psycopg2 for postgresql, but after a
comment from pope1ni I followed their suggestion to include more). It
changes the set of extra_requires to:

{{{
{
'admindocs': ['docutils'],
'argon2': ['argon2_cffi>=16.1.0'],
'bcrypt': ['bcrypt'],
'gis': [
'geoip2',
'numpy'
],
'geoip': ['geoip2'],
'jinja2': ['Jinja2>=2.9.2'],
'memcached': ['python3-memcached'],
'numpy': ['numpy'],
'mysql': ['mysqlclient>=1.3.7'],
'oracle': ['Oracle'],
'images': ['Pillow'],
'postgresql': ['psycopg2>=2.5.4'],
'pylibmc': ['pylibmc; sys.platform != "win32"'],
'sqlparse': ['sqlparse'],
'selenium': ['selenium'],
'test-parallel': ['tblib'],
'yaml': ['PyYAML'],
}
}}}

So end users can do things like:

{{{
$ pip install Django[mysql,sqlparse,images,memcached]
$ pip install Django[postgresql,gis]
$ pip install Django[selenium,test-parallel]
}}}

Not only would this make it much easier to install a particular
combination of Django and optional dependencies correctly, it also makes
some of the optional features/dependencies a little bit more discoverable.

This might not the full set of optional dependencies, or the right
grouping/naming, so a review is more than welcome.

Besides adding this to `setup.py`, additional changes to the
documentation, test instructions and scripts, code warnings etc. will
probably be necessary.

--
Ticket URL: <https://code.djangoproject.com/ticket/28905>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,

Dec 7, 2017, 9:37:20 AM12/7/17

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------
Reporter: Jaap Roes | Owner: nobody
Type: | Status: new
Cleanup/optimization |

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jaap Roes):

* component: Uncategorized => Packaging

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:1>

Django

unread,

Dec 7, 2017, 10:39:21 AM12/7/17

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------
Reporter: Jaap Roes | Owner: nobody
Type: | Status: new
Cleanup/optimization |

Component: Packaging | Version: master
Severity: Normal | Resolution:

Keywords: setup optional | Triage Stage: Accepted
dependencies packages |
requirements |
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sergey Fedoseev):

* has_patch: 0 => 1
* stage: Unreviewed => Accepted

Comment:

[https://github.com/django/django/pull/9440 PR]

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:2>

Django

unread,

Dec 8, 2017, 6:39:08 AM12/8/17

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------
Reporter: Jaap Roes | Owner: nobody
Type: | Status: new
Cleanup/optimization |

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jaap Roes):

I have no idea where to go from here. I assume posting this to the
developer mailinglist is the best way to gather feedback on this and come
to a consensus of which packages and groupings.

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:3>

Django

unread,

Feb 12, 2018, 5:31:26 AM2/12/18

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------
Reporter: Jaap Roes | Owner: nobody
Type: | Status: new
Cleanup/optimization |

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Carlton Gibson):

Django

unread,

Feb 12, 2018, 7:35:18 AM2/12/18

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------
Reporter: Jaap Roes | Owner: nobody
Type: | Status: new
Cleanup/optimization |

Has patch: 1 | Needs documentation: 1

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Carlton Gibson):

* needs_docs: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:5>

Django

unread,

May 24, 2018, 10:24:58 AM5/24/18

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------
Reporter: Jaap Roes | Owner: nobody
Type: | Status: new
Cleanup/optimization |

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Jaap Roes):

* easy: 0 => 1

Comment:

I don't have the time/energy to get this done, but I feel that this would
be an excellent, and fairly easy, ticket for someone to work on during the
Djangocon sprints

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:6>

Django

unread,

May 24, 2018, 10:16:15 PM5/24/18

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Reporter: Jaap Roes | Owner: oliver
Type: | Status: assigned

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by oliver):

* owner: nobody => oliver
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:7>

Django

unread,

May 24, 2018, 10:18:04 PM5/24/18

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Reporter: Jaap Roes | Owner: (none)
Type: | Status: new
Cleanup/optimization |

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by oliver):

* owner: oliver => (none)
* status: assigned => new

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:8>

Django

unread,

May 27, 2018, 7:49:13 AM5/27/18

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Reporter: Jaap Roes | Owner: (none)
Type: | Status: new
Cleanup/optimization |

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Israel Fermín Montilla):

Can I claim it?, I have some spare time I could invest on this

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:9>

Django

unread,

May 28, 2018, 2:21:45 PM5/28/18

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Reporter: Jaap Roes | Owner: (none)
Type: | Status: new
Cleanup/optimization |

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Herbert Fortes):

* cc: Herbert Fortes (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:10>

Django

unread,

May 29, 2018, 3:37:34 AM5/29/18

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Reporter: Jaap Roes | Owner: (none)
Type: | Status: new
Cleanup/optimization |

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jaap Roes):

Yes please claim it, it would be really helpful to, for example, identify
pieces of documentation that should be updated when this lands.

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:11>

Django

unread,

Jul 4, 2018, 7:40:31 AM7/4/18

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Reporter: Jaap Roes | Owner: Niranj
Type: | Rajasekaran
Cleanup/optimization | Status: assigned

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Niranj Rajasekaran):

* status: new => assigned

* owner: (none) => Niranj Rajasekaran

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:12>

Django

unread,

Nov 14, 2018, 5:43:44 AM11/14/18

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jaap Roes):

I just discovered another optional dependency, on Linux one can install
`pyinotify` for better reloading behaviour
(https://docs.djangoproject.com/en/dev/ref/django-admin/#runserver).

So maybe adding `{'dev': ['pyinotify ; sys_platform == "linux"']}` would
be helpful as well.

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:13>

Django

unread,

Nov 18, 2018, 6:41:09 PM11/18/18

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Reporter: Jaap Roes | Owner: Nick Pope
Type: | Status: assigned
Cleanup/optimization |

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Nick Pope):

* owner: Niranj Rajasekaran => Nick Pope
* needs_docs: 1 => 0

Comment:

New [https://github.com/django/django/pull/10663 PR].

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:14>

Django

unread,

Nov 26, 2018, 5:51:01 AM11/26/18

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jaap Roes):

Thanks Nick for taking this ticket and trying to move it forwards.

The additional documentation about optional dependencies makes so much
sense, it alone would satisfy some of the discoverability issues with
optional dependencies. It would be nice if the overview of optional
dependencies could link to the relevant sections in the docs. E.g.
caching, databases, GIS, password hashers, template engines, serializers
etc.

The grouping/inclusion of dependencies seems to be the main stumbling
block. Adding `extra_requires` groups adds a maintenance burden on these
groups, as in, they'll need to work for future versions of Django. For
this reason I think it's probably not the right choice to name groups
directly after the package they are installing (`pylibmc`, `pyinotify`,
`selenium` etc.).

Maybe it would be easier to just start out with just adding the database
extra's? Those seem non-controversial and the package names and version
specs are non-obvious, so having those in Django's `setup.py` has a clear
advantage.

It might also be interesting to figure out how to "deprecate" extra
requirement groups, so at some point it would be possible to phase out the
`bcrypt` or `argon2` extra requirements.

I was also thinking, extra requirements would also be a nice place to
include other "official" Django projects. E.g. `pip install
django[mysql,localflavor,comments]` or `pip install
django[postgresql,channels]`.

Anyway, I don't think this ticket will go anywhere until someone with a
more authoritative voice makes a decision what's wanted and unwanted.

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:15>

Django

unread,

Dec 8, 2018, 6:40:27 PM12/8/18

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Nick Pope):

Thanks for the feedback, please see responses below.

> ... It would be nice if the overview of optional dependencies could link

to the relevant sections in the docs. E.g. caching, databases, GIS,
password hashers, template engines, serializers etc.

Agreed. Will add in links to useful sections in the documentation.

> ... For this reason I think it's probably not the right choice to name

groups directly after the package they are installing (`pylibmc`,
`pyinotify`, `selenium` etc.).
>
> Maybe it would be easier to just start out with just adding the database

extra's? ...
I still feel that adding support for some of the packages other than
database backends would be very useful. There should at the very least be
an easy option to install all non-database backend packages, particularly
useful for removing `tests/requirements/*.txt`.

> It might also be interesting to figure out how to "deprecate" extra
requirement groups, so at some point it would be possible to phase out the
`bcrypt` or `argon2` extra requirements.

I'm not sure it is possible to "deprecate" - in fact it doesn't really
make sense - a package must be available during a deprecation period.
However, when the package is removed, the group must either remain empty
in `extra_requires` or be removed entirely if documented in the release
notes.

> I was also thinking, extra requirements would also be a nice place to
include other "official" Django projects. E.g. `pip install
django[mysql,localflavor,comments]` or `pip install
django[postgresql,channels]`.

I think it best to leave these out of the `extra_requires` option, but
perhaps document them on the optional dependencies page along with a link
to other [https://pypi.org/search/?q=&o=&c=Framework+%3A%3A+Django django-
related projects].

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:16>

Django

unread,

Jan 28, 2019, 10:42:08 AM1/28/19

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Carlton Gibson):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:17>

Django

unread,

Feb 3, 2020, 5:27:40 AM2/3/20

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Reporter: Jaap Roes | Owner: Rishabh
Type: | Arya
Cleanup/optimization | Status: assigned

Changes (by Rishabh Arya):

* owner: Nick Pope => Rishabh Arya

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:18>

Django

unread,

Oct 21, 2020, 7:06:48 AM10/21/20

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Comment (by Manav Agarwal):

Can I claim this? I think I can move it forward.

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:19>

Django

unread,

Jan 21, 2021, 7:29:15 AM1/21/21

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Reporter: Jaap Roes | Owner: Nick Pope
Type: | Status: assigned
Cleanup/optimization |

Changes (by Nick Pope):

* owner: Rishabh Arya => Nick Pope

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:20>

Django

unread,

Sep 24, 2022, 7:30:32 AM9/24/22

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Reporter: Jaap Roes | Owner: Nick Pope
Type: | Status: assigned
Cleanup/optimization |

Component: Packaging | Version: dev

Severity: Normal | Resolution:
Keywords: setup optional | Triage Stage: Accepted
dependencies packages |
requirements |

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Carlton Gibson):

* easy: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:21>

Django

unread,

Jan 30, 2024, 8:01:39 AMJan 30

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Natalia Bidart):

* cc: Nick Pope (added)

Comment:

Nick, do you see any value in trying to resurface/push this? I have been
recently asking about dependency management following a crusade to pin
`black` to `23.12.1` for `stable/4.2.x`
([https://github.com/django/django/pull/17781 PR #17781] and
[https://github.com/django/django/pull/17793 PR #17793]), so I wondered if
there would be a way to define deps (and versions) in a single place to be
useful for all GitHub Actions, docs building, pre-commit hooks and
tests/tox definitions.
--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:22>

Django

unread,

Feb 1, 2024, 4:32:49 AMFeb 1

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Claude Paroz):

FWIW, I just pushed a commit I had prepare some times ago as a
[https://github.com/django/django/pull/17806 PR], which moves content of
`setup.*` to `pyproject.toml`. I don't know if that changes anything in
the game…
--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:23>

Django

unread,

Feb 1, 2024, 4:48:08 AMFeb 1

to django-...@googlegroups.com

#28905: Overhaul extra_requires to include more optional dependencies
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Nick Pope):

Hi Natalia. Sorry for the delay getting back to you. I'll dust off what I
had and see where things stand - certainly things could be simplified and
dependency stuff is scattered, in particular from a documentation PoV.
Some of my ongoing work toward refreshing `django-docker-box` has exposed
some areas that are particularly non obvious.

Thanks, Claude. I'll review that PR. It feels like it is time to
reconsider the more complete migration to `pyproject.toml` in light of the
`distutils` removal, deprecation of support for calling `setup.py`
directly, and stabilisation of support for `pyproject.toml` in most
tooling including `setuptools`.
--
Ticket URL: <https://code.djangoproject.com/ticket/28905#comment:24>

Reply all

Reply to author

Forward