Vendoring multipledispatch

[Michał 'Khorne' Lowas-Rzechonek]

Hi,

On last DjangoCon sprints I started working on ticket #26355, WIP is at https://github.com/django/django/pull/6395.

During the implementation I've realized that I need a double-dispatch mechanism for registering overloaded operators.

Current approach uses Python operator overloading exclusively, which can't (generally) take into account types of both operands.

So, I think I'm left with two options: reinvent https://pypi.python.org/pypi/multipledispatch, or import that into django.utils along with the license etc.

I appreciate this is not a decision to be taken lightly, so I'd rather ask here first, than submit a PR with shady imports.

Also, there might be a simpler/cleaner solution that doesn't required 3rd party code.

opinions?

-- 

Michał 'Khorne' Lowas-Rzechonek

[Donald Stufft]

> On Apr 4, 2016, at 7:50 AM, Michał 'Khorne' Lowas-Rzechonek <kho...@gmail.com> wrote:
>
> So, I think I'm left with two options: reinvent https://pypi.python.org/pypi/multipledispatch, or import that into django.utils along with the license etc.
>

Without looking at this specific thing too closely, maybe it’s time for Django to gain a required dependency instead of bundling or reinventing everything?


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Florian Apolloner]

On Monday, April 4, 2016 at 3:58:09 PM UTC+2, Donald Stufft wrote:

Without looking at this specific thing too closely, maybe it’s time for Django to gain a required dependency instead of bundling or reinventing everything?

+I_do_not_know_how_much

[Anssi Kääriäinen]

+1 to allowing dependencies.

For multipledispatch, I'd like to first verify the following items:
1. Does the author consider the API for multipledispatch stable?
2. Are the rules in multipledispatch what we want? If we vendor it,
we can easily change the matching rules.

> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/9470fda1-67ab-4d92-b819-597d364fb460%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[James Pic]

Adding dependencies would definitely be a huge step forward. I think
Django doesn't have them because pip wasn't as awesome as it is today
back in the early days, but nowadays it would definitely make sense.
That would mean a bit more work for distribution package maintainers
but if we can start communicating with them about it then would there
be any reason not to do this move ?

[Marc Tamlyn]

Does anyone (potentially from OS packaging worlds maybe) have a good reason NOT to have a dependency?

In the event that Django does have Pip dependencies, I still think we should establish some rules. Things like they should be stable and mature, and perhaps widely known, and still be under active maintenance. These are all hazy ideas of course, but I think adding any new dependency should be something which we are cautious of, not something we do without careful consideration.

If someone can put together some documentation or a DEP for how this process would work, I think this is a matter significant enough to talk about properly, and in a thread more clearly named. Such a proposal should also ideally reference what we do or have vendored before, why, and how well of otherwise that process has worked.

Marc

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CALC3Kae1Mt5LzYRq0PQbQcEMKwvbK%2Begr4FuC6baB5NZZtSbtA%40mail.gmail.com.

[Sylvain Fankhauser]

The only problem I can see is dependencies conflicts, where Django would need package X version > 1.0 and another installed package would need package X version 0.9. I have this issue with a project of mine that required six >= 1.10 and for some reason Xcode on OSX with the system Python (ie. not installed with brew for example) forces you to have six 1.9, which comes before in the PYTHONPATH. I know that Lektor for example solves this by having an install script that creates a virtualenv and installs it inside this virtualenv. Also James mentioned that "back in the days pip wasn't as awesome as it is today", but we have to keep in mind that a lot of OSes still use an old pip version by default (Debian stable for example ships pip 1.5).

Anyway as long as Django is installed in a virtualenv this shouldn't be too much of an issue, but I think we should expect some issues from the users and these should be documented otherwise people might get frustrated.

Cheers,
Sylvain

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMwjO1HqgJ%3DM93yWgmatZtc4s8%3DRB8FFfga24aJ3X_D_Ey2xdw%40mail.gmail.com.

[Erik Cederstrand]

> Den 6. apr. 2016 kl. 13.42 skrev Marc Tamlyn <marc....@gmail.com>:
>
> Does anyone (potentially from OS packaging worlds maybe) have a good reason NOT to have a dependency?

Here is a list off the top of my head. This is not necessarily an argument against dependencies, just some things to consider.


1: Availability. If Django depends on version x.y.z and x.y.z is removed from PyPI, or the whole package is deleted, then Django is no longer installable (google "NPM kik" for a recent example).

2: Customization. We need to tweak functionality in some non-upstreamable way, cherry-pick new functionality, or fix security issues before they are published on PyPI.

3: Version conflicts, as mentioned by Sylvain.

4: Security/stability. We depend on version x.y and a witty developer uploads dependency x.y.z+1 with an Easter egg, or the PyPI developer account is hacked and x.y is replaced.


These issues are amplified in a world where many people have automated production deployments running 'pip install -U -r requirements.txt'. Issues could spread very fast.

This may not be much different than what people are already exposed to with their own project dependencies, but vendoring (directly or by dependency) is endorsement by the Project, so any issues in the dependencies will fall back on the Project.


Erik

[Donald Stufft]

For 1, 3, and 4, Django is already exposed to these problems for the vast majority of use cases since if you want to use any database other than SQLite then you have to install something or if you want to use an ImageField, or bcrypt, etc. Having ``pip install Django`` work but not ``pip install Django psycopg2`` when you’re running a site that uses PostgreSQL doesn’t get you anything extra there.

[Donald Stufft]

On Apr 7, 2016, at 1:31 AM, Sylvain Fankhauser <sylvain.f...@gmail.com> wrote:

The only problem I can see is dependencies conflicts, where Django would need package X version > 1.0 and another installed package would need package X version 0.9. I have this issue with a project of mine that required six >= 1.10 and for some reason Xcode on OSX with the system Python (ie. not installed with brew for example) forces you to have six 1.9, which comes before in the PYTHONPATH. I know that Lektor for example solves this by having an install script that creates a virtualenv and installs it inside this virtualenv.

Yea. This is a problem with OS X Python from the system. They ship a bunch of libraries that are on the sys.path by default but which can’t be replaced (even more so in the most recent version of OS X when SIP is turned on which prevents installing anything into the system Python as well). This issue has been reported to Apple and I think there is a good chance that they’ll have it fixed in a way that a user can install packages into.

Overall though, users shouldn’t install things into their system Python which I think is supported by the fact that the system Python on recent versions of OS X are not writable, even with sudo, unless you disable one of the major security components of your machine. That doesn’t affect just any potential dependencies Django might have, but also Django itself.

Also James mentioned that "back in the days pip wasn't as awesome as it is today", but we have to keep in mind that a lot of OSes still use an old pip version by default (Debian stable for example ships pip 1.5).

pip 1.5 is new enough that you get most of the reliability benefits. Major things it’s missing are:

* Implicit wheel cache, allowing sdists to only be compiled once and the built binaries reused between installs.

* Implicit http cache, allowing files to only need to be downloaded once.

* Manylinux1 support, allowing people to publish binary wheels for Linux systems to PyPI.

* Conditional dependencies inside of a sdist without a hack workaround.

Minor, but nice quality of life things are:

* Massively reduced verbosity on install.

* Nicer progress bars.

Anyway as long as Django is installed in a virtualenv this shouldn't be too much of an issue, but I think we should expect some issues from the users and these should be documented otherwise people might get frustrated.

Cheers,
Sylvain

[René Fleschenberg]

Hi,

 

> Anyway as long as Django is installed in a virtualenv this shouldn't be

> too much of an issue, but I think we should expect some issues from the

> users and these should be documented otherwise people might get

> frustrated.

 

It can still be an issue for things that share the same virtualenv with Django, i.e. Django apps and projects. Any library that Django depends on imposes a restriction on which version of this library user code can use.

 

For example, if Django requires `requests>=2.9,<3.0`, this means that Django-based apps and projects cannot use requests 3.0+ in their code (at least not easily).

 

For this reason, I believe that Django should be very conservative about introducing new dependencies, but maybe not outrule them completely.

 

--

René