Should we require pytz for timezone support in Django?

[Josh Smeaton]

While writing timezone tests for https://github.com/django/django/pull/6243 I ran into some issues where pytz seemed to be required for just about every database and platform combination except postgres on linux. The docs for timezone support (https://docs.djangoproject.com/en/dev/topics/i18n/timezones/) say:

Installing pytz is highly recommended, but may not be mandatory depending on your particular database backend, operating system and time zone. If you encounter an exception querying dates or times, please try installing it before filing a bug. 

I tried to find some documentation on the broader internet that would more accurately describe when and for what platforms pytz is actually required for timezone support, but was unable to find anything. I've opened a new ticket https://code.djangoproject.com/ticket/26622 to clarify more explicitly when pytz is required.

However, I'm wondering if we should just go all-in on pytz and require it for timezone support. I've not run a Django site with timezone support without pytz, so I'm not sure what exact limitations exist or why you'd want to use timezone support without pytz. I'd be interested in hearing use cases for not requiring pytz if there are any. Explicitly requiring pytz will make test writing a lot easier, and will remove any doubt about whether or not pytz is required for users.

Josh 

[Marc Tamlyn]

It would seem reasonable to me to require it. I wonder whether the reasoning is based on distro packaging or similar?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/cb513e96-e75f-462f-a8ca-b5d916ad8eef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Aymeric Augustin]

The reasoning was based on:

- our reluctance to add dependencies back then (Django 1.4)

- sqlite3 not being strictly needed in all circumstances (and probably being needed in fewer circumstances originally)

I’ll try to find time to make a more detailed answer later.

-- 

Aymeric.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMwjO1H_n%3DGzAFZ%2BR07Qf2qnbmY8_dpy5FhdaBEXrQRB0gJ9sQ%40mail.gmail.com.

[Michael Manfre]

I can't recall the last Django project I worked on that didn't require pytz. It makes sense to me to require it.

Regards,

Michael Manfre

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/E0E058F8-7842-4F57-B7C2-A5BA2C2FD4D4%40polytechnique.org.

[akki]

Corresponding ticket - #26622.

[Philip James]

Is there consensus on making pytz required? (I, personally, am in favor, since I too can't remember the last project I did that didn't need it.)

If it's required, should it be vendored?

[Aymeric Augustin]

> On 04 Jun 2016, at 00:55, Philip James <p...@philipjohnjames.com> wrote:
>
> If it's required, should it be vendors?

That doesn’t sound optimal because pytz’ release cycle is tied to the TZ database’s cycle, which is different from Django’s.

--
Aymeric.

[Donald Stufft]

> On Jun 4, 2016, at 1:18 PM, Aymeric Augustin <aymeric....@polytechnique.org> wrote:
>
> That doesn’t sound optimal because pytz’ release cycle is tied to the TZ database’s cycle, which is different from Django’s.

IMO just stick pytz in install_requires and be done with it. It’s a very trivial dependency, and is available as a Wheel, various eggs, and sdists. Just to put some numbers behind this:

93.6% of all downloads of Django from PyPI [1] are originating using a tool that will properly support dependencies out of the box. Another 3.5% of that are using a tool that we can’t detect— this is largely older versions of pip and devpi that didn’t accurately report themselves in the User-Agent header, so is going to also be largely things that support dependencies properly. Another 2.3% of the downloads are coming from mirroring clients like devpi, bandersnatch, etc where it’s likely the consumers are mostly or entirely automated tooling that understands dependencies again.

Finally, 0.5% of all downloads for Django from PyPI are made using the browser, where people may or may not be passing that to a tool that properly understands dependencies.

In addition, the latest versions of all Python versions that Django supports come with pip already available, eliminating the bootstrapping issues.

Given these numbers, and the existing of ensurepip, I think it’s easily time for Django to just start depending on things, particularly when those things are something as trivial to install as a pure python library like pytz.

[1] Using the now public metrics database.

—
Donald Stufft

[Carl Meyer]

On 06/03/2016 03:55 PM, Philip James wrote:
> Is there consensus on making pytz required? (I, personally, am in favor,
> since I too can't remember the last project I did that didn't need it.)
>
> If it's required, should it be vendored?

-1 on vendoring pytz. I think it's fine to just make it a required
dependency if that saves us significant work on a valuable new feature
or refactoring. I don't really see the hurry to make it a required
dependency just for its own sake, though.

Carl

[Philip James]

Adding pytz to install_requires might not contribute to a bug fix or new feature right now, but it does remove a point of confusion from the docs. I think it also reduces friction in getting Django up-and-running, which seems valuable for Django users and Django contributors.

PJJ

http://philipjohnjames.com

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/57532114.20003%40oddbird.net.

[Carl Meyer]

I just realized that I think something's gotten lost in this thread:
Josh's original point was about requiring pytz _if you want timezone
support_ (that is, if USE_TZ=True). That would I presume involve raising
ImproperlyConfigured (or maybe having a site check raise an error?) if
USE_TZ=True and pytz is not installed. I'd support this for sure: the
"well, it might work OK without pytz if you're on Postgres" caveat is
extra complexity for not much gain.

But this is not the same thing as putting pytz in install_requires,
which makes it a required dependency for everyone using Django, even if
they want USE_TZ=False.

I don't know how many USE_TZ=False projects there are. I always set
USE_TZ=True, but I can imagine some small locally-oriented sites might
have reason to prefer USE_TZ=False? If there is a significant number,
and we want to continue to support their use-case, it's a non-zero added
burden to require them to pull in a frequently-updated useless dependency.

Carl

[Carl Meyer]

FWIW: https://github.com/search?q=USE_TZ+%3D+False&type=Code

Carl

[Josh Smeaton]

It sounds like we have consensus that we can make pytz a required dependency when USE_TZ = True, and clear up the associated documentation. 

I think now we should try and find (rough) consensus on whether or not we should have pytz as a required dependency in `install_requires`.

> If there is a significant number, and we want to continue to support their use-case, it's a non-zero added burden to require them to pull in a frequently-updated useless dependency. 

Let's assume there is a reasonable number of deployments with and without timezone support. Even then I'm personally in favour of requiring pytz. 

django-admin.py startproject defaults to USE_TZ = True which means that new projects that install django, start the project, and then run the internal web server are going to get missing dependency problems. A dependency which, as Donald points out, is universally available and runs very little risk of disrupting an install. It'll also mean that existing projects that upgrade to timezone support do not have any extra dependencies to install, though I admit these kinds of projects are more likely to specifically read the timezone documentation. I think the future of django relies more and more on timezone support.

On the other hand we have projects that do not have timezone support. These will usually be older projects rather than newer ones. Having pytz installed isn't going to negatively affect a project (it's a small yet often updating dependency), and it can be uninstalled manually (or django installed with "--no-deps") if it concerns specific deployments. Mainly though, requiring pytz doesn't result in a failure mode for these projects. There is certainly a non-zero cost for these users, but I don't think it's as big as asking users of the default project to manually install a dependency that is required.

I think non-timezone users that care whether or not they have pytz installed will be the smallest group of users, and they can install/upgrade Django with the --no-deps flag.

Cheers

[Carl Meyer]

> On Jun 5, 2016, at 6:09 PM, Josh Smeaton <josh.s...@gmail.com> wrote:
>
> It sounds like we have consensus that we can make pytz a required dependency when USE_TZ = True, and clear up the associated documentation.
>
> I think now we should try and find (rough) consensus on whether or not we should have pytz as a required dependency in `install_requires`.
>
> > If there is a significant number, and we want to continue to support their use-case, it's a non-zero added burden to require them to pull in a frequently-updated useless dependency.
>
> Let's assume there is a reasonable number of deployments with and without timezone support. Even then I'm personally in favour of requiring pytz.
>
> django-admin.py startproject defaults to USE_TZ = True which means that new projects that install django, start the project, and then run the internal web server are going to get missing dependency problems.

That's an excellent point. I agree, we should just add pytz to install_requires if we require it for USE_TZ=True, otherwise the default new-project experience is unpleasant.

Carl

[Aymeric Augustin]

> On 06 Jun 2016, at 03:46, Carl Meyer <ca...@oddbird.net> wrote:
>
>> django-admin.py startproject defaults to USE_TZ = True which means that new projects that install django, start the project, and then run the internal web server are going to get missing dependency problems.
>
> That's an excellent point. I agree, we should just add pytz to install_requires if we require it for USE_TZ=True, otherwise the default new-project experience is unpleasant.

I agree as well.

One of these days we should reverse the “actual default value” so we don’t have to override it in the “default project template” (and then people obviously have no idea what the “default” is…)

--
Aymeric.

[Tim Graham]

I created a ticket and PR to add pytz to install_requires. I think it's a nice simplification and will result in a better user experience.
https://code.djangoproject.com/ticket/27327
https://github.com/django/django/pull/7361

[Aymeric Augustin]

I reviewed the patch and I think it’s good. It removes quite a lot of messy code.

There’s a DEP in progress about allowing dependencies here:

https://github.com/django/deps/pull/26

It looks like that there’s consensus on the general idea of allowing dependencies

and on pytz being an acceptable dependency, and that the DEP isn’t finished

simply because no one has taken the time to finish it.

If some contributors are uncomfortable with preempting the discussion of that DEP

or don’t think there’s consensus, now’s a good time to say so :-) Tim, you could

also ask for the technical board’s opinion if you want.

-- 

Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5c96fb2e-a620-4a1a-a584-64c3318f9837%40googlegroups.com.

[Kevin Brown]

I agree with requiring pytz if you have timezone support enabled in Django, as that cuts out a set of edge cases and likely simplifies the code base a bit in the long run. And I even agree with forcing it to be installed with Django if we want to encourage people to use Django with timezone support by default.

But I don't see the value in requiring pytz to be installed whenever you install Django, even if you explicitly disable timezone support. This means that there is the potential to force pytz to be installed in cases where it may never be used, and at the moment isn't even required.

If there was interest in dropping support for using Django without timezones (so removing USE_TZ = False), then I would understand the push for making pytz required for everyone.

[Aymeric Augustin]

Hello Kevin,

> On 09 Oct 2016, at 20:09, Kevin Brown <ke...@kevinbrown.in> wrote:
>
> If there was interest in dropping support for using Django without timezones (so removing USE_TZ = False), then I would understand the push for making pytz required for everyone.

You’re making the assumption that pytz is only needed when USE_TZ = True. In practice it isn’t that simple.

If you look at Tim’s patch you’ll see that code that relies pytz doesn’t look like `if settings.USE_TZ: # do something with pytz` in general. Several APIs need to account for time zones to produce accurate results even when USE_TZ = False, for example, if memory serves, template tags that humanize durations, probably others. Also django.utils.timezone can be useful even when USE_TZ = False.

The patch makes it clear that managing with / without pytz alternatives everywhere creates a lot of noise and a lot of error messages that boil down to “yeah, this doesn’t work without pytz, install it”. Forcing the install upfront has a negligible cost and removes the possibility to hit these frustrating errors

Also supporting two code paths cause a non-negligible overhead for the development of Django. Every time someone wants to add a test that involves datetimes, they have to figure out what happens with or without pytz, whether the test should be skipped and in which circumstances, etc. Often, I still have to intervene myself, even though it’s been four years since I added time zone support to Django. The developer experience in this area is awful :-/

Of course the point of Django is to have few people make efforts so that everyone else doesn’t have to, but in this case, the burden added on everyone else seems barely noticeable. If the concern is the size of a Django install, honestly, there are other areas to target (e.g. ship contrib apps, translations, etc. separately).

I hope this clarifies the situation,

--
Aymeric.

[Tim Graham]

I forgot to mention (and document) that (if I'm following the code correctly) this change removes support for setting TIME_ZONE=None. According to the docs, TIME_ZONE=None "causes cause Django to fall back to using the system timezone. However, this is discouraged when USE_TZ = True, because it makes conversions between local time and UTC less reliable." Is removing that option acceptable as a backwards-incompatible change?

[Aymeric Augustin]

On 10 Oct 2016, at 21:31, Tim Graham <timog...@gmail.com> wrote:

I forgot to mention (and document) that (if I'm following the code correctly) this change removes support for setting TIME_ZONE=None. According to the docs, TIME_ZONE=None "causes cause Django to fall back to using the system timezone. However, this is discouraged when USE_TZ = True, because it makes conversions between local time and UTC less reliable." Is removing that option acceptable as a backwards-incompatible change?

I don’t have an opinion on that.

I don’t think USE_TZ = None provides a particularly useful behavior. I think it’s best to state the expected value explicitly.

Perhaps that option was implemented because someone asked for it? I don’t remember.

-- 

Aymeric.

[Tim Graham]

Here's the ticket that added support for TIME_ZONE=None: https://code.djangoproject.com/ticket/1480

[Florian Apolloner]

On Monday, October 10, 2016 at 9:50:51 PM UTC+2, Aymeric Augustin wrote:

I think it’s best to state the expected value explicitly.

In theory yes, in pratice I think that is at least annoying on Windows which only allows for the System-TZ I think?

Cheers,
Florian

[Carl Meyer]

Hi Kevin,

I made this same argument a while back. The main problem with it is that
the default startproject sets USE_TZ = True. That means that with your
proposal, the default experience for every new Django dev would be 1.
install Django, 2. run startproject, 3. run runserver and get an error
that pytz isn't installed. That's not acceptable. Having pytz installed
in some cases when it isn't strictly needed is a very small price to pay
for a smoother path in the common case.

Carl

[Tim Graham]

According to the docs for settings.TIMEZONE: "Django cannot reliably use alternate time zones in a Windows environment. If you’re running Django on Windows, TIME_ZONE must be set to match the system time zone."

If we keep supporting settings.TIME_ZONE=None I think that means this branch stays:

https://github.com/django/django/blob/a3a10f8abe43c443f217d75153f50b53fbe71be2/django/utils/timezone.py#L180-L182

which means we have to keep LocalTimezone:

https://github.com/django/django/blob/a3a10f8abe43c443f217d75153f50b53fbe71be2/django/utils/timezone.py#L82-L151

or is there some other approach we could take?

Is anyone reading this using settings.TIME_ZONE=None?

[Aymeric Augustin]

> On 12 Oct 2016, at 18:22, Tim Graham <timog...@gmail.com> wrote:
>
> is there some other approach we could take?

To obtain the pytz timezone for the local time, you’d need something like:
https://github.com/regebro/tzlocal

It isn’t exactly straightforward… This approach seems sub-optimal.

--
Aymeric.

[Tim Graham]

Would you be comfortable recommending that package for anyone who needs the autodetection?  This seems to work:

from tzlocal import get_localzone

TIME_ZONE = get_localzone().zone

What would change compared to the behavior of TIME_ZONE=None is that Django will set os.environ['TZ'] (except on Windows). I don't really understand the use case where doing that may have been a problem [0].

[0] https://code.djangoproject.com/ticket/1480

[Tim Graham]

I asked on django-users [0] if anyone is using settings.TIME_ZONE=None and didn't get any responses. Does it seem okay to drop that for now and react if we get any complaints about its removal?

[0] https://groups.google.com/d/topic/django-users/CxAiS6PXs3U/discussion

[Aymeric Augustin]

Hello Tim,

Considering that there’s no evidence of anyone using the feature, that it doesn’t work well in all circumstances, that we have an alternative which works better and has no downsides (considering the new policy on dependencies), I think that’s reasonable.

I’m afraid we’ll get a couple complaints a few months after we’ll drop security support for 1.8, not during the 1.11 alpha period… But that doesn’t change my arguments. If we get feedback during 1.11 alpha then we’ll have to reconsider.

(Disclaimer — I’m more tolerant of backwards-incompatible changes than many other committers.)

-- 

Aymeric.

-- 
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/f83d112d-d2c6-4201-91ad-80578f23df0d%40googlegroups.com.