[Django] #24076: Query may fail with pytz exception
Django
----------------------------------------------+--------------------
Reporter: lvella | Owner: nobody
Type: Bug | Status: new
Component: Database layer (models, ORM) | Version: 1.6
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------------------+--------------------
If I give date object as input to a filter on a DateTimeField field, like
this:
{{{
class Ticket(models.Model):
register_time = models.DateTimeField(auto_now_add=True)
day = datetime.strptime('2014-10-19', '%Y-%m-%d').date()
Ticket.objects.filter(register_time__gte=day)
}}}
I may get a pytz.exceptions.NonExistentTimeError exception. The exact
exception was:
pytz.exceptions.NonExistentTimeError: 2014-10-19 00:00:00
This date is the start of DST in my timezone:
{{{
TIME_ZONE = 'America/Sao_Paulo'
}}}
I believe this is a bug because Django tries to convert my input to a
datetime object before building the query, and in this case, this
conversion yields an invalid datetime.
This bug seems very related to issue #9596, because in these cases,
datetime should be converted to date, not the opposite.
A minimal showcase is attached. Unzip and run ./manage.py test
--
Ticket URL: <https://code.djangoproject.com/ticket/24076>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: lvella | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by aaugustin):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
That's the intended behavior -- Django won't guess what you mean if you
pass invalid inputs.
However, if #9596 gets fixed with the idea of comment 24, you will be able
to run `Ticket.objects.filter(register_time__date__gte=day)`.
Until then, you're stuck with comparing with an aware datetime object. See
https://www.python.org/dev/peps/pep-0431/#ambiguous-times for why this is
awfully complicated.
--
Ticket URL: <https://code.djangoproject.com/ticket/24076#comment:1>
Django
Reporter: lvella | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by lvella):
Replying to [comment:1 aaugustin]:
> That's the intended behavior -- Django won't guess what you mean if you
pass invalid inputs.
But if using date to compare with datetime works 99% of the time, and
there is no big warning in the documentation, saying that it must not be
done, it clearly violates the principle of least astonishment.
In such cases, either it should always work, or should always fail with an
error. And it is not that hard to make it work, for instead of trying to
add time information to the date before passing it to the database, the
database type should be converted to date, dropping the time information
in the process.
Actually, my workaround in the case was to force Django to deliver my
input in SQL as date, not as datetime:
{{{
q = Ticket.objects.filter(register_time__gte=day + (F('id') - F('id')))
print(q.query)
}}}
yields:
{{{
SELECT "reproduce_ticket"."id", "reproduce_ticket"."register_time" FROM
"reproduce_ticket" WHERE "reproduce_ticket"."register_time" >= 2014-10-19
+ ("reproduce_ticket"."id" - "reproduce_ticket"."id")
}}}
and it worked!
Compare with the original query (with another date so it won't raise the
exception):
{{{
q = Ticket.objects.filter(register_time__gte=day)
print(q.query)
SELECT "reproduce_ticket"."id", "reproduce_ticket"."register_time" FROM
"reproduce_ticket" WHERE "reproduce_ticket"."register_time" >= 2014-11-19
02:00:00
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24076#comment:2>
Django
Reporter: lvella | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by aaugustin):
* component: Database layer (models, ORM) => Documentation
* stage: Unreviewed => Accepted
Comment:
Thanks for the proposal. Unfortunately it won't work on databases other
than PostgreSQL because they store datetimes in UTC. As a consequence
they're vulnerable to the effect I described in question 3 in the
[https://docs.djangoproject.com/en/1.7/topics/i18n/timezones/#troubleshooting
timezone troubleshooting FAQ].
I considered this issue carefully when I implemented support for time
zones in Django 1.4. I was aware of the points you're making. I decided
that forbidding mixing dates and datetimes outright was too extreme. It
would require many changes in projects that work just fine because they
never see datetimes during the DST switch.
It's the left-hand-side of a lookup that determines the type, not the
right-hand-side. Therefore the correct solution is to implement an
explicit `__date` lookup i.e. #9596. That's the opposite of what you
requested when you filed the ticket but that also what you recommended in
your last comment:
> the database type should be converted to date, dropping the time
information in the process
This solution requires timezone conversion of datetimes in the database
but it shouldn't be an issue as it already exists to support the `dates()`
and `datetimes()` queryset methods.
I'm going to requalify this as a documentation issue because comparing
dates with datetimes is probably a common error and I don't think the docs
warn against it. I don't know where the warning should be added, though.
--
Ticket URL: <https://code.djangoproject.com/ticket/24076#comment:3>
Django
Reporter: lvella | Owner: Akshat verma
Type: Bug | Status: assigned
Component: Documentation | Version: 1.6
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Akshat verma):
* owner: nobody => Akshat verma
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/24076#comment:4>
Django
Reporter: lvella | Owner: Akshat verma
Type: Bug | Status: assigned
Component: Documentation | Version: 1.6
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 1 | UI/UX: 1
-------------------------------+----------------------------------------
Changes (by Akshat verma):
* cc: Akshat verma (added)
* needs_better_patch: 0 => 1
* needs_tests: 0 => 1
* easy: 0 => 1
* needs_docs: 0 => 1
* has_patch: 0 => 1
* ui_ux: 0 => 1
Old description:
> If I give date object as input to a filter on a DateTimeField field, like
> this:
>
> {{{
> class Ticket(models.Model):
> register_time = models.DateTimeField(auto_now_add=True)
>
> day = datetime.strptime('2014-10-19', '%Y-%m-%d').date()
> Ticket.objects.filter(register_time__gte=day)
> }}}
>
> I may get a pytz.exceptions.NonExistentTimeError exception. The exact
> exception was:
>
> pytz.exceptions.NonExistentTimeError: 2014-10-19 00:00:00
>
> This date is the start of DST in my timezone:
>
> {{{
> TIME_ZONE = 'America/Sao_Paulo'
> }}}
>
> I believe this is a bug because Django tries to convert my input to a
> datetime object before building the query, and in this case, this
> conversion yields an invalid datetime.
>
> This bug seems very related to issue #9596, because in these cases,
> datetime should be converted to date, not the opposite.
>
> A minimal showcase is attached. Unzip and run ./manage.py test
New description:
If I give date object as input to a filter on a DateTimeField field, like
this:
{{{
class Ticket(models.Model):
register_time = models.DateTimeField(auto_now_add=True)
day = datetime.strptime('2014-10-19', '%Y-%m-%d').date()
Ticket.objects.filter(register_time__gte=day)
}}}
I may get a pytz.exceptions.NonExistentTimeError exception. The exact
exception was:
pytz.exceptions.NonExistentTimeError: 2014-10-19 00:00:00
This date is the start of DST in my timezone:
{{{
TIME_ZONE = 'America/Sao_Paulo'
}}}
I believe this is a bug because Django tries to convert my input to a
datetime object before building the query, and in this case, this
conversion yields an invalid datetime.
This bug seems very related to issue #9596, because in these cases,
datetime should be converted to date, not the opposite.
A minimal showcase is attached. Unzip and run ./manage.py test
# Changes by Akshat verma
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/core/management/__init__.py", line 385, in
execute_from_command_line
utility.execute()
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/core/management/__init__.py", line 354, in execute
django.setup()
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/__init__.py", line 18, in setup
from django.utils.log import configure_logging
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/utils/log.py", line 10, in <module>
from django.views.debug import ExceptionReporter,
get_exception_reporter_filter
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/views/debug.py", line 10, in <module>
from django.http import (HttpResponse, HttpResponseServerError,
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/http/__init__.py", line 4, in <module>
from django.http.response import (HttpResponse, StreamingHttpResponse,
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/http/response.py", line 13, in <module>
from django.core.serializers.json import DjangoJSONEncoder
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/core/serializers/__init__.py", line 23, in <module>
from django.core.serializers.base import SerializerDoesNotExist
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/core/serializers/base.py", line 6, in <module>
from django.db import models
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/db/models/__init__.py", line 6, in <module>
from django.db.models.query import Q, QuerySet, Prefetch # NOQA
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/db/models/query.py", line 13, in <module>
from django.db.models.fields import AutoField, Empty
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/db/models/fields/__init__.py", line 15, in <module>
from django.db.models.lookups import default_lookups,
RegisterLookupMixin
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/db/models/lookups.py", line 6, in <module>
from django.utils import timezone
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/utils/timezone.py", line 149, in <module>
utc = pytz.utc if pytz else UTC()
AttributeError: 'module' object has no attribute 'utc'
#Here is my model, which worked fine before installing pytz
from django.db import models
from django.conf import settings
# Create your models here.
class Item(models.Model):
title = models.CharField(max_length=100, )
description = models.TextField()
seller = models.ForeignKey(settings.AUTH_USER_MODEL)
price = models.DecimalField(max_digits=11, decimal_places=2)
timestamp = models.DateTimeField(auto_now_add=True)
last_updated = models.DateTimeField(auto_now=True)
def __str__(self):
return self.title
#Edit: I added my views.py file
from .models import Item
from django.views.generic import ListView, DetailView
class ItemListView(ListView):
model = Item
class ItemDetailView(DetailView):
model = Item
def get_context_data(self, **kwargs):
# Call the base implementation first to get a context
#This is getting a dictionary of the objects being displayed to
#the template
context = super(ItemDetailView, self).get_context_data(**kwargs)
# Add in a QuerySet of all items up for trade by the user
context['other_things_for_trade_by_user'] =
Item.objects.filter(seller=context['seller'])
return context
--
Comment:
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/core/management/__init__.py", line 385, in
execute_from_command_line
utility.execute()
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/core/management/__init__.py", line 354, in execute
django.setup()
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/__init__.py", line 18, in setup
from django.utils.log import configure_logging
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/utils/log.py", line 10, in <module>
from django.views.debug import ExceptionReporter,
get_exception_reporter_filter
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/views/debug.py", line 10, in <module>
from django.http import (HttpResponse, HttpResponseServerError,
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/http/__init__.py", line 4, in <module>
from django.http.response import (HttpResponse, StreamingHttpResponse,
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/http/response.py", line 13, in <module>
from django.core.serializers.json import DjangoJSONEncoder
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/core/serializers/__init__.py", line 23, in <module>
from django.core.serializers.base import SerializerDoesNotExist
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/core/serializers/base.py", line 6, in <module>
from django.db import models
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/db/models/__init__.py", line 6, in <module>
from django.db.models.query import Q, QuerySet, Prefetch # NOQA
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/db/models/query.py", line 13, in <module>
from django.db.models.fields import AutoField, Empty
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/db/models/fields/__init__.py", line 15, in <module>
from django.db.models.lookups import default_lookups,
RegisterLookupMixin
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/db/models/lookups.py", line 6, in <module>
from django.utils import timezone
File "/Users/user/.virtualenvs/app/lib/python3.4/site-
packages/django/utils/timezone.py", line 149, in <module>
utc = pytz.utc if pytz else UTC()
AttributeError: 'module' object has no attribute 'utc'
#Here is my model, which worked fine before installing pytz
from django.db import models
from django.conf import settings
# Create your models here.
class Item(models.Model):
title = models.CharField(max_length=100, )
description = models.TextField()
seller = models.ForeignKey(settings.AUTH_USER_MODEL)
price = models.DecimalField(max_digits=11, decimal_places=2)
timestamp = models.DateTimeField(auto_now_add=True)
last_updated = models.DateTimeField(auto_now=True)
def __str__(self):
return self.title
#Edit: I added my views.py file
from .models import Item
from django.views.generic import ListView, DetailView
class ItemListView(ListView):
model = Item
class ItemDetailView(DetailView):
model = Item
def get_context_data(self, **kwargs):
# Call the base implementation first to get a context
#This is getting a dictionary of the objects being displayed to
#the template
context = super(ItemDetailView, self).get_context_data(**kwargs)
# Add in a QuerySet of all items up for trade by the user
context['other_things_for_trade_by_user'] =
Item.objects.filter(seller=context['seller'])
return context
--
Ticket URL: <https://code.djangoproject.com/ticket/24076#comment:5>
Django
Reporter: lvella | Owner: Akshat verma
Type: Bug | Status: assigned
Component: Documentation | Version: 1.6
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Easy pickings: 0 | UI/UX: 0
Changes (by Akshat verma):
* owner: nobody => Akshat verma
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/24076#comment:4>
Django
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Changes (by Adam Zapletal):
* cc: Adam Zapletal (added)
Comment:
I wonder if this issue can be closed with no documentation changes for the
following reasons:
1. The `__date__gte` lookup (mentioned in comment 1) and others for
`DateTimeField` have been implemented for a long time.
2. `pytz` has been removed as of Django 5.0.
3. Support for time zones has now been core to Django for almost 12 years.
4. The documentation for the lookup mentioned above seems sufficient to me
to help future users wondering how to compare a `DateTimeField` with a
`datetime.date`:
https://docs.djangoproject.com/en/dev/ref/models/querysets/#date.
However, I'd be happy to work on a documentation patch if someone can
provide an idea of what is needed to close this ticket.
Django
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Comment (by Mariusz Felisiak):
The following comment by Aymeric is crucial, and, as far as I'm aware,
it's not fixed:
> I'm going to requalify this as a documentation issue because comparing
dates with datetimes is probably a common error and I don't think the docs
warn against it. I don't know where the warning should be added, though.
Ticket URL: <https://code.djangoproject.com/ticket/24076#comment:5>
Django
Reporter: lvella | Owner: Adam Zapletal
Type: Bug | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Changes (by Adam Zapletal):
* has_patch: 0 => 1
Thanks! I opened a PR with an attempt at documenting this. Please let me
know if the content or location needs to be improved.
--
Ticket URL: <https://code.djangoproject.com/ticket/24076#comment:6>
Django
Reporter: lvella | Owner: Adam Zapletal
Type: Bug | Status: assigned
Component: Documentation | Version: 1.6
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Changes (by Sarah Boyce):
* needs_better_patch: 0 => 1
Ticket URL: <https://code.djangoproject.com/ticket/24076#comment:7>
Django
Reporter: lvella | Owner: Adam Zapletal
Type: Bug | Status: assigned
Component: Documentation | Version: 1.6
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Changes (by Adam Zapletal):
--
Ticket URL: <https://code.djangoproject.com/ticket/24076#comment:8>
Django
| Zapletal
Type: Bug | Status: assigned
Component: Documentation | Version: 1.6
Severity: Normal | Resolution:
| checkin
Changes (by Sarah Boyce):
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/24076#comment:9>
Django
Reporter: lvella | Owner: Adam
| Zapletal
Component: Documentation | Version: 1.6
Severity: Normal | Resolution: fixed
| checkin
Has patch: 1 | Needs documentation: 0
Changes (by Sarah Boyce <42296566+sarahboyce@…>):
* resolution: => fixed
* status: assigned => closed
Comment:
In [changeset:"99273fd525129a973639044dfb12cfd732d8f1d6" 99273fd]:
{{{#!CommitTicketReference repository=""
revision="99273fd525129a973639044dfb12cfd732d8f1d6"
Fixed #24076 -- Added warnings on usage of dates with DateTimeField and
datetimes with DateField.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24076#comment:10>