[Django] #35683: django.utils.timezone.make_naive can underflow for timezones close to datetime.min

[Django]

#35683: django.utils.timezone.make_naive can underflow for timezones close to
datetime.min
-----------------------+-----------------------------------------
Reporter: tybug | Type: Bug
Status: new | Component: Uncategorized
Version: 5.1 | Severity: Normal
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------+-----------------------------------------
I'm a contributor over at [https://github.com/HypothesisWorks/hypothesis
Hypothesis].
[https://github.com/HypothesisWorks/hypothesis/pull/4086#issuecomment-2290742590
We found] that saving (or retrieving?) models with aware DateTimeField
values close to datetime.min can error if:
* timezone support is enabled (USE_TZ=True)
* you are using the sqlite database (presumably also any other db backend
without tz aware support, if they exist?)

Explicitly:

{{{
from datetime import datetime, timezone
from zoneinfo import ZoneInfo
from django.utils.timezone import make_naive

dt = datetime.min.replace(tzinfo=ZoneInfo(key='Africa/Addis_Ababa'))
make_naive(dt, timezone.utc) # OverflowError
}}}

where make_naive is called from
[https://github.com/django/django/blob/a57596e443ecb67140e1a9fc0f0e87446b2d0174/django/db/backends/sqlite3/operations.py#L269
here]. I would guess the overflow case for datetime.max is symmetric and
similar as well.

I have half an expectation that this will be a wontfix, but I figured I'd
give a heads up here. I hope you'll forgive the lack of a django
reproducer - I live in testing land and don't have a local django
environment handy :). Hopefully the cause is clear and reproducible with
vanilla django models.
--
Ticket URL: <https://code.djangoproject.com/ticket/35683>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35683: django.utils.timezone.make_naive can underflow for timezones close to
datetime.min

-------------------------------+--------------------------------------
Reporter: tybug | Owner: (none)

Type: Bug | Status: new
Component: Uncategorized | Version: 5.1

Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------+--------------------------------------
Description changed by tybug:

Old description:

> I'm a contributor over at [https://github.com/HypothesisWorks/hypothesis
> Hypothesis].
> [https://github.com/HypothesisWorks/hypothesis/pull/4086#issuecomment-2290742590
> We found] that saving (or retrieving?) models with aware DateTimeField
> values close to datetime.min can error if:
> * timezone support is enabled (USE_TZ=True)
> * you are using the sqlite database (presumably also any other db backend
> without tz aware support, if they exist?)
>
> Explicitly:
>
> {{{
> from datetime import datetime, timezone
> from zoneinfo import ZoneInfo
> from django.utils.timezone import make_naive
>
> dt = datetime.min.replace(tzinfo=ZoneInfo(key='Africa/Addis_Ababa'))
> make_naive(dt, timezone.utc) # OverflowError
> }}}
>
> where make_naive is called from
> [https://github.com/django/django/blob/a57596e443ecb67140e1a9fc0f0e87446b2d0174/django/db/backends/sqlite3/operations.py#L269
> here]. I would guess the overflow case for datetime.max is symmetric and
> similar as well.
>
> I have half an expectation that this will be a wontfix, but I figured I'd
> give a heads up here. I hope you'll forgive the lack of a django
> reproducer - I live in testing land and don't have a local django
> environment handy :). Hopefully the cause is clear and reproducible with
> vanilla django models.

New description:

I'm a contributor over at [https://github.com/HypothesisWorks/hypothesis
Hypothesis].
[https://github.com/HypothesisWorks/hypothesis/pull/4086#issuecomment-2290742590
We found] that saving (or retrieving?) models with aware DateTimeField
values close to datetime.min can error if:
* timezone support is enabled (USE_TZ=True)
* you are using the sqlite database (presumably also any other db backend
without tz aware support, if they exist?)

Explicitly:

{{{
from datetime import datetime, timezone
from zoneinfo import ZoneInfo
from django.utils.timezone import make_naive

dt = datetime.min.replace(tzinfo=ZoneInfo(key='Africa/Addis_Ababa'))
make_naive(dt, timezone.utc) # OverflowError
}}}

where make_naive is called from
[https://github.com/django/django/blob/a57596e443ecb67140e1a9fc0f0e87446b2d0174/django/db/backends/sqlite3/operations.py#L269
here]. I would guess the overflow case for datetime.max is symmetric and
similar as well.

I hope you'll forgive the lack of a django reproducer - I live in testing
land and don't have a local django environment handy :). Hopefully the
cause is clear and reproducible with vanilla django models.

--

--
Ticket URL: <https://code.djangoproject.com/ticket/35683#comment:1>

[Django]

#35683: django.utils.timezone.make_naive can underflow for timezones close to
datetime.min

-------------------------------------+-------------------------------------
Reporter: Liam DeVoe | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Sarah Boyce):

* component: Uncategorized => Database layer (models, ORM)
* stage: Unreviewed => Accepted

Comment:

Replicated, thank you!

Should be an issue in MySQL and Oracle as well - refs #23820
Rough testcase:
{{{#!diff
--- a/tests/backends/sqlite/test_operations.py
+++ b/tests/backends/sqlite/test_operations.py
@@ -1,8 +1,10 @@
import unittest
+from datetime import datetime

from django.core.management.color import no_style
from django.db import connection
-from django.test import TestCase
+from django.test import TestCase, override_settings
+from django.utils import timezone

from ..models import Person, Tag

@@ -86,3 +88,10 @@ class SQLiteOperationsTests(TestCase):
"zzz'",
statements[-1],
)
+
+ @override_settings(USE_TZ=True, TIME_ZONE="UTC")
+ def test_adapt_datetimefield_value_close_to_datetime_min(self):
+ africa_nairobi_tz = timezone.get_fixed_timezone(180)
+ value = datetime.min.replace(tzinfo=africa_nairobi_tz)
+
+ connection.ops.adapt_datetimefield_value(value) # OverflowError
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35683#comment:2>

[Django]

#35683: django.utils.timezone.make_naive can underflow for timezones close to
datetime.min
-------------------------------------+-------------------------------------
Reporter: Liam DeVoe | Owner:

| shubhamsugara22
Type: Bug | Status: assigned

Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by shubhamsugara22):

* owner: (none) => shubhamsugara22
* status: new => assigned

Comment:

will try to work on it but will have to check first
Bug is causing issue on other db as well if so look for fix
--
Ticket URL: <https://code.djangoproject.com/ticket/35683#comment:3>

[Django]

#35683: django.utils.timezone.make_naive can underflow for timezones close to
datetime.min
-------------------------------------+-------------------------------------

Reporter: Liam DeVoe | Owner: Shubham
| Singh Sugara

Type: Bug | Status: assigned
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham):

What behavior do you expect? If the timezone conversion causes the
datetime to be too small or too big, what can be done?
--
Ticket URL: <https://code.djangoproject.com/ticket/35683#comment:4>

[Django]

#35683: django.utils.timezone.make_naive can underflow for timezones close to
datetime.min
-------------------------------------+-------------------------------------
Reporter: Liam DeVoe | Owner: Shubham
| Singh Sugara
Type: Bug | Status: assigned
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Shubham Singh Sugara):

Replying to [comment:4 Tim Graham]:

> What behavior do you expect? If the timezone conversion causes the
datetime to be too small or too big, what can be done?

For now i can think of this solution
** Max case should be also included if datetime.min is giving error than
there is a possibility datetime.max also giving the error due to corner
cases

We can Update make_naive Function:
Modify the make_naive function in Django to handle boundary cases(max
and min). For instance, if an OverflowError is detected, the function
could return the closest safe datetime value or handle the error
gracefully in another way.

for now i think this can work across all db as changing other parts of
modules will be db specific
but if any other suggestion open to discussion
--
Ticket URL: <https://code.djangoproject.com/ticket/35683#comment:5>

[Django]

#35683: django.utils.timezone.make_naive can underflow for timezones close to
datetime.min
-------------------------------------+-------------------------------------
Reporter: Liam DeVoe | Owner: Shubham
| Singh Sugara
Type: Bug | Status: assigned
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham):

The use case for storing `datetime.min` and `max` is unclear, but it seems
doubtful that we should silently change the value the user provided. I
would close this ticket as "needs info" or "wontfix" unless the reporter
can provide more explanation.
--
Ticket URL: <https://code.djangoproject.com/ticket/35683#comment:6>

[Django]

#35683: django.utils.timezone.make_naive can underflow for timezones close to
datetime.min
-------------------------------------+-------------------------------------
Reporter: Liam DeVoe | Owner: Shubham
| Singh Sugara
Type: Bug | Status: assigned
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Liam DeVoe):

I half expected this ticket to be wontfix when I opened it :). Here's one
sane alternative I can think of: if you're converting to naive solely for
serializing to the db, and when you deserialize from the db you make it tz
aware again, then the only time it's out of bounds for the
serialization/storage portion. But you control the serialization format,
so you could extend the limits by 24 hours in each direction to
accommodate those overflows, relying on the invariant that they will be
made immediately tz aware and thus in bounds before reaching the user.

I'm not familiar with django internals, so this may not be acceptable or
the assumed invariants may not be correct - that's up to you guys. I agree
that silently capping is not a good idea. Possibly this should just be
wontifx'd but documented; it does seem surprising to a user that I cannot
store an in-bounds tz aware datetime, because there is no reason to expect
it is converted to a naive datetime.
--
Ticket URL: <https://code.djangoproject.com/ticket/35683#comment:7>

[Django]

#35683: django.utils.timezone.make_naive can underflow for timezones close to
datetime.min
-------------------------------------+-------------------------------------
Reporter: Liam DeVoe | Owner: Shubham
| Singh Sugara

Type: Bug | Status: closed

Component: Database layer | Version: 5.1
(models, ORM) |

Severity: Normal | Resolution: wontfix

Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce):

* resolution: => wontfix
* status: assigned => closed

Comment:

Thank you both for the discussion!
Given I believe this was found and validated via testing scenarios (rather
than "in real life"), and much of this code has been this way for years,
going to update to wontfix.
If anyone finds that this causes them an issue with their application,
they're welcome to add this discussion.
--
Ticket URL: <https://code.djangoproject.com/ticket/35683#comment:8>