[Django] #19738: "manage.py shell" on a fresh project raises RuntimeWarning about naive datetime, if IPython is installed
Django
datetime, if IPython is installed
-----------------------------------------+------------------------
Reporter: carljm | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 1.4
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------
IPython uses SQLite internally to store shell session history. At the
beginning and end of every shell session, it issues a SQLite query to
get/save the history. For some reason this query includes a (timezone-
naive) datetime.
Django's SQLite DB backend registers a global datetime adapter function
which issues a warning in case it receives a naive datetime when timezone
support is activated.
In combination, this means that on a fresh `startproject` if you fire up
`manage.py shell` without making a single change to settings, if IPython
is installed you'll get "RuntimeWarning: SQLite received a naive datetime
(2013-02-04 17:14:13.070091) while time zone support is active." on shell
startup and exit. That's an ugly experience for a newcomer to Django - it
_looks_ to them as if Django is shipping a broken setup by default.
Probably this is Django's fault for registering a potentially-noisy
process-global SQLite adapter, and we should consider whether to just
remove that warning in the adapter. The model layer will already warn
about naive datetimes, so this warning is only needed for raw SQL.
--
Ticket URL: <https://code.djangoproject.com/ticket/19738>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
datetime, if IPython is installed
Reporter: carljm | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 1.4
(models, ORM) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by carljm):
* type: Uncategorized => Bug
* component: Uncategorized => Database layer (models, ORM)
--
Ticket URL: <https://code.djangoproject.com/ticket/19738#comment:1>
Django
datetime, if IPython is installed
Reporter: carljm | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 1.4
(models, ORM) | Resolution:
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by charettes):
* stage: Unreviewed => Accepted
Comment:
I've also been experiencing this issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/19738#comment:2>
Django
datetime, if IPython is installed
Reporter: carljm | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 1.4
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by crodjer):
I made an attempt to temporarily hide the warning, shouldn't be much
harmful.
https://github.com/crodjer/django/commit/8b24546f2176d47a3443184728a71b2807d2a168
Quoting the commit message:
> It is more of an ugly hack, required because of django injecting
> global callback in a database api.
>
> Ideally, instead of registering adapter for `datetime.datetime`
> (django.db.backends.sqlite3.base.py Line 76), django should do it
> for a pseudo wrapper over `datetime.datetime` and use it everywhere,
> so that other applications and libraries don't have to worry about
> the behaviour being infected.
[[BR]]
Also, while trying to debug this, I came across another similar issue
(#19789) which involved a DeprecationWarning in IPython. It is probably
irrelevant but still decided to link it here.
--
Ticket URL: <https://code.djangoproject.com/ticket/19738#comment:3>
Django
datetime, if IPython is installed
Reporter: carljm | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
* version: 1.4 => master
Comment:
https://github.com/django/django/pull/1404
--
Ticket URL: <https://code.djangoproject.com/ticket/19738#comment:4>
Django
datetime, if IPython is installed
Reporter: carljm | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 1 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by aaugustin):
I disagree with the approach in PR #1404 because it introduces a data
corruption bug.
--
Ticket URL: <https://code.djangoproject.com/ticket/19738#comment:5>
Django
datetime, if IPython is installed
Reporter: carljm | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 1 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by aaugustin):
Hiding the warning appears to be the least awful solution at this point.
crodjer's patch probably does the job.
--
Ticket URL: <https://code.djangoproject.com/ticket/19738#comment:6>
Django
datetime, if IPython is installed
Reporter: carljm | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 1 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by ptone):
Looking back at that PR, not sure what I was thinking other than maybe I
had badly misread Carl's suggestion of removing the warning, as "removing
the adapter" - other than that, I can't explain why I was taking that
approach.
So to be clear, the main edge case we don't have an solution for, is raw
sql issued by user's Django related code, that may use naive datetimes -
as these will not generate any warnings as they would in the standard
shell?
I think this is worth documenting/warning somehow - but is uncommon enough
it shouldn't be too noisy verbose.
--
Ticket URL: <https://code.djangoproject.com/ticket/19738#comment:7>
Django
datetime, if IPython is installed
Reporter: carljm | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 1 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by Sylvain_):
This warning is still present in:
{{{
$ python --version
Python 2.7.5+
$ python -c "import django; print(django.get_version())"
1.5.4
}}}
I'm a python newcomer and I was looking a way to respect $PYTHONSTARTUP.
Following #6682 I also installed IPython.
both ipython, and django came from ubuntu packages
{{{
$ python-django 1.5.4-1ubuntu1 High-level Python web development
framework
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 13.10
Release: 13.10
Codename: saucy
}}}
this gives:
{{{
$ python manage.py shell
/usr/lib/python2.7/dist-packages/django/db/backends/sqlite3/base.py:53:
RuntimeWarning: SQLite received a naive datetime (2014-01-26
18:40:37.242040) while time zone support is active.
RuntimeWarning)
/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/embed.py:239:
DeprecationWarning: With-statements now directly support multiple context
managers
with nested(self.builtin_trap, self.display_trap):
Python 2.7.5+ (default, Sep 19 2013, 13:49:51)
Type "copyright", "credits" or "license" for more information.
IPython 0.13.2 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]:
}}}
I found a workaround here: http://stackoverflow.com/questions/11708821
/django-ipython-sqlite-complains-about-naive-datetime
But this fixes only the sqlite warning.
{{{
$ ipython profile create
[ProfileCreate] Generating default config file:
u'/home/sylvain/.config/ipython/profile_default/ipython_config.py'
}}}
and added those lines
{{{
import warnings
import exceptions
warnings.filterwarnings("ignore", category=exceptions.RuntimeWarning,
module='django.db.backends.sqlite3.base', lineno=53)
}}}
rereading the [https://docs.djangoproject.com/en/1.5/intro/tutorial01/
tutorial]
I found a better way (ipython removed):
{{{
export PYTHONSTARTUP=$PWD/_pythonrc
export DJANGO_SETTINGS_MODULE=mysite.settings
$ cat _pythonrc
# Import the model classes we just wrote.
from polls.models import Poll, Choice
from django.utils import timezone
$ python
Python 2.7.5+ (default, Sep 19 2013, 13:49:51)
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> Poll.objects.all()
[<Poll: What's up?>]
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/19738#comment:8>
Django
datetime, if IPython is installed
Reporter: carljm | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 1 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by carljm):
I took a look at applying crodjer's patch, but there are two issues I see
with it:
1) It only solves half the problem. IPython sends a naive datetime to
SQLite both at session start and session end. Crodjer's patch is careful
to restore the warnings after session start, but there is still a warning
raised at session end.
2) In the latest (1.0+) versions of IPython, the public embed API no
longer gives us separate "init" and "start" methods, just a single
`start_ipython` method that does not return until the shell session is
ended. This means that we can no longer disable the warning, initialize
IPython, and then re-enable the warning before the user actually begins
their shell interaction.
We can of course solve both of those problems by simply ignoring the
warnings before starting IPython and never restoring them. But this means
that someone using raw SQL via the IPython shell could save naive
datetimes to their database without getting a warning, which re-opens a
(small) window to the data-corruption bug that worried Aymeric above.
It seems that some users are [http://stackoverflow.com/questions/11708821
/django-ipython-sqlite-complains-about-naive-datetime fixing this for
themselves] by unconditionally disabling the warning in settings.py, which
is much worse, because it means they could get silent data corruption
anywhere in their code at any time, not only when using the IPython shell.
So being purist about our response to this may end up putting at least
some of our users in an even worse situation with regard to possible data
corruption (though of course they are making an explicit choice to put
themselves in that situation).
Another ugly alternative (of many ugly alternatives) is that we could fix
this in at least recent IPython versions by actually monkeypatching
IPython itself to send timezone-aware, rather than naive, datetimes at
session start and end. This of course ties us to internal IPython
implementation details with no backwards-compatibility guarantees.
In summary: I have no good suggestions for fixing this. I think it is
Django who is being a badly-behaved library here by registering process-
global adapters in the SQLite library (and/or SQLite is a bad library for
exposing this adapter registration as a process-global API rather than a
per-connection API), but it seems we are caught between a rock and a hard
place.
--
Ticket URL: <https://code.djangoproject.com/ticket/19738#comment:9>
Django
datetime, if IPython is installed
Reporter: carljm | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/19738#comment:10>
Django
datetime, if IPython is installed
Reporter: carljm | Owner: aaugustin
Type: Bug | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by aaugustin):
* owner: nobody => aaugustin
* needs_better_patch: 1 => 0
* has_patch: 1 => 0
* status: new => assigned
Comment:
The landscape has changed quite a bit after
e9103402c0fa873aea58a6a11dba510cd308cb84.
I'm working on simplifying conversions of datetimes read from the
database. I may be able to fix this bug at the same time.
--
Ticket URL: <https://code.djangoproject.com/ticket/19738#comment:11>
Django
datetime, if IPython is installed
Reporter: carljm | Owner: aaugustin
Type: Bug | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by aaugustin):
* has_patch: 0 => 1
Comment:
https://github.com/django/django/pull/4601 fixes this.
--
Ticket URL: <https://code.djangoproject.com/ticket/19738#comment:12>
Django
datetime, if IPython is installed
Reporter: carljm | Owner: aaugustin
Type: Bug | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/19738#comment:13>
Django
datetime, if IPython is installed
Reporter: carljm | Owner: aaugustin
Component: Database layer | Version: master
(models, ORM) |
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Aymeric Augustin <aymeric.augustin@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"d9521f66b1851b0eacd55bc78f801dc64123e333" d9521f6]:
{{{
#!CommitTicketReference repository=""
revision="d9521f66b1851b0eacd55bc78f801dc64123e333"
Removed global timezone-aware datetime adapters.
Refs #23820.
Fixed #19738.
Refs #17755. In order not to introduce a regression for raw queries,
parameters are passed through the connection.ops.value_to_db_* methods,
depending on their type.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/19738#comment:14>
Django
datetime, if IPython is installed
Reporter: Carl Meyer | Owner: Aymeric
| Augustin
Type: Bug | Status: closed
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"e707e4c709c2e3f2dad69643eb838f87491891f8" e707e4c]:
{{{
#!CommitTicketReference repository=""
revision="e707e4c709c2e3f2dad69643eb838f87491891f8"
Refs #19738 -- Removed timezone conversion in SQL queries executed outside
of the ORM.
Per deprecation timeline.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/19738#comment:15>