[Django] #25774: Refactor of datetime expressions and better, official support for right-hand-side date part extraction
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Reporter: ryuusenshi | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version: master
(models, ORM) | Keywords:
Severity: Normal | db,expressions,date,time,extract,transform
Triage Stage: Unreviewed | Has patch: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
As previously stated in [https://code.djangoproject.com/ticket/25556 this
ticket] and [https://groups.google.com/forum/#!topic/django-
developers/GU8OrPUMjrc this thread], I have begun work on the following:
- making the so called datetime transforms (currently residing in
`django.db.models.lookups`) part of the officially supported ORM API as
part of the db expressions,
- making these transforms easily usable on the right hand side of
lookups / Q objects
[[BR]]
To this effect, I propose the following:
- replace module `django.db.models.expressions` with a package of the
same name
- add module `django.db.models.expressions.datetimes`
- rename `XTransform` to `XExtract` and move them from
`django.db.models.lookups` to `django.db.models.lookups` to
`django.db.models.expressions.datetimes`
- change `YearLookup` and `YearComparisonLookup` to allow for
functions on the right hand side
- extensive testing of these db expressions
- adding this to the release notes
- any other change that might support this
[[BR]]
I'm also adding a [PR] with some of these already implemented. What
remains to be done is:
- replace references `XTransform` type classes in
`django.db.models.lookups` with `XExtract` classes from
`django.db.models.expressions.datetimes`
- replace these references anywhere else in the project (I've noticed
there are some in the tests, for instance)
[[BR]]
With the API in the [PR] it becomes possible to do the following lookup:
{{{
from django.db.models.expressions.datetimes import DateExtract
Person.objects.filter(birth_date__year=DateExtract('job__start_date',
lookup_name='year'))
}}}
which is equivalent to:
{{{
from django.db.models.expressions.datetimes import YearExtract
Person.objects.filter(birth_date__year=YearExtract('job__start_date'))
}}}
Additionally, @jarshwah suggested that these should maybe be named:
`ExtractX` instead of `XExtract`.
--
Ticket URL: <https://code.djangoproject.com/ticket/25774>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Reporter: ryuusenshi | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version: master
(models, ORM) |
Keywords: | Triage Stage:
db,expressions,date,time,extract,transform| Unreviewed
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_docs: => 1
* needs_tests: => 1
* needs_better_patch: => 0
Old description:
New description:
As previously stated in [https://code.djangoproject.com/ticket/25556 this
ticket] and [https://groups.google.com/forum/#!topic/django-
developers/GU8OrPUMjrc this thread], I have begun work on the following:
- making the so called datetime transforms (currently residing in
`django.db.models.lookups`) part of the officially supported ORM API as
part of the db expressions,
- making these transforms easily usable on the right hand side of
lookups / Q objects
[[BR]]
To this effect, I propose the following:
- replace module `django.db.models.expressions` with a package of the
same name
- add module `django.db.models.expressions.datetimes`
- rename `XTransform` to `XExtract` and move them from
`django.db.models.lookups` to `django.db.models.lookups` to
`django.db.models.expressions.datetimes`
- change `YearLookup` and `YearComparisonLookup` to allow for
functions on the right hand side
- extensive testing of these db expressions
- adding this to the release notes
- any other change that might support this
[[BR]]
I'm also adding a [https://github.com/django/django/pull/5683 PR] with
some of these already implemented. What remains to be done is:
- replace references `XTransform` type classes in
`django.db.models.lookups` with `XExtract` classes from
`django.db.models.expressions.datetimes`
- replace these references anywhere else in the project (I've noticed
there are some in the tests, for instance)
[[BR]]
With the API in the [https://github.com/django/django/pull/5683 PR] it
becomes possible to do the following lookup:
{{{
from django.db.models.expressions.datetimes import DateExtract
Person.objects.filter(birth_date__year=DateExtract('job__start_date',
lookup_name='year'))
}}}
which is equivalent to:
{{{
from django.db.models.expressions.datetimes import YearExtract
Person.objects.filter(birth_date__year=YearExtract('job__start_date'))
}}}
Additionally, @jarshwah suggested that these should maybe be named:
`ExtractX` instead of `XExtract`.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:1>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Reporter: ryuusenshi | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version: master
(models, ORM) |
Keywords: | Triage Stage: Accepted
db,expressions,date,time,extract,transform|
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
* stage: Unreviewed => Accepted
Comment:
I think you're on the right track for most of these points. I'm not yet
convinced that adding `DateExtract` is necessary, but once the patch is
written I'll take another look then. I just worry that there'll be two
ways to extract a date part and that may lead to confusion.
Would like others opinions on ExtractX rather than XExtract. I think it
reads and sorts better.
YearLookup and YearComparisonLookup are going to be interesting. They'll
still need to work the same way when used on the left hand side. Perhaps
we can rename to YearLookupOptimised (for example, that's a bad name) and
register that as a lookup, but provide a different ExtractYear that does
what you'd expect on the right hand side. The optimised version would then
just be a hidden implementation detail that users shouldn't touch
themselves.
I'll have a look at the actual PR tomorrow. Great work so far.
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:2>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Reporter: ryuusenshi | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version: master
(models, ORM) |
Keywords: | Triage Stage: Accepted
db,expressions,date,time,extract,transform|
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Comment (by ryuusenshi):
Josh, I've actually tweaked `YearLookup` and `YearComparisonLookup`
already.
I've reversed the order of `process_lhs` and `process_rhs` to make
`process_rhs` occur first, and then based on the result of that
processing, specifically, based on whether there are any `rhs_params`
returned, the lhs processing branches to accommodate right hand side based
on both a parameter and a function.
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:3>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Reporter: ryuusenshi | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version: master
(models, ORM) |
Keywords: | Triage Stage: Accepted
db,expressions,date,time,extract,transform|
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
* cc: info+coding@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:4>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Keywords: | Triage Stage: Accepted
db,expressions,date,time,extract,transform|
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
* owner: nobody => jarshwah
* status: new => assigned
Comment:
Since David appears to have left this ticket I'll pick this up. Initial
patch: https://github.com/django/django/pull/6243
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:5>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Type: New feature | Status: assigned
(models, ORM) |
Keywords: | Triage Stage: Ready for
db,expressions,date,time,extract,transform| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_docs: 1 => 0
* needs_tests: 1 => 0
* stage: Accepted => Ready for checkin
Comment:
Patch is ready for review.
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:6>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Type: New feature | Status: assigned
(models, ORM) |
Keywords: | Triage Stage: Accepted
db,expressions,date,time,extract,transform|
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
* stage: Ready for checkin => Accepted
Comment:
The review queue is "Accepted + Has patch".
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:7>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Type: New feature | Status: assigned
(models, ORM) |
Keywords: | Triage Stage: Accepted
db,expressions,date,time,extract,transform|
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
* cc: aksheshdoshi@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:8>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Type: New feature | Status: assigned
(models, ORM) |
Keywords: | Triage Stage: Accepted
db,expressions,date,time,extract,transform|
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:9>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Type: New feature | Status: assigned
(models, ORM) |
Keywords: | Triage Stage: Accepted
db,expressions,date,time,extract,transform|
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:10>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Type: New feature | Status: assigned
(models, ORM) |
Keywords: | Triage Stage: Accepted
db,expressions,date,time,extract,transform|
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* keywords: db,expressions,date,time,extract,transform =>
db,expressions,date,time,extract,transform 1.10
* needs_better_patch: 0 => 1
Comment:
Left comments for improvement on the PR.
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:11>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Type: New feature | Status: assigned
(models, ORM) |
Keywords: | Triage Stage: Ready for
db,expressions,date,time,extract,transform| checkin
1.10 |
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
* stage: Accepted => Ready for checkin
Comment:
Ready for merge unless feature required for timezone support requiring
pytz as mentioned
https://github.com/django/django/pull/6243#issuecomment-219282120
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:12>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Type: New feature | Status: assigned
(models, ORM) |
Keywords: | Triage Stage: Accepted
db,expressions,date,time,extract,transform|
1.10 |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
* stage: Ready for checkin => Accepted
Comment:
I added another commit to provide tzinfo support for Extract subclasses.
New commit is:
https://github.com/django/django/pull/6243/commits/4f08553fb0431184fdcc3e0eb9c3c52fa2bba09e
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:13>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Type: New feature | Status: assigned
(models, ORM) |
Keywords: | Triage Stage: Ready for
db,expressions,date,time,extract,transform| checkin
1.10 |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:14>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Type: New feature | Status: closed
Component: Database layer | Version: master
(models, ORM) |
Keywords: | Triage Stage: Ready for
db,expressions,date,time,extract,transform| checkin
1.10 |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"2a4af0ea43512370764303d35bc5309f8abce666" 2a4af0ea]:
{{{
#!CommitTicketReference repository=""
revision="2a4af0ea43512370764303d35bc5309f8abce666"
Fixed #25774 -- Refactor datetime expressions into public API
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:16>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Keywords: | Triage Stage: Ready for
db,expressions,date,time,extract,transform| checkin
1.10 |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Comment (by Josh Smeaton <josh.smeaton@…>):
In [changeset:"77b73e79a4750dcbfabc528bf00cad81ff5bb4d9" 77b73e79]:
{{{
#!CommitTicketReference repository=""
revision="77b73e79a4750dcbfabc528bf00cad81ff5bb4d9"
Refs #25774 -- Made Oracle truncate microseconds if USE_TZ=False.
The tests for this change are in the fix for #25774.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:15>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Type: New feature | Status: closed
Component: Database layer | Version: master
(models, ORM) |
Keywords: | Triage Stage: Ready for
db,expressions,date,time,extract,transform| checkin
1.10 |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette <charette.s@…>):
In [changeset:"082c52dbedd76c312cebf3b23e04c449a94c20b6" 082c52db]:
{{{
#!CommitTicketReference repository=""
revision="082c52dbedd76c312cebf3b23e04c449a94c20b6"
Refs #25774, #26348 -- Allowed Trunc functions to operate with time
fields.
Thanks Josh for the amazing testing setup and Tim for the review.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:18>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Type: New feature | Status: closed
(models, ORM) |
Keywords: | Triage Stage: Ready for
db,expressions,date,time,extract,transform| checkin
1.10 |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette <charette.s@…>):
In [changeset:"90468079ec6f72a1b8b6a908d81d3201a3204b24" 9046807]:
{{{
#!CommitTicketReference repository=""
revision="90468079ec6f72a1b8b6a908d81d3201a3204b24"
Refs #25774 -- Adjusted datetime database function docs field names.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:17>
Django
hand-side date part extraction
-------------------------------------+-------------------------------------
Type: New feature | Status: closed
(models, ORM) |
Keywords: | Triage Stage: Ready for
db,expressions,date,time,extract,transform| checkin
1.10 |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette <charette.s@…>):
In [changeset:"cf6f0e9978af750855d61fcee42ffc75c62c81f9" cf6f0e9]:
{{{
#!CommitTicketReference repository=""
revision="cf6f0e9978af750855d61fcee42ffc75c62c81f9"
[1.10.x] Refs #25774 -- Adjusted datetime database function docs field
names.
Backport of 90468079ec6f72a1b8b6a908d81d3201a3204b24 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25774#comment:19>