[Django] #32406: Allow QuerySet.update() to return fields on supported backends.
Django
-------------------------------------+-------------------------------------
Reporter: Tom | Owner: nobody
Carrick |
Type: New | Status: assigned
feature |
Component: Database | Version: master
layer (models, ORM) |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
[https://groups.google.com/g/django-developers/c/qQ5DT91nBLM Mailing list]
For example:
{{{
Foo.objects.update(x="abc", returning=["pk", "x"])
}}}
To return something like:
{{{
[
{"pk": 1, "x": "abc"},
{"pk": 2, "x": "abc"},
{"pk": 3, "x": "abc"},
]
}}}
The exact API and implementation is still a little unclear, but there
seems to be support for doing ''something''.
--
Ticket URL: <https://code.djangoproject.com/ticket/32406>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
| Carrick
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: nobody => Tom Carrick
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:1>
Django
-------------------------------------+-------------------------------------
| Carrick
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Unreviewed => Accepted
Comment:
OK, I'll provisionally accept on the basis of the discussion.
It would be good if we could pin down an API that was generally agreed
upon (ref the return value seems the main sticking point) before
implementation began.
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:2>
Django
-------------------------------------+-------------------------------------
| Carrick
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Chris Jerdonek):
Recently, on issue #32381 about `bulk_update()`,
[https://code.djangoproject.com/ticket/32381#comment:1 I suggested] the
idea of returning different values as attributes of a single result object
(e.g. a `namedtuple`). That was about `bulk_update()` rather than
`update()`, but the principle is the same. In this case, the two possible
values under consideration could be approximately named something like
`number_matched` and `values`.
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:3>
Django
-------------------------------------+-------------------------------------
| Carrick
Component: Database layer | Version: master
Severity: Normal | worksforme
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => worksforme
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:4>
Django
-------------------------------------+-------------------------------------
Type: | Carrick
Cleanup/optimization | Status: closed
Component: Database layer | Version: master
(models, ORM) | Resolution:
Severity: Normal | worksforme
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 1 | UI/UX: 1
-------------------------------------+-------------------------------------
Changes (by Hery127):
* cc: Hery127 (added)
* needs_better_patch: 0 => 1
* needs_tests: 0 => 1
* easy: 0 => 1
* keywords: => Mango
* needs_docs: 0 => 1
* has_patch: 0 => 1
* ui_ux: 0 => 1
* type: New feature => Cleanup/optimization
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:5>
Django
-------------------------------------+-------------------------------------
| Carrick
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Diego Lima (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:4>
Django
-------------------------------------+-------------------------------------
| Carrick
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Johannes Maron (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:5>
Django
-------------------------------------+-------------------------------------
| Carrick
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Johannes Maron):
Hi there, I implemented the `returning` support in the past. I believe
this feature is possible, however, I think we need to be sure how this
differs from setting `db_returning` on the fields itself. In case you
didn't know (it's not documented yet) You have multiple return values by
setting that attribute to true. I would be curious if you could provide a
more detailed use case. Best, Joe
FYI, this might be somewhat related to #30032 and #470
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:6>
Django
-------------------------------------+-------------------------------------
| Carrick
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Tom Carrick):
Johannes, I originally saw a couple of use-cases:
1. There is some before update trigger that changes the data. If I
understand it, `db_returning` should cover this (I had no idea it existed
as it's not documented).
2. You are creating an API (or not an API) with a bulk update feature,
and you want to return the results to the user without making another
query to gather them.
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:7>
Django
-------------------------------------+-------------------------------------
| Carrick
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Johannes Maron):
Hi there,
Interesting cases. DB triggers make things slightly more difficult as the
`db_returning` feature is currently only tested for inserts not updates.
I would see two things here, first, to add `db_retuning` support to a
`save` call (single object update). Second, you could build on those API
changes to add this functionality to `update`.
Honestly, with #470 on its way. It stands to reason, if it made sense to
refresh and object from the database by default. But that's a mailing list
discussion for future me ;)
In any event, this proposal seems justified to me. If you find the time to
tackle it, I am happy to help out with reviews.
Best,
Joe
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:8>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: new
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: Tom Carrick => (none)
* status: assigned => new
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:9>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: new
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Michael Rosner (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:10>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: assigned
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: (none) => aivarsk
* status: new => assigned
Comment:
I have a proof of concept in my branch, I will clean up and open a PR this
week. But I took a slightly different approach so I would like to get some
feedback.
I added a new `.update_returning` method that returns a query set instead
of changing the existing `.update` that returns a number. That allows us
to do things like:
we can get the model after the update which is the main use case IMO. I
have used this approach in accounting systems, there are some articles
showing when it's useful https://hakibenita.com/django-concurrency#update-
and-immediately-return
{{{
#!python
updated = (
UpdateReturningModel.objects.filter(key=obj.key)
.update_returning(hits=F("hits") + 1)
.get()
)
}}}
we can also do some lazy loading
{{{
#!python
updated = (
UpdateReturningModel.objects.filter(key=obj.key)
.update_returning(hits=F("hits") + 1)
.only("hits")
.get()
)
updated = (
UpdateReturningModel.objects.filter(key=obj.key)
.update_returning(hits=F("hits") + 1)
.defer("hits", "content")
.get()
)
}}}
or keep working without models
{{{
#!python
updated = UpdateReturningModel.objects.update_returning(
hits=F("hits") + 1
).values("pk", "hits")
updated = UpdateReturningModel.objects.update_returning(
hits=F("hits") + 1
).values_list("hits", flat=True)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:11>
Django
-------------------------------------+-------------------------------------
| Kalvāns
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
Comment:
I created a [https://github.com/django/django/pull/17307 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:12>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak):
We don't have, and never need a separate method for `INSERT ...
RETURNING`), so I'm not sure why you want to handle `UPDATE ... RETURNING`
in a separate method.
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:13>
Django
-------------------------------------+-------------------------------------
| Kalvāns
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Aivars Kalvāns):
Just because the {{{update}}} is returning the number of rows updated. I
can do the same with {{{Foo.objects.update(x="newx", returning=True)}}}
assuming no model will have a {{{returning}}} field. Whatever fits best, I
just want to be able to do {{{returning}}} without executing a raw SQL.
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:14>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak):
Replying to [comment:14 Aivars Kalvāns]:
> Just because the {{{update}}} is returning the number of rows updated. I
can do the same with {{{Foo.objects.update(x="newx", returning=True)}}}
assuming no model will have a {{{returning}}} field. Whatever fits best, I
just want to be able to do {{{returning}}} without executing a raw SQL.
Have you seen the [https://groups.google.com/g/django-
developers/c/qQ5DT91nBLM mailing list discussion]? There are some
proposals to use the current methods without breaking backward
compatibility, e.g. `namedtuple`.
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:15>
Django
-------------------------------------+-------------------------------------
| Kalvāns
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Aivars Kalvāns):
Yes I saw it. But I did not see there a way to implement the new behavior
without breaking the API. Is there something I missed? Returning a tuple
with {{{(n, [data])}}} will break the code that expects a simple integer,
making the {{{returning}}} kwarg special also can break something (but
unlikely) and chaining of method calls like {{{.update().value()}}} is not
possible because the update is expected to execute the SQL before
returning.
So asides from naming the function I think that returning models by
default is nice because it includes PK and the result is "usable". If you
don't need the models, you can ask for values or values_list. There are
people using {{{QuerySet.raw()}}} to achieve similar result
(https://hakibenita.com/django-concurrency#update-and-immediately-return)
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:16>
Django
-------------------------------------+-------------------------------------
| Kalvāns
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Aivars Kalvāns):
Maybe doing {{{Foo.objects.update(x="newx", returning=True)}}} and
returning a QuerySet would work better. I found only a single project with
{{{returning}}} model field in github
https://github.com/annalee/alienplanit/blob/b54722d683a5e8eba03f4467a367bcf24339bb32/submissions/models.py#L35
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:17>
Django
-------------------------------------+-------------------------------------
| Kalvāns
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak):
Please describe your proposition on the [https://groups.google.com/g
/django-developers/c/qQ5DT91nBLM mailing list], where you'll reach a wider
audience and see what other think.
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:18>
Django
-------------------------------------+-------------------------------------
| Kalvāns
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: John Speno (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:19>
Django
-------------------------------------+-------------------------------------
| Kalvāns
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
Marking "Patch needs improvement" as the API is being discussed and the
current proposal in the discussion is different to the patch
--
Ticket URL: <https://code.djangoproject.com/ticket/32406#comment:20>