[Django] #30317: model_to_dict() makes separate db calls for each missing field

13 views

Skip to first unread message

Django

unread,

Apr 3, 2019, 1:12:28 AM4/3/19

to django-...@googlegroups.com

#30317: model_to_dict() makes separate db calls for each missing field
------------------------------------------------+------------------------
Reporter: Chris Hranj | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Forms | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------------------+------------------------
Currently, if a `model_to_dict` is called on a model instance that does
not already contain all of its fields (as a result of a `.only()`, etc),
it will make an individual database call for every single missing field
when attempting to access it, unless the `fields` argument is passed and
happens to contain only fields from the `.only()` used to query the object
originally.

The following example explains it more clearly (I'm using shell_plus
(https://django-extensions.readthedocs.io/en/latest/shell_plus.html) to
see the database interactions):

{{{
>>> from django.forms.models import model_to_dict
>>> sample_book = Book(
... title='Some Book',
... author='Jane Doe',
... description='A good book.',
... publisher='ABC Publishing',
... some_field='some text',
... some_other_field='some more text',
... )
>>> sample_book.save()
INSERT INTO "books_book" ("title", "author", "description", "publisher",
"some_field", "some_other_field")
VALUES ('Some Book', 'Jane Doe', 'A good book.', 'ABC Publishing', 'some
text', 'some more text')

Execution time: 0.001141s [Database: default]

>>> book = Book.objects.only('title').first()
SELECT "books_book"."id",
"books_book"."title"
FROM "books_book"
ORDER BY "books_book"."id" ASC
LIMIT 1

Execution time: 0.000123s [Database: default]

>>> model_to_dict(book)
SELECT "books_book"."id",
"books_book"."author"
FROM "books_book"
WHERE "books_book"."id" = 3

Execution time: 0.000098s [Database: default]

SELECT "books_book"."id",
"books_book"."description"
FROM "books_book"
WHERE "books_book"."id" = 3

Execution time: 0.000149s [Database: default]

SELECT "books_book"."id",
"books_book"."publisher"
FROM "books_book"
WHERE "books_book"."id" = 3

Execution time: 0.000067s [Database: default]

SELECT "books_book"."id",
"books_book"."some_field"
FROM "books_book"
WHERE "books_book"."id" = 3

Execution time: 0.000064s [Database: default]

SELECT "books_book"."id",
"books_book"."some_other_field"
FROM "books_book"
WHERE "books_book"."id" = 3

Execution time: 0.000062s [Database: default]
}}}

Ideally, this should make fewer (or just one) database calls if there are
missing fields.

Please let me know if this would be a welcome/worthwhile optimization. I
would love to work on it, as I've never contributed to django before and I
would really like to. Thank you!

--
Ticket URL: <https://code.djangoproject.com/ticket/30317>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,

Apr 3, 2019, 4:35:28 AM4/3/19

to django-...@googlegroups.com

#30317: model_to_dict() makes separate db calls for each missing field

-------------------------------------+-------------------------------------

Reporter: Chris Hranj | Owner: nobody

Type: | Status: closed
Cleanup/optimization |
Component: Forms | Version: master
Severity: Normal | Resolution: wontfix

-------------------------------------+-------------------------------------
Changes (by Carlton Gibson):

* status: new => closed
* resolution: => wontfix

Comment:

Hi Chris.

Thanks for the report. (Unless you have an unforeseen concrete proposal in
mind) I think we have to say `wontfix` here.

`model_to_dict` is just using the field properties. It has no knowledge of
whether they deferred or not. (And that's not something we'd add there.)

So, you really need to be passing in objects that already have the
required fields loaded. That you're not doing this suggests you're over-
using `only() here. See again the warning note:

> The defer() method (and its cousin, only(), below) are only for advanced
use-cases. They provide an optimization for when you have analyzed your
queries closely and understand exactly what information you need and have
measured that the difference between returning the fields you need and the
full set of fields for the model will be significant.

If you're sure you're using `only()` correctly, your best bet is probably
to add a `defer(None)` and re-fetch once before running through
`model_to_dict()`.