[Django] #28459: Improve performance of ValuesListIterable

Django

unread,

Aug 1, 2017, 2:38:38 PM8/1/17

to django-...@googlegroups.com

#28459: Improve performance of ValuesListIterable
-------------------------------------+-------------------------------------
Reporter: Sergey | Owner: nobody
Fedoseev |
Type: | Status: new
Cleanup/optimization |
Component: Database | Version: master
layer (models, ORM) |
Severity: Normal | Keywords:
Triage Stage: Accepted | Has patch: 1
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Before:
{{{
In [2]: %timeit for x in City.objects.values_list('id'): pass
106 ms ± 2.88 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [3]: %timeit for x in City.objects.values_list('id', 'id'): pass
124 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}

After:
{{{
In [2]: %timeit for x in City.objects.values_list('id'): pass
77.7 ms ± 1.49 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [3]: %timeit for x in City.objects.values_list('id', 'id'): pass
95.9 ms ± 1.71 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}

[https://github.com/django/django/pull/8835 PR]

--
Ticket URL: <https://code.djangoproject.com/ticket/28459>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,

Aug 1, 2017, 3:28:10 PM8/1/17

to django-...@googlegroups.com

#28459: Improve performance of ValuesListIterable
-------------------------------------+-------------------------------------

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Tim Graham <timograham@…>):

* status: new => closed
* resolution: => fixed

Comment:

In [changeset:"2d136ede8abd6876b33607eea094e6903eb77d77" 2d136ede]:
{{{
#!CommitTicketReference repository=""
revision="2d136ede8abd6876b33607eea094e6903eb77d77"
Fixed #28459 -- Improved performance of ValuesListIterable.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:1>

Django

unread,

Aug 1, 2017, 4:50:53 PM8/1/17

to django-...@googlegroups.com

#28459: Improve performance of QuerySet

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev):

[https://github.com/django/django/pull/8843 PR (Improved performance of
Model.from_db() when fields are deferred)]

Before:
{{{
In [3]: %timeit for x in City.objects.only('id'): pass
1.07 s ± 10.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
}}}
After:
{{{
In [2]: %timeit for x in City.objects.only('id'): pass
961 ms ± 5.68 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:2>

Django

unread,

Aug 2, 2017, 3:44:17 AM8/2/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev):

[https://github.com/django/django/pull/8845 PR (Improved performance of
select_related() when model is prefetched from its parent.)]

Before:
{{{
In [2]: %timeit for x in Human.objects.select_related('person'): pass
196 ms ± 2.05 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}

After:
{{{
In [2]: %timeit for x in Human.objects.select_related('person'): pass
186 ms ± 1.59 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:3>

Django

unread,

Aug 2, 2017, 9:16:47 AM8/2/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham <timograham@…>):

In [changeset:"78f7f6802175c6ef9a1c5160bbad0b9d2661f603" 78f7f68]:
{{{
#!CommitTicketReference repository=""
revision="78f7f6802175c6ef9a1c5160bbad0b9d2661f603"
Refs #28459 -- Improved performance of Model.from_db() when fields are
deferred.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:4>

Django

unread,

Aug 3, 2017, 3:22:39 PM8/3/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev):

[https://github.com/django/django/pull/8849 PR (Improved performance of
SQLCompiler.results_iter())]

Before:
{{{
In [3]: %timeit for x in City.objects.values_list('id'): pass
74 ms ± 1.45 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [2]: %timeit for x in City.objects.annotate(v=models.Value(1,
output_field=models.IntegerField())).values_list('v'): pass
343 ms ± 2.62 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
}}}
After:
{{{
In [4]: %timeit for x in City.objects.values_list('id'): pass
69.1 ms ± 744 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [3]: %timeit for x in City.objects.annotate(v=models.Value(1,
output_field=models.IntegerField())).values_list('v'): pass
254 ms ± 1.61 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:5>

Django

unread,

Aug 4, 2017, 2:30:27 AM8/4/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev):

[https://github.com/django/django/pull/8851 PR (Optimized ModelState
instantiation)]

Before:
{{{
In [5]: %timeit Skill()
6.05 µs ± 54.2 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [4]: %timeit for x in Skill.objects.all(): pass
156 ms ± 1.28 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}
After:
{{{
In [2]: %timeit Skill()
5.78 µs ± 29.5 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [3]: %timeit for x in Skill.objects.all(): pass
149 ms ± 1.62 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:6>

Django

unread,

Aug 7, 2017, 2:54:12 AM8/7/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Josh Smeaton):

These are awesome! We need more of these kind of patches.

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:7>

Django

unread,

Aug 7, 2017, 5:06:32 PM8/7/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham <timograham@…>):

In [changeset:"ca46f4688cd609ec3a056fc4c66fb17176a9d712" ca46f468]:
{{{
#!CommitTicketReference repository=""
revision="ca46f4688cd609ec3a056fc4c66fb17176a9d712"
Refs #28459 -- Improved performance of SQLCompiler.results_iter().
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:8>

Django

unread,

Aug 7, 2017, 7:49:33 PM8/7/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham <timograham@…>):

In [changeset:"ba1d82df63a5cc49c4a472fc57acdb735c6fb66c" ba1d82df]:
{{{
#!CommitTicketReference repository=""
revision="ba1d82df63a5cc49c4a472fc57acdb735c6fb66c"
Refs #28459 -- Improved performance of select_related() when model is
prefetched from its parent.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:9>

Django

unread,

Aug 8, 2017, 2:13:05 AM8/8/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev):

[https://github.com/django/django/pull/8867 PR (Improved performance of
ValuesIterable)]

Before:
{{{
In [6]: %timeit -n 10 for x in City.objects.values('id', 'b'): pass
212 ms ± 4.3 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}
After:
{{{
In [5]: %timeit for x in City.objects.values('id', 'b'): pass
151 ms ± 850 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:10>

Django

unread,

Aug 8, 2017, 2:14:34 AM8/8/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev):

[https://github.com/django/django/pull/8854 PR (Improved performance of
BaseExpression.convert_value())]

Before:
{{{

In [2]: %timeit for x in City.objects.annotate(v=models.Value(1,
output_field=models.IntegerField())).values_list('v'): pass

262 ms ± 3.42 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [3]: %timeit for x in City.objects.annotate(v=models.Value('1',
output_field=models.TextField())).values_list('v'): pass
243 ms ± 3.77 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
}}}
After:
{{{

In [2]: %timeit for x in City.objects.annotate(v=models.Value(1,
output_field=models.IntegerField())).values_list('v'): pass

191 ms ± 2.08 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [3]: %timeit for x in City.objects.annotate(v=models.Value('1',
output_field=models.TextField())).values_list('v'): pass
65.6 ms ± 878 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:11>

Django

unread,

Aug 8, 2017, 8:22:25 AM8/8/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev <fedoseev.sergey@…>):

In [changeset:"fb0bdff981deac05388620d5670612eb96183bba" fb0bdff]:
{{{
#!CommitTicketReference repository=""
revision="fb0bdff981deac05388620d5670612eb96183bba"
Refs #28459 -- Improved performance of ValuesIterable.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:12>

Django

unread,

Aug 8, 2017, 11:56:14 AM8/8/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev):

[https://github.com/django/django/pull/8871 PR (Improved performance of
sql.compiler.cursor_iter())]
Before:
{{{
In [6]: %timeit for x in City.objects.values_list('id'): pass
67.2 ms ± 1.03 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}
After:
{{{
In [6]: %timeit for x in City.objects.values_list('id'): pass
58 ms ± 580 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:13>

Django

unread,

Aug 8, 2017, 2:07:17 PM8/8/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham <timograham@…>):

In [changeset:"b8731c3fff3f36ed96655e74ab1c86263eba0119" b8731c3]:
{{{
#!CommitTicketReference repository=""
revision="b8731c3fff3f36ed96655e74ab1c86263eba0119"
Refs #28459 -- Improved performance of sql.compiler.cursor_iter().
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:14>

Django

unread,

Aug 10, 2017, 11:06:44 AM8/10/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham <timograham@…>):

In [changeset:"22ff86ec52bb536ad9d1fa3ee0c9a3d5eb9925c1" 22ff86e]:
{{{
#!CommitTicketReference repository=""
revision="22ff86ec52bb536ad9d1fa3ee0c9a3d5eb9925c1"
Refs #28459 -- Made Oracle get_db_converters() return converter for empty
strings only when it's needed.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:15>

Django

unread,

Aug 10, 2017, 3:08:41 PM8/10/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev):

[https://github.com/django/django/pull/8878 PR (Improved performance of
loading DecimalField on SQLite)]
Before:
{{{
In [5]: %timeit for x in Test.objects.values_list('decimal'): pass
150 ms ± 1.76 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [6]: %timeit for x in Test.objects.values_list(models.F('decimal') +
1): pass
123 ms ± 1.08 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}
After:
{{{
In [2]: %timeit for x in Test.objects.values_list('decimal'): pass
28.8 ms ± 1.05 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [3]: %timeit for x in Test.objects.values_list(models.F('decimal') +
1): pass
84.6 ms ± 541 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:16>

Django

unread,

Aug 10, 2017, 3:32:43 PM8/10/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham <timograham@…>):

In [changeset:"a146b65628e702a9a3ed5be21542ca45366fbb29" a146b656]:
{{{
#!CommitTicketReference repository=""
revision="a146b65628e702a9a3ed5be21542ca45366fbb29"
Refs #28459 -- Improved performance of loading DecimalField on SQLite.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:17>

Django

unread,

Aug 11, 2017, 4:02:44 PM8/11/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham <timograham@…>):

In [changeset:"5cc746206726c538c36a2830e7c068f1c8a0e7c8" 5cc74620]:
{{{
#!CommitTicketReference repository=""
revision="5cc746206726c538c36a2830e7c068f1c8a0e7c8"
Refs #28459 -- Optimized ModelState instantiation.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:18>

Django

unread,

Aug 11, 2017, 6:35:04 PM8/11/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham <timograham@…>):

In [changeset:"47ccefeada926ffbccaa354dec7a987a7e7ca701" 47ccefea]:
{{{
#!CommitTicketReference repository=""
revision="47ccefeada926ffbccaa354dec7a987a7e7ca701"
Refs #28459 -- Improved performance of BaseExpression.convert_value().
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:19>

Django

unread,

Aug 18, 2017, 6:01:14 AM8/18/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev):

[https://github.com/django/django/pull/8934 PR (Improved performance of
FlatValuesListIterable)]

Before:
{{{
In [3]: %timeit -n 100 for x in City.objects.values_list('id', flat=True):
pass
61.2 ms ± 719 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
}}}
After:
{{{
In [30]: %timeit -n 100 for x in City.objects.values_list('id',
flat=True): pass
58.8 ms ± 514 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:20>

Django

unread,

Aug 18, 2017, 6:53:40 AM8/18/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev):

[https://github.com/django/django/pull/8935 PR (Improved performance of
SQLCompiler.apply_converters())]

Before:
{{{
In [3]: %timeit for x in City.objects.annotate(v=models.Value(1,
output_field=models.IntegerField())).values_list('v'): pass
176 ms ± 725 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [6]: %timeit for x in City.objects.annotate(v=models.Value(1,
output_field=models.IntegerField())).values('v'): pass
257 ms ± 2.18 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
}}}
After:
{{{
In [3]: %timeit for x in City.objects.annotate(v=models.Value(1,
output_field=models.IntegerField())).values_list('v'): pass
170 ms ± 1.25 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [5]: %timeit for x in City.objects.annotate(v=models.Value(1,
output_field=models.IntegerField())).values('v'): pass
224 ms ± 1.88 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:21>

Django

unread,

Aug 22, 2017, 11:26:29 AM8/22/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham <timograham@…>):

In [changeset:"4dfd6b88d520b43b6363946e5ee58ba14cd1efe6" 4dfd6b88]:
{{{
#!CommitTicketReference repository=""
revision="4dfd6b88d520b43b6363946e5ee58ba14cd1efe6"
Refs #28459 -- Improved performance of FlatValuesListIterable.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:22>

Django

unread,

Aug 23, 2017, 4:26:41 PM8/23/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham <timograham@…>):

In [changeset:"dca67bb2c27cfa1b23cb6519c544ce467813e0ec" dca67bb]:
{{{
#!CommitTicketReference repository=""
revision="dca67bb2c27cfa1b23cb6519c544ce467813e0ec"
Refs #28459 -- Improved performance of SQLCompiler.apply_converters().
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:23>

Django

unread,

Sep 13, 2017, 9:00:57 AM9/13/17

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev):

[https://github.com/django/django/pull/9070 PR (Improved performance of
loading models from DB)]

{{{
class TestModel(models.Model):
field1 = models.IntegerField(null=True)
field2 = models.IntegerField(null=True)
field3 = models.IntegerField(null=True)
field4 = models.IntegerField(null=True)
field5 = models.IntegerField(null=True)
}}}
`TestModel.objects.count() == 20000`, values for all fields `is None`.

Before:
{{{
In [7]: %timeit for x in TestModel.objects.all(): pass
163 ms ± 1.41 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}

After:
{{{
In [7]: %timeit for x in TestModel.objects.all(): pass
143 ms ± 1.07 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:24>

Django

unread,

Nov 18, 2017, 5:17:30 AM11/18/17

to django-...@googlegroups.com

#28459: Improve performance of QuerySet
-------------------------------------+-------------------------------------
Reporter: Sergey Fedoseev | Owner: nobody

Type: | Status: new
Cleanup/optimization |

Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Tomer Chachamu):

* status: closed => new
* resolution: fixed =>

Comment:

There's a patch so reopening the card.

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:25>

Django

unread,

Dec 23, 2017, 5:29:10 AM12/23/17

to django-...@googlegroups.com

#28459: Improve performance of QuerySet
-------------------------------------+-------------------------------------
Reporter: Sergey Fedoseev | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev):

[https://github.com/django/django/pull/9486 PR]

Before:
{{{
In [2]: %timeit for x in DecimalModel.objects.values_list('d'): pass
128 ms ± 531 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [3]: %timeit for x in DecimalModel.objects.values_list(models.F('d') +
1): pass
39.5 ms ± 542 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}

After:
{{{
In [2]: %timeit for x in DecimalModel.objects.values_list('d'): pass
31.8 ms ± 490 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [3]: %timeit for x in DecimalModel.objects.values_list(models.F('d') +
1): pass
20.3 ms ± 579 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:26>

Django

unread,

Dec 26, 2017, 6:12:51 PM12/26/17

to django-...@googlegroups.com

#28459: Improve performance of QuerySet
-------------------------------------+-------------------------------------
Reporter: Sergey Fedoseev | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham <timograham@…>):

In [changeset:"d0f569b350bca89eeb186523d8905a6e31b5a947" d0f569b3]:
{{{
#!CommitTicketReference repository=""
revision="d0f569b350bca89eeb186523d8905a6e31b5a947"
Refs #28459 -- Improved performance of loading DecimalField on SQLite.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:27>

Django

unread,

Dec 27, 2017, 2:27:43 AM12/27/17

to django-...@googlegroups.com

#28459: Improve performance of QuerySet
-------------------------------------+-------------------------------------
Reporter: Sergey Fedoseev | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev):

[https://github.com/django/django/pull/9494 PR (Improved performance of
loading DurationField on SQLite and MySQL.)]

Before:
{{{
%timeit for x in DurationModel.objects.values_list('d'): pass
168 ms ± 678 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}
After:
{{{
%timeit for x in DurationModel.objects.values_list('d'): pass
23.1 ms ± 243 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:28>

Django

unread,

Dec 27, 2017, 12:23:43 PM12/27/17

to django-...@googlegroups.com

#28459: Improve performance of QuerySet
-------------------------------------+-------------------------------------
Reporter: Sergey Fedoseev | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham <timograham@…>):

In [changeset:"ae1baa7d1d7fcb823e71ce9cb7c17de47ab2ff5e" ae1baa7d]:
{{{
#!CommitTicketReference repository=""
revision="ae1baa7d1d7fcb823e71ce9cb7c17de47ab2ff5e"
Refs #28459 -- Improved performance of loading DurationField on SQLite and
MySQL.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:29>

Django

unread,

Dec 29, 2017, 4:25:15 AM12/29/17

to django-...@googlegroups.com

#28459: Improve performance of QuerySet
-------------------------------------+-------------------------------------
Reporter: Sergey Fedoseev | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev):

[https://github.com/django/django/pull/9509 PR (Used default date
converter on SQLite for better performance)]

Before:
{{{
%timeit for x in TestModel.objects.values_list('date'): pass
83.3 ms ± 829 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}
After:
{{{
%timeit for x in TestModel.objects.values_list('date'): pass
40.7 ms ± 613 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:30>

Django

unread,

Dec 29, 2017, 10:12:25 AM12/29/17

to django-...@googlegroups.com

#28459: Improve performance of QuerySet
-------------------------------------+-------------------------------------
Reporter: Sergey Fedoseev | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev <fedoseev.sergey@…>):

In [changeset:"51ae4e1f324afcf1c7f6a102335dc78bfa51c126" 51ae4e1f]:
{{{
#!CommitTicketReference repository=""
revision="51ae4e1f324afcf1c7f6a102335dc78bfa51c126"
Refs #28459 -- Used default date converter on SQLite for better
performance.

See https://docs.python.org/3/library/sqlite3.html#default-adapters-and-
converters.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:31>

Django

unread,

Jan 4, 2018, 4:07:52 PM1/4/18

to django-...@googlegroups.com

#28459: Improve performance of QuerySet
-------------------------------------+-------------------------------------
Reporter: Sergey Fedoseev | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev):

[https://github.com/django/django/pull/9542 PR (Improved performance of
duration expressions on SQLite)]

Before:
{{{
%timeit for x in

Date.objects.annotate(d=models.ExpressionWrapper(models.F('date') +
models.F('duration'), models.DateTimeField())).values_list('d'): pass
591 ms ± 21.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

}}}
After:
{{{
%timeit for x in

Date.objects.annotate(d=models.ExpressionWrapper(models.F('date') +
models.F('duration'), models.DateTimeField())).values_list('d'): pass
390 ms ± 2.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:32>

Django

unread,

Jan 4, 2018, 5:16:31 PM1/4/18

to django-...@googlegroups.com

#28459: Improve performance of QuerySet
-------------------------------------+-------------------------------------
Reporter: Sergey Fedoseev | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham <timograham@…>):

In [changeset:"2115be616b10f4542df7155e3a4c3f02f97cc2a3" 2115be6]:
{{{
#!CommitTicketReference repository=""
revision="2115be616b10f4542df7155e3a4c3f02f97cc2a3"
Refs #28459 -- Improved performance of duration expressions on SQLite.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:33>

Django

unread,

Mar 1, 2018, 4:49:12 AM3/1/18

to django-...@googlegroups.com

#28459: Improve performance of QuerySet
-------------------------------------+-------------------------------------
Reporter: Sergey Fedoseev | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev):

[https://github.com/django/django/pull/9739 PR (Improved performance of
time difference expressions on MySQL)]

Before:
{{{
%timeit
bool(TestModel.objects.values_list(models.ExpressionWrapper(models.F('time')
- models.F('time'), output_field=models.DurationField())))
35.4 ms ± 901 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}

After:
{{{
%timeit
bool(TestModel.objects.values_list(models.ExpressionWrapper(models.F('time')
- models.F('time'), output_field=models.DurationField())))
30 ms ± 810 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:34>

Django

unread,

Mar 1, 2018, 10:24:20 AM3/1/18

to django-...@googlegroups.com

#28459: Improve performance of QuerySet
-------------------------------------+-------------------------------------
Reporter: Sergey Fedoseev | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sergey Fedoseev <fedoseev.sergey@…>):

In [changeset:"ba4a9862407c8e8574c60b9f911f6b09aff1fa12" ba4a9862]:
{{{
#!CommitTicketReference repository=""
revision="ba4a9862407c8e8574c60b9f911f6b09aff1fa12"
Refs #28459 -- Improved performance of time difference expressions on
MySQL.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:35>

Django

unread,

Mar 15, 2018, 9:07:30 PM3/15/18

to django-...@googlegroups.com

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Tim Graham):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:36>

Django

unread,

Aug 23, 2022, 11:55:50 PM8/23/22

to django-...@googlegroups.com

#28459: Improve performance of QuerySet
-------------------------------------+-------------------------------------
Reporter: Sergey Fedoseev | Owner: nobody

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Simon Charette):

* status: new => closed
* resolution: => fixed

Comment:

I think we can close this one as fixed now. There is nothing actionable
here and separate tickets can be created if further performance
improvement opportunities are identified.

--
Ticket URL: <https://code.djangoproject.com/ticket/28459#comment:37>

Reply all

Reply to author

Forward