[Django] #34378: Using in_bulk() with id_list and order

Django

unread,

Mar 2, 2023, 1:44:13 AM3/2/23

to django-...@googlegroups.com

#34378: Using in_bulk() with id_list and order_by()
-------------------------------------+-------------------------------------
Reporter: Ekaterina | Owner: nobody
Vakhrusheva |
Type: Bug | Status: new
Component: Database | Version: 4.1
layer (models, ORM) |
Severity: Normal | Keywords: ib_bulk
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
If you call in_bulk() together with the order_by(“field_1”,
“field_2”).distinct(“field_1”) and specifying field_name=“field_1”, then
when the result is received, the sorting order of the elements passed to
order_by() is preserved. Code example:

{{{
locations = (
UserLocation.objects.order_by("user_id", "-date_create")
.distinct("user_id")
.in_bulk(field_name="user_id")
)
print(locations)
}}}

**>>>{2693: <UserLocation: 2023-03-01 15:17>}**

However, if you specify an additional id_list argument in the in_bulk
method, the sorting specified earlier in the request disappears, and as a
result, another instance of the model is returned. Code example:

{{{
locations = (
UserLocation.objects.order_by("user_id", "-date_create")
.distinct("user_id")
.in_bulk([2693], field_name="user_id")
)
print(locations)
}}}

**>>>{2693: <UserLocation: 2023-03-01 14:27>}**

A user with id 2693 has several locations that differ in the date_create.
The most recent is with a time of **15:17**. When sorting queryset by
“-date_create” I want to get the most recent record. Therefore, the
sorting is reset when passing the id_list parameter.

In my opinion, the sorting from queryset was removed before it was
possible to pass different field_name to the in_bulk() method. Apparently,
it was not taken into account that when sorting is reset, the result
obtained when using DISTINCT changes.

[https://forum.djangoproject.com/t/using-in-bulk/19192]

--
Ticket URL: <https://code.djangoproject.com/ticket/34378>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,

Mar 2, 2023, 2:03:11 AM3/2/23

to django-...@googlegroups.com

#34378: Using in_bulk() with id_list and order_by()
-------------------------------------+-------------------------------------
Reporter: Ekaterina | Owner: nobody
Vakhrusheva |
Type: Bug | Status: new

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak):

* stage: Unreviewed => Accepted

Comment:

Ordering was removed in #15116, however we shouldn’t clear it anymore
because dictionaries preserve the order, so it's not useless.

--
Ticket URL: <https://code.djangoproject.com/ticket/34378#comment:1>

Django

unread,

Mar 2, 2023, 2:10:26 AM3/2/23

to django-...@googlegroups.com

| Unreviewed

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Ekaterina Vakhrusheva):

* stage: Accepted => Unreviewed

Old description:

New description:

In my opinion, the sorting from queryset was removed before it became

possible to pass different field_name to the in_bulk() method. Apparently,
it was not taken into account that when sorting is reset, the result
obtained when using DISTINCT changes.

[https://forum.djangoproject.com/t/using-in-bulk/19192]

--

--
Ticket URL: <https://code.djangoproject.com/ticket/34378#comment:2>

Django

unread,

Mar 2, 2023, 2:14:05 AM3/2/23

to django-...@googlegroups.com

Comment (by Ekaterina Vakhrusheva):

Replying to [comment:1 Mariusz Felisiak]:

> Ordering was removed in #15116, however we shouldn’t clear it anymore
because dictionaries preserve the order, so it's not useless.

i think, this is not useless, not only because dictionaries preserve the
order, but also because the order of the elements of the queryset affects
the value returned by .distinct()

--
Ticket URL: <https://code.djangoproject.com/ticket/34378#comment:3>

Django

unread,

Mar 2, 2023, 2:25:14 AM3/2/23

to django-...@googlegroups.com

Keywords: ib_bulk | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak):

* stage: Unreviewed => Accepted

Comment:

I think you misunderstood my reply: ''"it's **not** useless"''. I accepted
the ticket.

--
Ticket URL: <https://code.djangoproject.com/ticket/34378#comment:4>

Django

unread,

Mar 2, 2023, 2:31:25 AM3/2/23

to django-...@googlegroups.com

#34378: Using in_bulk() with id_list and order_by()
-------------------------------------+-------------------------------------
Reporter: Ekaterina | Owner: nobody
Vakhrusheva |
Type: Bug | Status: new
Component: Database layer | Version: 4.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: ib_bulk | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mariusz Felisiak):

Ekaterina, would you like to prepare a patch? (a regression test is
required.)

--
Ticket URL: <https://code.djangoproject.com/ticket/34378#comment:5>

Django

unread,

Mar 2, 2023, 3:47:23 AM3/2/23

to django-...@googlegroups.com

#34378: Using in_bulk() with id_list and order_by()
-------------------------------------+-------------------------------------
Reporter: Ekaterina | Owner: nobody
Vakhrusheva |
Type: Bug | Status: new
Component: Database layer | Version: 4.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: ib_bulk | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Ekaterina Vakhrusheva):

Yes, it would be great. But I’ve never done patches before. Is there
anything I need to know about the patch creation process other than
[https://docs.djangoproject.com/en/dev/internals/contributing/writing-code
/submitting-patches/ this guide]

--
Ticket URL: <https://code.djangoproject.com/ticket/34378#comment:6>

Django

unread,

Mar 2, 2023, 3:47:44 AM3/2/23

to django-...@googlegroups.com

#34378: Using in_bulk() with id_list and order_by()
-------------------------------------+-------------------------------------

Reporter: Ekaterina | Owner: Ekaterina
Vakhrusheva | Vakhrusheva
Type: Bug | Status: assigned

Changes (by Ekaterina Vakhrusheva):

* owner: nobody => Ekaterina Vakhrusheva
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/34378#comment:7>

Django

unread,

Mar 2, 2023, 3:56:33 AM3/2/23

to django-...@googlegroups.com

#34378: Using in_bulk() with id_list and order_by()
-------------------------------------+-------------------------------------
Reporter: Ekaterina | Owner: Ekaterina
Vakhrusheva | Vakhrusheva
Type: Bug | Status: assigned
Component: Database layer | Version: 4.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: ib_bulk | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mariusz Felisiak):

[https://docs.djangoproject.com/en/dev/internals/contributing/writing-code
/coding-style/ "Coding style"] can also be helpful.

This is a bugfixes so docs changes are not necessary.
We need a fix (it should be enough to remove `order_by()` from the
`in_bulk()`
[https://github.com/django/django/blob/9953c804a9375956a542da94665662d306dff48d/django/db/models/query.py#L1107-L1113
implementation]) and regression tests in `tests.lookup.tests.LookupTests`
(please be sure that you covered both branches, with and without
`batch_size`).

--
Ticket URL: <https://code.djangoproject.com/ticket/34378#comment:8>

Django

unread,

Mar 2, 2023, 2:45:44 PM3/2/23

to django-...@googlegroups.com

#34378: Using in_bulk() with id_list and order_by()
-------------------------------------+-------------------------------------
Reporter: Ekaterina | Owner: Ekaterina
Vakhrusheva | Vakhrusheva
Type: Bug | Status: assigned
Component: Database layer | Version: 4.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: ib_bulk | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Ekaterina Vakhrusheva):

[https://github.com/django/django/pull/16613 PR]

--
Ticket URL: <https://code.djangoproject.com/ticket/34378#comment:9>

Django

unread,

Mar 2, 2023, 2:47:05 PM3/2/23

to django-...@googlegroups.com

#34378: Using in_bulk() with id_list and order_by()
-------------------------------------+-------------------------------------
Reporter: Ekaterina | Owner: Ekaterina
Vakhrusheva | Vakhrusheva
Type: Bug | Status: assigned
Component: Database layer | Version: 4.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: ib_bulk | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Ekaterina Vakhrusheva):

* has_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/34378#comment:10>

Django

unread,

Mar 3, 2023, 12:03:13 AM3/3/23

to django-...@googlegroups.com

#34378: Using in_bulk() with id_list and order_by()
-------------------------------------+-------------------------------------
Reporter: Ekaterina | Owner: Ekaterina
Vakhrusheva | Vakhrusheva
Type: Bug | Status: assigned
Component: Database layer | Version: 4.1
(models, ORM) |
Severity: Normal | Resolution:

Keywords: ib_bulk | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/34378#comment:11>

Django

unread,

Mar 3, 2023, 2:19:11 AM3/3/23

to django-...@googlegroups.com

#34378: Using in_bulk() with id_list and order_by()
-------------------------------------+-------------------------------------
Reporter: Ekaterina | Owner: Ekaterina
Vakhrusheva | Vakhrusheva

Type: Bug | Status: closed

Component: Database layer | Version: 4.1
(models, ORM) |

Severity: Normal | Resolution: fixed

Changes (by Mariusz Felisiak <felisiak.mariusz@…>):

* status: assigned => closed
* resolution: => fixed

Comment:

In [changeset:"55bcbd8d172b689811fae17cde2f09218dd74e9c" 55bcbd8d]:
{{{
#!CommitTicketReference repository=""
revision="55bcbd8d172b689811fae17cde2f09218dd74e9c"
Fixed #34378 -- Made QuerySet.in_bulk() not clear odering when id_list is
passed.

This reverts 340eaded4e30cf25bcd4e9781d33a617fe9c0f84.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/34378#comment:12>

Django

unread,

Jun 9, 2025, 4:40:35 PM6/9/25

to django-...@googlegroups.com

#34378: Using in_bulk() with id_list and order_by()
-------------------------------------+-------------------------------------
Reporter: Ekaterina | Owner: Ekaterina
Vakhrusheva | Vakhrusheva
Type: Bug | Status: closed
Component: Database layer | Version: 4.1
(models, ORM) |
Severity: Normal | Resolution: fixed
Keywords: ib_bulk | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by nessita <124304+nessita@…>):

In [changeset:"a68e8565cdd4fc3f8b738fc516095dab142b9d65" a68e8565]:
{{{#!CommitTicketReference repository=""
revision="a68e8565cdd4fc3f8b738fc516095dab142b9d65"
Refs #34378, #36143, #36416 -- Fixed isolation of
LookupTests.test_in_bulk_preserve_ordering_with_batch_size().

`max_query_params` is a property, so it must be patched on the class.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/34378#comment:13>

Django

unread,

Aug 13, 2025, 2:33:16 PM8/13/25

to django-...@googlegroups.com

#34378: Using in_bulk() with id_list and order_by()
-------------------------------------+-------------------------------------
Reporter: Ekaterina | Owner: Ekaterina
Vakhrusheva | Vakhrusheva
Type: Bug | Status: closed
Component: Database layer | Version: 4.1
(models, ORM) |
Severity: Normal | Resolution: fixed
Keywords: ib_bulk | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Natalia <124304+nessita@…>):

In [changeset:"a4e27c0c6bcc37ad7d7f6098e5d44edbbcca2c42" a4e27c0]:
{{{#!CommitTicketReference repository=""
revision="a4e27c0c6bcc37ad7d7f6098e5d44edbbcca2c42"
[5.2.x] Refs #34378, #36143, #36416 -- Fixed isolation of

LookupTests.test_in_bulk_preserve_ordering_with_batch_size().

`max_query_params` is a property, so it must be patched on the class.

Backport of a68e8565cdd4fc3f8b738fc516095dab142b9d65 from main.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/34378#comment:14>

Django

unread,

Aug 21, 2025, 10:47:52 AM8/21/25

to django-...@googlegroups.com

#34378: Using in_bulk() with id_list and order_by()
-------------------------------------+-------------------------------------
Reporter: Ekaterina | Owner: Ekaterina
Vakhrusheva | Vakhrusheva
Type: Bug | Status: closed
Component: Database layer | Version: 4.1
(models, ORM) |
Severity: Normal | Resolution: fixed
Keywords: ib_bulk | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sarah Boyce <42296566+sarahboyce@…>):

In [changeset:"d3cf24e9b415b41f570c9f426b2cd113b5fdb4de" d3cf24e9]:
{{{#!CommitTicketReference repository=""
revision="d3cf24e9b415b41f570c9f426b2cd113b5fdb4de"
Refs #36430, #36416, #34378 -- Simplified batch size calculation in
QuerySet.in_bulk().
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/34378#comment:15>

Reply all

Reply to author

Forward

[Django] #34378: Using in_bulk() with id_list and order_by()

Django

Django

Django

Django

Django

Django

Django

Django

Django

Django

Django

Django

Django

Django

Django

Django