[Django] #26400: QuerySet bulk_create method to handle generators to prevent loading all objects in memory at once
Django
objects in memory at once
----------------------------------------------+-------------------------
Reporter: likeon | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Database layer (models, ORM) | Version: master
Severity: Normal | Keywords: bulk_create
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------------------+-------------------------
In my case I need to create huge amount of objects using bulk create
method with batch_size parameter.
The problem is I don't have enough memory to store all objects in a list.
Even if I transmit generator it would be converted to list anyway
(https://github.com/django/django/blob/1.9.4/django/db/models/query.py#L438).
I want to implement a feature to handle generators properly without
loading all objects in memory, but bulk_create method returns list of
objects as a result. That is unacceptable on large amounts of data.
How can I properly implement the method: create a new one or add a
parameter to actual method?
--
Ticket URL: <https://code.djangoproject.com/ticket/26400>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
objects in memory at once
Reporter: likeon | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: bulk_create | Triage Stage:
| Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by charettes):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
I'm not sure this is worth including into Django, is there a reason you
can't split your `bulk_create` calls into batches that fit into memory?
{{{#!python
from itertools import islice, chain
# Taken from https://code.activestate.com/recipes/303279-getting-items-in-
batches/
def split(iterable, size):
sourceiter = iter(iterable)
while True:
batchiter = islice(sourceiter, size)
yield chain([batchiter.next()], batchiter)
for batch in split(large_generator, 10000): # Adjust size to fit in
memory
MyModel.objects.bulk_create(batch)
}}}
You could even make this a manager method if required.
--
Ticket URL: <https://code.djangoproject.com/ticket/26400#comment:1>
Django
objects in memory at once
Reporter: likeon | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
| Unreviewed
Easy pickings: 0 | UI/UX: 0
Comment (by likeon):
It's not the first time I need that. I mean sure I can write that split
thing one more time as I did in other projects, but I don't see any reason
why the feature couldn't be a part of Django.
--
Ticket URL: <https://code.djangoproject.com/ticket/26400#comment:2>
Django
objects in memory at once
Reporter: likeon | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
| Unreviewed
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
My first inclination was the same as Simon's but if you want to show what
the changes to `bulk_create()` (or a new method) would look like, we can
run it by the DevelopersMailingList to get some other opinions.
--
Ticket URL: <https://code.djangoproject.com/ticket/26400#comment:3>
Django
objects in memory at once
Reporter: likeon | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
| Unreviewed
Easy pickings: 0 | UI/UX: 0
Comment (by charettes):
Replying to [comment:2 likeon]:
> It's not the first time I need that opportunity. I mean sure I can write
that split thing one more time as I did in other projects, but I don't see
any reason why the feature couldn't be a part of Django.
We'd have to alter both the signature and the return type of `bulk_create`
in order to pass a flag enabling this feature and make sure not to return
a list of the created objects. At this point I think this should be
handled by another method/function.
I personally don't believe this use case is common enough to warrant an
inclusion in Django but as Tim pointed out you could try leveraging
support from the community on the developer mailing list.
--
Ticket URL: <https://code.djangoproject.com/ticket/26400#comment:4>
Django
objects in memory at once
Reporter: likeon | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: bulk_create | Triage Stage:
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* stage: Unreviewed => Someday/Maybe
--
Ticket URL: <https://code.djangoproject.com/ticket/26400#comment:5>
Django
objects in memory at once
Reporter: Alexander Sterchov | Owner: nobody
Type: | Status: closed
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Keywords: bulk_create | Triage Stage:
| Someday/Maybe
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* status: new => closed
* resolution: => wontfix
Comment:
Closing in absence of follow up or discussion.
--
Ticket URL: <https://code.djangoproject.com/ticket/26400#comment:6>
Django
objects in memory at once
Reporter: Alexander Sterchov | Owner: nobody
Type: | Status: closed
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution: wontfix
Keywords: bulk_create | Triage Stage:
| Someday/Maybe
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
#28231 is a follow up ticket requesting similar behavior. The current
consensus seems to be to document the behavior rather than to change it.
--
Ticket URL: <https://code.djangoproject.com/ticket/26400#comment:7>