[Django] #29599: chunk_size for InMemoryUploadedFile is not used
Django
------------------------------------------------+------------------------
Reporter: Ali Aliyev | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: File uploads/storage | Version: master
Severity: Normal | Keywords: chunks
Triage Stage: Unreviewed | Has patch: 1
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------------------+------------------------
Hello,
is it okay that the `chunk_size` is not used for `InMemoryUploadedFile.
chunks`?
Examples to compare:
https://github.com/django/django/blob/master/django/core/files/uploadedfile.py#L92
https://github.com/django/django/blob/master/django/core/files/base.py#L48
--
Ticket URL: <https://code.djangoproject.com/ticket/29599>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Ali Aliyev | Owner: nobody
Cleanup/optimization |
Component: File | Version: master
uploads/storage |
Severity: Normal | Resolution: invalid
Keywords: chunks | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* status: new => closed
* resolution: => invalid
Comment:
> is it okay...
Yes.
In general, why do you handle a file in chunks? So that you can control
how much is in memory at one time.
With `InMemoryUploadedFile` you already decided you'd handle the whole
thing in memory at once.
So the idea of handling it in `chunks` doesn't really make sense.
(See
[https://github.com/django/django/blob/1c05fe65f280cedaddf5e5f308e5e45449b02e93/django/core/files/uploadedfile.py#L95
the comment a few lines below].)
--
Ticket URL: <https://code.djangoproject.com/ticket/29599#comment:1>
Django
Reporter: Ali Aliyev | Owner: nobody
Cleanup/optimization |
Component: File | Version: master
uploads/storage |
Severity: Normal | Resolution: invalid
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Ali Aliyev):
Thanks for the quick response!
I have my own file storage class where `_save` method is called:
https://github.com/django/django/blob/master/django/core/files/storage.py#L49
The problem is that I have to upload chunks (each chunks should not be
more than 4mb) but `content` is instance of the `InMemoryUploadedFile` in
which case I had to implement my own `chunks` method:
{{{
class AzureStorage(Storage):
...
def _read_in_chunks(self, file_object, chunk_size=1024):
while True:
data = file_object.read(chunk_size)
if not data:
break
yield data
def _save(self, name, content):
if hasattr(content.file, 'content_type'):
content_type = content.file.content_type
else:
content_type = mimetypes.guess_type(name)[0]
chunks = self._read_in_chunks(
content,
settings.AZURE_CHUNK_SIZE * 1024 * 1024
)
blocks_list = []
for chunk in chunks:
block_id = uuid.uuid4()
self.connection.put_block(self.azure_container, name, chunk,
block_id)
blocks_list.append(str(block_id))
self.connection.put_block_list(
self.azure_container,
name,
blocks_list,
x_ms_blob_content_type=content_type
)
return name
}}}
so `content.cunks` will not work here
Replying to [comment:1 Carlton Gibson]:
> > is it okay...
>
> Yes.
>
> In general, why do you handle a file in chunks? So that you can control
how much is in memory at one time.
>
> With `InMemoryUploadedFile` you already decided you'd handle the whole
thing in memory at once.
> So the idea of handling it in `chunks` doesn't really make sense.
>
> (See
[https://github.com/django/django/blob/1c05fe65f280cedaddf5e5f308e5e45449b02e93/django/core/files/uploadedfile.py#L95
the comment a few lines below].)
--
Ticket URL: <https://code.djangoproject.com/ticket/29599#comment:2>