Re: [Django] #12157: FileSystemStorage does file I/O inefficiently, despite providing options to permit larger blocksizes

[Django]

#12157: FileSystemStorage does file I/O inefficiently, despite providing options to
permit larger blocksizes
-------------------------------------+-------------------------------------
Reporter: alecmuffett | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: File | Version: 1.1
uploads/storage |
Severity: Normal | Resolution:
Keywords: io, | Triage Stage: Accepted
FileSystemStorage, buffering, |
performance |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mike Dearman):

Obviously this is an old ticket, but I was looking over the existing notes
and codebase to see what was possible:

Part of the complexity to implement this ticket properly seems to revolve
around the lack of parameter (or setting?) to pass a buffering policy from
the Storage.open API to the internal Storage._open method. The developer
also often doesn't call Storage.open directly, and utilizes the lazy open
pattern from FieldFile, etc. If we are trying to use the File.chunks
method, which has a chunk_size parameter (or default setting), it seems
too late to be able to adjust/affect the buffering policy (as the file was
already opened and returned in the File instance).

If we tried to just set the buffering parameter inside
FileSystemStorage._open as previously suggested, there's also the
challenge of (possibly) adjusting the buffering depending on the file mode
used. Based on https://docs.python.org/3/library/functions.html#open ,
open's `buffering` policy/heuristic has more functionality depending on
binary or text mode, that a fixed (say 64k) buffer doesn't account for.

If a model FieldFile is used to access the underlying storage file, it
often implicitly calls Storage.open. However, developers can call
FieldFile.open explicitly too. To provide buffering there, we'd have to
expand the Storage.open API and File.open API definitions too I'd think?
Not sure the appetite for this kind of change though.

Anyway, just thinking out loud and maybe it'll spark new thoughts, or at
least recap the current state.
--
Ticket URL: <https://code.djangoproject.com/ticket/12157#comment:7>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.