[Django] #21179: How-to output CSV from Django should suggest using `StreamingHttpResponse`
Django
------------------------------------------------+------------------------
Reporter: charettes | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 1 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 1
UI/UX: 0 |
------------------------------------------------+------------------------
The [https://docs.djangoproject.com/en/dev/howto/outputting-csv/
Outputting CSV with Django] how-to doesn't even mention
`StreamingHttpResponse` even if [https://docs.djangoproject.com/en/1.5/ref
/request-response/#django.http.StreamingHttpResponse it’s useful for
generating large CSV files.]
I suggest we replace the example with something along the following:
{{{#!python
import csv
from StringIO import StringIO
from django.http import StreamingHttpResponse
def some_view(request):
rows = (
['First row', 'Foo', 'Bar', 'Baz'],
['Second row', 'A', 'B', 'C', '"Testing"', "Here's a quote"]
)
# Define a generator to stream data directly to the client
def stream():
buffer_ = StringIO()
writer = csv.writer(buffer_)
for row in rows:
writer.writerow(row)
buffer_.seek(0)
data = buffer_.read()
buffer_.seek(0)
buffer_.truncate()
yield data
# Create the streaming response object with the appropriate CSV
header.
response = StreamingHttpResponse(stream(), content_type='text/csv')
response['Content-Disposition'] = 'attachment;
filename="somefilename.csv"'
return response
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/21179>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: charettes | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 1
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Changes (by EvilDMP):
* stage: Unreviewed => Accepted
Comment:
Yes, and also https://docs.djangoproject.com/en/dev/ref/request-
response/#django.http.StreamingHttpResponse should link to this, in the
text "For instance, it’s useful for generating large CSV files"
--
Ticket URL: <https://code.djangoproject.com/ticket/21179#comment:1>
Django
Reporter: charettes | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: master
Keywords: afraid-to-commit | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 1
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Changes (by EvilDMP):
* keywords: => afraid-to-commit
--
Ticket URL: <https://code.djangoproject.com/ticket/21179#comment:2>
Django
Reporter: charettes | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: master
Keywords: afraid-to-commit | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Comment (by mjtamlyn):
I'm not convinced. I've output many a CSV file and never needed the
streaming response to get performance. Whilst this is a useful addition to
mention at this point in the docs, I don't think we should be recommending
the more complex option.
--
Ticket URL: <https://code.djangoproject.com/ticket/21179#comment:3>
Django
Reporter: charettes | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: master
Keywords: afraid-to-commit | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Comment (by aaugustin):
The code example looks like C, not like Python... I don't want to see
`buffer_.seek(0)` in our docs.
Streaming responses don't change much when you pull all the data in RAM,
and if the data comes from a queryset, Django currently does that even if
you use `.iterator()`. It seems much more interesting to me to optimize
the database side than the HTTP response side.
--
Ticket URL: <https://code.djangoproject.com/ticket/21179#comment:4>
Django
Reporter: charettes | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: master
Keywords: afraid-to-commit | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Comment (by charettes):
Thinking about it I must agree that without server-side cursor support
(#16614) the tradeoff is not worth turning the simple example into a
overly complex one.
I just thought it was odd that `StreamingHttpResponse`'s documentation
mentions that ''it’s useful for generating large CSV files'' but
[https://docs.djangoproject.com/en/dev/howto/outputting-csv/ our provided
tutorial] doesn't even mention it.
What do you guys think of adding an admonition with no specific example to
the ''how-to'' explaining `StreamingHttpResponse` might be useful in this
case?
--
Ticket URL: <https://code.djangoproject.com/ticket/21179#comment:5>
Django
Reporter: charettes | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: master
Keywords: afraid-to-commit | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Comment (by EvilDMP):
`StreamingHttpResponse` could still do with some example code in the docs,
even if it doesn't replace the existing example.
--
Ticket URL: <https://code.djangoproject.com/ticket/21179#comment:6>
Django
Reporter: charettes | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: master
Keywords: afraid-to-commit | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Comment (by anubhav9042):
Any ideas regarding what type of example should be given in the docs for
StreamingHttpResponse?
--
Ticket URL: <https://code.djangoproject.com/ticket/21179#comment:7>
Django
Reporter: charettes | Owner: zr
Type: Cleanup/optimization | Status: assigned
Component: Documentation | Version: master
Severity: Normal | Resolution:
Keywords: afraid-to-commit | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Changes (by zr):
* status: new => assigned
* owner: nobody => zr
--
Ticket URL: <https://code.djangoproject.com/ticket/21179#comment:8>
Django
Reporter: charettes | Owner:
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: master
Keywords: afraid-to-commit | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Changes (by zr):
* owner: zr =>
* status: assigned => new
Comment:
Hello, I would like to work on this ticket.
I think that some information on how to test a view that returns a
StreamingHttpResponse() would be useful. The Django test Client actually
returns an iterable response, and the `.streaming_content` property is an
instance of <itertools.imap>. You would then need to concatenate it into a
string in order to test it, as you would do with the standard
HttpResponse.
--
Ticket URL: <https://code.djangoproject.com/ticket/21179#comment:9>
Django
Reporter: charettes | Owner:
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: master
Keywords: afraid-to-commit | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Comment (by zr):
I was thinking of something along these lines:
{{{
#!python
import csv
from django.http import StreamingHttpResponse
class Echo(object):
def write(self, value):
return value
def some_streaming_view(request):
rows = (["Row {0}".format(idx), str(idx)] for idx in xrange(100))
buffer_ = Echo()
writer = csv.writer(buffer_)
response = StreamingHttpResponse((writer.writerow(row) for row in
rows),
content_type="text/csv")
response['Content-Disposition'] = 'attachment;
filename="somefilename.csv"'
return response
}}}
I have tested it with curl, a simple test case with the Django test
client, and a regular browser.
--
Ticket URL: <https://code.djangoproject.com/ticket/21179#comment:10>
Django
Reporter: charettes | Owner: zr
Type: Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: afraid-to-commit | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Changes (by zr):
* status: new => assigned
* owner: => zr
--
Ticket URL: <https://code.djangoproject.com/ticket/21179#comment:11>
Django
Reporter: charettes | Owner: zr
Type: Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: afraid-to-commit | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Comment (by zr):
You can also test this with an infinite series, such as the classic
Fibonacci function, if you replace the range generator with something
like:
I tested this and the memory use did not increase significantly even after
streaming over a gigabyte of data for a single request.
--
Ticket URL: <https://code.djangoproject.com/ticket/21179#comment:12>
Django
Reporter: charettes | Owner: zr
Type: Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: afraid-to-commit | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Comment (by EvilDMP):
The example above looks good to me. Please do submit a pull request -
thanks.
--
Ticket URL: <https://code.djangoproject.com/ticket/21179#comment:13>
Django
Reporter: charettes | Owner: zr
Type: Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: afraid-to-commit | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Changes (by zr):
* has_patch: 0 => 1
Comment:
I have opened a pull request here:
https://github.com/django/django/pull/2358
I am using a slight variation of the above example, using Python 3
friendly code and some additional comments, as suggested by `bmispelon`.
--
Ticket URL: <https://code.djangoproject.com/ticket/21179#comment:14>
Django
Reporter: charettes | Owner: zr
Type: Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: afraid-to-commit | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Comment (by zr):
Resubmitted a new pull request: https://github.com/django/django/pull/2397
--
Ticket URL: <https://code.djangoproject.com/ticket/21179#comment:15>
Django
Reporter: charettes | Owner: zr
Type: Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: afraid-to-commit | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Changes (by timo):
* needs_docs: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/21179#comment:16>
Django
Reporter: charettes | Owner: zr
Component: Documentation | Version: master
Severity: Normal | Resolution: fixed
Keywords: afraid-to-commit | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 1 | UI/UX: 0
Changes (by Tim Graham <timograham@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"fad47367bf622635b4cf931db72310cce41cebb4"]:
{{{
#!CommitTicketReference repository=""
revision="fad47367bf622635b4cf931db72310cce41cebb4"
Fixed #21179 -- Added a StreamingHttpResponse example for CSV files.
Thanks charettes for the suggestion.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/21179#comment:17>