StreamingHttpResponse into a Template

[Stefano Tranquillini]

Django 1.5 is just came out and it ships the StreamingHttpResponse. 

What i want to do is to print the output of a stream response into a template, having for example a bar that grows based on the streamed data (something similar to a loading bar).

so far i've found a solution that basically is to write the pieces of the final html page into the generator function, so each yield does a render_to_text of a piece of the final html page (i splitted it manually) putting each time the data just processed. 

The problem of this solution is that i can't have the loading bar. What I'm able to have are incremental data, so i can print out the list of processed data, but it's different to what i want to do.

Is there a way to do what i would like to do? 

thanks.

[Masklinn]

On 2013-02-27, at 18:23 , Stefano Tranquillini wrote:
> Django 1.5 is just came out and it ships the StreamingHttpResponse.
>
> What i want to do is to print the output of a stream response into a
> template

I'm pretty sure that doesn't make sense: the template has to be
rendered, and I'm pretty sure a given Django template doesn't render to
a stream given https://code.djangoproject.com/ticket/13910 is still
open.

What you *can* do is work the other way around, create a
StreamingHttpResponse and fill it with multiple renderings of a template.

> , having for example a bar that grows based on the streamed data
> (something similar to a loading bar).

If your render target is HTML, that doesn't make any sense either.
That's just not how HTML works. You might be able to send a CSS
stream with incremental rules overwriting one another and growing
a "loading bar", but that's abuse of the whole thing and not of
much use.

> so far i've found a solution that basically is to write the pieces of the
> final html page into the generator function, so each yield does a
> render_to_text of a piece of the final html page (i splitted it manually)
> putting each time the data just processed.
>
> The problem of this solution is that i can't have the loading bar. What I'm
> able to have are incremental data, so i can print out the list of processed
> data, but it's different to what i want to do.
>
> Is there a way to do what i would like to do?

Think about it again and explain what you're originally trying to
achieve. You seem to be approaching the problem from the wrong
direction, as far as I can read.

[Stefano Tranquillini]

Hi, thank you for the answer.

basically what i've to do is to iterate the same operation for a bunch of user (e.g. send an email). 

i've around 200 users and each mail takes 2 seconds. 

so 400 seconds in total and the server goes in timeout. Have to wait 400 seconds on a blank page is not really nice as well.

what i want to do is to stream out to a page an info that a new email was sent. This will avoid the timeout (i guess) and i can keep track of the job. 

--
You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/eQ2SVB_uYuE/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

--
Stefano

[Masklinn]

On 2013-02-27, at 18:47 , Stefano Tranquillini wrote:

> Hi, thank you for the answer.
> basically what i've to do is to iterate the same operation for a bunch of
> user (e.g. send an email).
> i've around 200 users and each mail takes 2 seconds.
> so 400 seconds in total and the server goes in timeout. Have to wait 400
> seconds on a blank page is not really nice as well.
>
> what i want to do is to stream out to a page an info that a new email was
> sent. This will avoid the timeout (i guess) and i can keep track of the
> job.

This kind of stuff is generally done asynchronously, dispatching the
sending to some sort of queue[0] to be executed either through a cronjob
or through a dedicated worker (*not* a WSGI/Django handler) waiting for
any mail to send.

Notifications, if any, are provided separately (e.g. by reloading a
"status" page every 10~15~30s or so; by using a javascript widget
polling the same kind of "status" endpoint or by using more "push"-y
async techniques). If you want notifications you probably want to use
a queue/worker system (rather than cronjobs) so they can get started
immediately and provide feedback. Something like Celery[2] for instance.

Sending email (or any other kind of very long operation) within the HTTP
request is pretty much always A Bad Idea: it ties up resources (e.g.
HTTP/WSGI workers, database transactions) better used serving requests
to users and tends to make the code more gnarly and less maintainable.

As far as I've seen anyway.

[0] For instance Pinax's Django Mailer[1] has an in-request part which
shoves mails to send in the DB, and a management command to execute via
e.g. a cronjob which does the actual sending
[1] https://github.com/pinax/django-mailer
[2] celeryproject.org

[Stefano Tranquillini]

Yes, you are perfectly right about the fact that this is a bad implementation.

the fact is that where i host the code it does not support celery (so far). so i've to find out a workaround to be able to process a bunch of external APIs Call (that include the one for sending email). 

we'll probably keep the javascript reloading to process junks, waiting for the support of celery.

BTW: the StreamingHttpResposnse, when should it be used? i get the example on the docs about writing a cvs, pdf or similar, but, what else?

Thanks for the help.

[Sam Solomon]

Not sure if this is what you meant by "we'll probably keep the javascript reloading to process junks, waiting for the support of celery", but one option without celery is to have the initial request save a bunch of "PendingEmail" objects in the database and then redirect you to a "send_queued_emails" page that sends as many emails as it can in 5 seconds and then returns a page telling you how many it needs to send remaining and javascript to refresh the page immediately (if you use browsers that will show a blank page as soon as the refresh is initiated, you can use ajax or something for these requests).

Something somewhat closer to the celery option, but possible to set up on slightly more restrictive systems is to do the same thing with "PendingEmail" objects and set up a cron job to run a django management command every minute that consumes as many PendingEmail objects it can in 55 seconds.

For either of these solutions, make sure autocommit is enabled for those functions and that you mark an email as sent before you start the api call otherwise if somehow two things are consuming from the same queue you could send duplicate emails. Alternatively (or additionally) you could even create a lock-file or some similar thing to make sure only one thing is consuming from the queue at a time.