Serving large files through HttpResponse

[Ceph]

I'm serving rather large, static files through Django. I'd rather not
use renaming trickery to serve them, but I don't know what the best
way to serve them through HttpResponse is. I used to use this:

response = HttpResponse(iter(open(file_path, "rb")))
return response

in my view. That stopped working in the previous week or two. I'm not
sure why, exactly. Is this method no longer supported? I am not even
sure if it is doing what I hoped it was doing. =)

The only other option I see to serving large files securely is
trickery using dynamically created sym links to the true static file
and then redirecting the user to those URLs and letting Apache serve
them. This isn't as secure, though, and permits multiple downloads
without them being recorded in my Django apps.

The files range in size from a few hundred KB up through about 50 MB,
mostly hefty print-ready PDFs.

Any suggestions?

[Simon Willison]

On Aug 20, 1:12 pm, Ceph <ceph...@gmail.com> wrote:
> The only other option I see to serving large files securely is
> trickery using dynamically created sym links to the true static file
> and then redirecting the user to those URLs and letting Apache serve
> them. This isn't as secure, though, and permits multiple downloads
> without them being recorded in my Django apps.

Lighttpd and nginx both have a clever way of dealing with this
problem. You can store the real file somewhere that is readable by the
web server but NOT exposed as a public URL. Then your Django
application can (having authenticated the request) respond with a
magical header which tells the nginx/lighttpd server to serve up the
file from that location.

Here's how to do it with nginx:

http://wiki.codemongers.com/NginxXSendfile

If your app is all about serving large files, it may well be worth
putting it behind an nginx proxy purely to gain access to this
feature. nginx is a really neat reverse proxy - I run
simonwillison.net as nginx proxying through to mod_python/Apache for
the dynamic pages and it's been working great for months.

Hope that helps,

Simon

[Graham Dumpleton]

If using mod_python, you might also look at Tramline:

http://infrae.com/products/tramline

This allows a response to set a special header which is then picked up
by an Apache output filter, with the output filter then inserting the
file contents direct into the output stream.

Using Python to do this sort of Apache filter will still incur some
overhead, but better than sending the file yourself. I imagine there
must be a C module for Apache for doing this sort of stuff as well
which would make it even more efficient.

Graham

> simonwillison.net as nginx proxying through tomod_python/Apache for