Serving an S3 file

[Mike Orr]

I'm migrating user-uploaded files from a local filesystem to MinIO (an
S3-compatible file server). I can't use FileResponse because I get
back a filehandle or a bytes, and I'd have to write it to a temporary
file to use FileResponse.I'm also exploring presigned URLs to have the
browser fetch the file, but right now I want to serve it server-side.
What Pyramid response args should I use, and how should I stream the
content? I looked at the source of FileResponse, and it creates an
'app_iter' object using 'environ["wsgi.file_wrapper"]'. Do I have to
do all that? Or can I just do 'body_file=result["body"]', which is the
filehandle and has a 'read' method. What is 'wsgi.file_wrapper'
anyway?

--
Mike Orr <slugg...@gmail.com>

[Mike Orr]

Here's the documentation for the filehandle object:
https://botocore.amazonaws.com/v1/documentation/api/latest/reference/response.html

There's an 'iter_chunks' method that yields fixed-size chunks of the
file. So can I do
'app_iter=result["Body"].iter_chunks()' ? It looks like this is what
FileResponse's
app_iter code is trying to do. The default chunk size is 1KB, while FileResponse
uses 256KB chunks. What size is best for this use case?
--
Mike Orr <slugg...@gmail.com>

[Jonathan Vanasco]

> What is 'wsgi.file_wrapper' anyway?

It's a WSGI hook defined in PEP 333 and 3333 (https://www.python.org/dev/peps/pep-3333/#optional-platform-specific-file-handling)

It basically lets your application defer to an iterator defined in middleware file responses.  In the Pyramid code, you'll see that it checks to see if there is a middleware filewrapper, and falls back to Pyramid's option.

If you have loaded the bytes, you can use a BytesIO object, which is in-memory. 

If the file might be large, you could write the data to tempfile.SpooledTemporaryFile -- which will be in-memory until it hits a max-size, and then writes to disk.