Serving large files, require custom auth

[Kenchu]

I'm using Python on GAE, and I want to serve large files to my users. But only to those who have been authenticated through custom authentication (i.e. not google).

The question is how to do this most efficiently. I recon since the authentication is custom, the request has to go through GAE. I cannot just use a static file somewhere. So GAE needs to read this file, and send it to the user.

The way I've thought of doing it is to upload a file to cloud storage, and then read it through GAE and send it if the user is authenticated. But I'm not sure this would be the best way to go about it. Wouldn't this virtually double the cost? First GAE reads from cloud storage (bandwidth cost), and then send it to user (bandwidth cost).

Another concern I have is how long GAE can maintain a connection. These files are rather big, so sending it could take some time.

I'm at a bit of a loss here since I haven't done this kind of thing before. I'm not sure what the best approach is, or if the one I've suggested is even viable.

Any guidance or tips would be greatly appreciated.

Thanks!

[Wilson MacGyver]

first on cost. GAE access to cloud storage is free. So you aren't doubling up the cost.

we've done something similar. we store the files in google cloud storage as private objects. We then use GAE

to authenticate and deliver the files. I don't know what your idea of large file is, it's been doing ok for us.

another way to solve this problem is bypass GAE all together on delivery. You can use the signed url function in cloud storage

https://cloud.google.com/storage/docs/access-control#Signed-URLs

then you just generate a url in GAE which will expire after certain amount of time. But you deliver the url to the end user. They can download it using their browser within that certain amount of time.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/f18ae707-0dd6-486b-a094-ecac088320d7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Omnem crede diem tibi diluxisse supremum.

[Kenchu]

Thanks Mac!

That url signing seems really useful. Think I'm gonna take that approach.

The files aren't that big. Around 200-300 MB perhaps. But some people have really slow connections so the connection would need to be open. I'm not sure if it applies anymore but I think GAE used to have an execution time limit of 1 minute. But overall, the idea of having a temporary "static" url appeals to me so definitely gonna check that url signing out.

Thanks!

[Kenchu]

Was a bit more cumbersome than I had hoped, but I've managed to get it working.

Some things to point out:

You need to add the libraries pycrypto and ssl To app.yaml.

You need to:

pip install requests==2.3 -t <your app folder>

You also need to configure credentials in your apps dashboard, and add a client id. There's more information about that here:

https://cloud.google.com/storage/docs/authentication#service_accounts

Once you've done these two things, it should work fine to use their samples:

https://cloud.google.com/storage/docs/access-control#signing-code-python

Or well, actually it didn't quite work 100%. But the _MakeUrl method does, and that's all that matters for me.

[Wilson MacGyver]

yea, you need to setup the credentials. We use java, so we didn't run into any python specific issues.

glad to hear it works ok. with file size of 200MB+, this is the best solution I think.