|Block On File Upload||Tobes||1/13/11 12:28 PM|
Sorry, this is a lazy question...
If we have 1 dyno, and a user uploads a file that takes 10 minutes,
will the other requests to the app be blocked for 10 minutes?
Looking at our logs it looks like that *is* the case. We do store
files on S3 btw, but our Rails app receives the file and moves it to
S3. It's the blocking we're interested in knowing about (I know Rails
doesn't block usually)
|Re: Block On File Upload||Jimmy Thrasher||1/13/11 12:33 PM|
Unless they've removed the 30s hard limit, you should see those requests dying after 30s, but yes, in theory if it takes 10 minutes it blocks your upload.
An approach I like but have never had the need to implement would be to use the "upload by POST" technique, described here: http://aws.amazon.com/articles/1434?_encoding=UTF8&jiveRedirect=1
You basically create a signed URL with an expiration date which allows a user to upload to a bucket you have rights to. There are third party utilities, often written in Flash, to help you accomplish this.
Hope that helps,
|Re: Block On File Upload||Keenan Brock||1/13/11 5:27 PM|
|Re: Block On File Upload||John Maxwell||1/15/11 11:54 AM|
So an upload to your app on heroku blocks the whole dyno whilst it is uploading? Surely that isn't right - as the dyno shouldn't even be aware of the post request until the web server itself has received the whole file? In that case its just blocked whilst the app processes it?
|Re: Block On File Upload||Jimmy||1/15/11 4:44 PM|
I think it's more precise to say that a dyno is a thin+rack combo, so the dyno *is* the web server receiving the request.
|Re: Block On File Upload||John Maxwell||1/15/11 4:47 PM|
So the dyno then reroutes requests back to Nginx? really?
|Re: Block On File Upload||Jimmy||1/16/11 3:57 AM|
I wouldn't call it rerouting, but it is likely a response to a TCP transaction. Nginx is just a reverse proxy routing requests to the dynos in the first place. This is what large scale deployments do anyway, so it's nothing new.
By the way, this is nothing that isn't written on the Heroku docs.. They explain it much better than I could.
Hope that helps,
|Re: Block On File Upload||John Maxwell||1/16/11 8:15 AM|
Having searched the docs for confirmation on this, there isn't anything which explicitly says what the situation is.
However, unless I'm being a plonker, which is possible - then nginx only reroutes the request to the dyno once the request is complete - ie the file has been received. Otherwise you'd be streaming data into the dyno, which is not how file uploads work! The dyno would obviously be blocked whilst processing the upload, and transferring to S3 unless you do that in the background, but actually uploading itself shouldn't block it, surely?
|Re: Block On File Upload||John Beynon||1/16/11 10:30 AM|
there's a bit at the bottom of this page http://docs.heroku.com/s3 that may give an additional clue
"This is the preferred approach if you’re working with file uploads bigger than 4mbs. The idea is to skip the hop on your application dyno, making a connection from the end user browser straight to S3."
|Re: Block On File Upload||John Maxwell||1/16/11 12:33 PM|
Yeah - that is talking about the outgoing hop to S3 from Heroku - which is a blocking I/O, not the incoming step into Nginx... Any word on this from an official source?
|Re: Block On File Upload||Oren Teich||1/16/11 3:18 PM|
A dyno is the entire stack. A request, the second it hits Heroku,
blocks a dyno. This could be because your application is processing,
or because you are handling an upload.
If you are doing decent size upload, you should have your app upload
|Re: Block On File Upload||John Maxwell||1/16/11 4:06 PM|
Right, that's more of a limitation that I thought apologies for being wrong (and sure I wasn't!) - I'm not massively familiar with Rack, but could you use a Rack Middleware to detect a file upload and spin up another dyno whilst it is in progress? Or is that not detectable within Rack?
|Re: Block On File Upload||Josal||1/17/11 12:38 AM|
In my app, when the server was uploading a file to S3 (fired from a first request), we got a request timeout error from heroku when the app received a second request. The solution was to use DJ in order to not block the dyno, and manage the upload from the job in the worker in background, and updating the upload status from the front-end querying the DB whenever you consider. We solved the problem this way.
Hope it helps.
|Re: Block On File Upload||Christos Zisopoulos||1/17/11 11:55 PM|
A cautionary tale:
We recently attempted a migration from Paperclip to Carrierwave (0.4.10). Despite working fine locally, we hit a snag as soon as we went live with it in production:
Carrierwave would block for inordinate amounts of time. Never found out why, it just did.
Heroku would timeout the connection after 30 seconds, but the dyno handling it would get stuck for up to half an hour. Rinse and repeat and a large amount of our dynos would end up blocked.
The Heroku router (HAProxy, I am guessing) doesn't seem to keep tabs on the blocked dynos, so it would keep proxying requests to them, effectively timing out and giving the dreaded H12 error in the logs.
Even though we had a few dynos still available, most requests would hit the blocked dynos and timeout.
The situation would clear up after 30 mins or so, when their reaper process would kill the dynos (seems like they are using Libratos' Silverline). Or we would have to manually restart the server ourselves with a `heroku restart`
Moral of the story: File upload is a delicate operation, even more so in Heroku. Do your best to monitor your service and understand how the underlying stack works. When possible upload straight to S3.