HowTo Upload Large Files to S3 on Heroku

[Lee Longmore]

Hello,

I'm trying to identify the best solution to uploading large files to
S3 via my Heroku-based app.

I am already using carrierwave and fog in my app to upload avatars.
But I am unable to load larger files (e.g. 2MB) as Heroku is timing
out consistently (> 30 secs).

As I am using delayed_job elsewhere in my app, I have tried using an
approach featured on the wiki:

http://www.freezzo.com/2011/01/06/how-to-use-delayed-job-to-handle-your-carrierwave-processing/

But this isn't solving the problem. Background jobs are being created
but I suspect that carrierwave is still trying to store/upload the
file beforehand.

Can anyone offer any advice?

Thanks
Lee.

[Joey]

We ran into similar difficulties uploading and processing large images
(>5 MB). There are actually a few Heroku limitations that might be
impacting you:
- The web front-end times out requests that don't start getting a
response within 30 seconds.
- The app process is killed if it exceeds the 500-600 MB memory
limit. It may seem like plenty, but manipulations on some images will
temporarily require more space. Check your log after a failure to see
if this is your problem.
- There's a max request size of 30 MB imposed by their web front-end
("413 Request Entity Too Large").

Even if you do image-processing in delayed_job tasks, the memory limit
can get you. Things to try:
- The carrierwave_backgrounder gem did not work for us. This blog
post describes our initial approach for delaying image-processing:
http://code.dblock.org/carrierwave-delayjob-processing-of-selected-versions
- The carrierwave_direct gem allows you to upload directly to S3.
This might be just what you need if you're running into the 30 second
request limit. (But uploading to a different host might limit your
ability to do fancy--multiple concurrent, or drag-and-drop-style--
uploading.)
- We ended up queueing all image-processing with delayed_job and
spinning up a dedicated, non-heroku server to do the processing jobs
(avoiding both the memory and same-origin limitations).

Hope that helps,
Joey

On Jan 14, 6:49 am, Lee Longmore <lee.longm...@gmail.com> wrote:
> Hello,
>
> I'm trying to identify the best solution to uploading large files to
> S3 via my Heroku-based app.
>
> I am already using carrierwave and fog in my app to upload avatars.
> But I am unable to load larger files (e.g. 2MB) as Heroku is timing
> out consistently (> 30 secs).
>
> As I am using delayed_job elsewhere in my app, I have tried using an
> approach featured on the wiki:
>

> http://www.freezzo.com/2011/01/06/how-to-use-delayed-job-to-handle-yo...

[Lee Longmore]

Thanks, Joey. Most appreciated.

My requirements do not involve any file processing fortunately - I'm
simply allowing users to attach files to messages which are then sent
by email via delayed_job. I'm also restricting files sizes to a few
megabytes. But even with modest file sizes (2MB), Heroku is timing out
due to the time it is taking to upload the files to S3 via carrierwave/
fog.

So ideally I was hoping for a way of decoupling the actual upload of
files to S3 into a background job but I can't find any off-the-shelf
way of doing this.

One workaround I'm considering is to store my message object without
the files initially (though initially set the attributes so that I can
call 'valid?' on them) and then to add the files as a second step via
a background job. Something like:

@message = @community.messages.build(params[:message])
if @message.valid?
attachments = @message.remove_attachments
@message.save!
@message.delay.update_attributes!(attachments) if
attachments.present?
@message.send_emails(attachments)
else
... render view with errors

The call to .notify_recipients is also a background job but by passing
'attachments' to this job it can be independent of the other job.

> Lee

On Jan 15, 1:14 am, Joey <j...@aghion.com> wrote:
> We ran into similar difficulties uploading and processing large images
> (>5 MB). There are actually a few Heroku limitations that might be
> impacting you:
>   - The web front-end times out requests that don't start getting a
> response within 30 seconds.
>   - The app process is killed if it exceeds the 500-600 MB memory
> limit. It may seem like plenty, but manipulations on some images will
> temporarily require more space. Check your log after a failure to see
> if this is your problem.
>   - There's a max request size of 30 MB imposed by their web front-end
> ("413 Request Entity Too Large").
>
> Even if you do image-processing in delayed_job tasks, the memory limit
> can get you. Things to try:
>   - The carrierwave_backgrounder gem did not work for us. This blog

> post describes our initial approach for delaying image-processing:http://code.dblock.org/carrierwave-delayjob-processing-of-selected-ve...

[Joey]

It's tricky to perform the store step in the background, since the
upload ends up in a temporary file location accessible only to the
dyno receiving that request. (I.e., a delayed_job won't necessarily
find the file there, and even a later request might be served by a
different web app.) For that reason, I think the direct-to-S3 upload
might work best for you.

I noticed that Heroku's Cedar stack imposes a slightly different
timeout rule... it requires a maximum of 30 seconds to the first
response byte, but after that there's a rolling 55 second window
(http://devcenter.heroku.com/articles/request-timeout). If upgrading
to Cedar is an option, you might be able to take advantage of that.