CarrierWave and Latency

[FredO]

I have been working on pilot project that (in part) uploads photos.
My first cut used RackSpace's CloudFiles (via the cloudfiles gem), in
order to store the photos in that great bit bucket in the cloud.

Photos are uploaded asynchronously (from the browser), and I had to
poll from the browser while waiting for each photo URL to become
available, and I could then display the photos in the browser.
Usually URL's would become available in 1 to 5 seconds.

However, I ran into a weird problem in that code seemed to work in the
context of a rails process, but not work in a stand-alone process.
(Yeah, there is a stand-alone process that also has to process and
store photos).

Time, being of the essence, I threw up my hands, and just saved the
photos in local file-system of the server (It's only a pilot right?).
Wow, what a difference! Using the file-system, the photos were
available in tenths of a second or less.

Of course in terms of storing gazillions of pictures this is not a
scalable solution.

So, now we are in the process of moving the application to AWS or
EngineYard. And the current thinking is to store the photos in an S3
bucket. And there is a recommendation to use CarrierWave and fog.
Well, it looks like this technology hides some of some of the grittier
details of uploading photos to S3, but I am still concerned about how
long it takes before a photo is uploaded to S3 and you can get back a
URL?

Has anyone had any experience with latency issues with S3 storage?

It seems to me that if S3 storage latency results in an objectionable
UI experience. I could stage uploaded files in the file system until
the user navigates away from that upload and edit page, and then move
them into S3 storage.

Has anyone done this before?

[Matt Clark]

HI Fred,

We've used S3 to store images on a few projects, and haven't had
latency issues to deal with. Another advantage to looping image
uploads through your server is that you can manipulate and validate
them at upload time instead of in some kind of background job.

Mattc

> --
> You received this message because you are subscribed to the Google Groups "pdxruby" group.
> To post to this group, send email to pdx...@googlegroups.com.
> To unsubscribe from this group, send email to pdxruby+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/pdxruby?hl=en.
>

[Jesse Cooke]

I've built close to the same thing for a client early last year. Image is uploaded & stored locally, processed in the background, and then uploaded to S3 and served via CloudFront. I had written a Rack::Metal endpoint that would respond with the URL when it was ready, and then the image would be displayed. Polling is not great, but it works alright. There are a lot of ways you can handle this: polling, EventSource + Streaming[1][2] or even WebSockets (though you don't need bidirectional).

You may have found services that take care of this for you, like Transloadit[3] which I've also used, but they can get backed up a little.

[1] http://www.sinatrarb.com/intro.html#Streaming%20Responses

[2] http://api.rubyonrails.org/classes/ActionController/Streaming.html

[3] http://transloadit.com/