S3 streaming upload and download

[João Pereira]

Hi,

Does anyone have some code sample/experience of how to perform this in goliath with large files? I need to process these downloads in chunks yielding them to #response from my store s3 backend class (doing the same for the current filesystem backend class without problems).

I have already seen the uber-s3 gem but no mentions to streaming and with happening gem the server goes down when trying to do this:

             # store/s3.rb

       def get

          item = Happening::S3::Item.new(@bucket,  ...)

          item.get.stream do |chunk|

            yield chunk

          end

       end

      # goliath server app file

      def response(env)

         # ...

         operation = proc { store.get { |chunk| env.chunked_stream_send chunk } }

         callback  = proc { env.chunked_stream_close }

         EM.defer operation, callback

         # Proper streaming response ...

      end

After issuing the request with curl, server aborts with:

[9742:INFO] 2012-01-24 15:21:20 :: Starting server on 0.0.0.0:9000 in development mode. Watch out for stones.

(eval):11:in `yield': can't yield from root fiber (FiberError)

from (eval):11:in `get'

        ...

[Peter Kieltyka]

Are you just downloading a file stored in a s3 bucket and sending it as a response in streams?

Why not just use em-http-request? It would work fine its publicly accessible. Otherwise you'll need s3 url auth to access the resource, but still doable.

[Joao Pereira]

Hi,

Yes, I need to be able to:

1. POST file to server, buffer it on a tempfile and upload it to s3 later
2. GET fiie from s3 and stream it in chunks to the client

Although the files are private, but I have access to the private and access key and bucket name. Would not em-http-request incur in the same error? As in this sample happening returns a em-http response, and #stream is actually just the em-http #stream method.

[Peter Kieltyka]

Cool. I wasn't aware that Happening returned an em-http object, at least thats a step into the right direction. Look at the source of em-http-request and try to figure out what is going on.

Also.. did you look at the example in the goliath source, "stream.rb" ?

[Joao Pereira]

Hi,

Currently I have this post/get streaming process working fine for the filesystem backend option, currently i'm trying to figure whats happening with that error in my s3 backend implementation.

[João Pereira]

As expected, tried to use em-http, making the file public on s3, and the problem stills there, cant yield from there. Any way to do this work?

[Ilya Grigorik]

Which of the get's is failing? 

item.get.stream do |chunk| -- ?

Hard to tell from your stack trace. It may be due to how the variables are getting captured once you start descending down multiple callbacks.

operation = proc { |env| store.get { |chunk| env.chunked_stream_send chunk } }

Try explicitly capturing the env param...

ig

[João Pereira]

Hi,

Sorry for the little details about the backtrace, I've created a simplified server example of my own (I've cleaned a lot of details, also it tests with a public dropbox file for convenience), ready to run with backtrace on comments in this small gist: https://gist.github.com/1675971

I've been around this for hours since yesterday and cant get this to work... :S

[Ilya Grigorik]

https://gist.github.com/1681479 - does the trick for me.

[João Pereira]

Great! It works perfectly, I was getting crazy around that, thanks :) Also, this is a best approach when compared with my previous defer, as it keeps the connection open till download end. 

One last trick question: 

I need to move the download logic to a separate class, so I have done it this way: https://gist.github.com/1675971 , moving the proc and callbacks to the store class. It successfully downloads the file but when it tries to hit env.chunked_stream_close server aborts, saying me that he doesn't know about 'env'. This way the server goes down, the file as been downloaded, it already have the expected size, but is corrupted as the connection was aborted.

How can I close the stream properly from there (outside server file)?

I have solved this problem by modifying the finish proc callback to yield nil and then, inside next_tick, if chunk.nil? I close the stream, but perhaps there is a best way to do this.

[Ilya Grigorik]

Wouldn't "store.get(env) { |chunk| env.chunked_stream_send chunk }" do the trick?

[Joao Pereira]

Yes it would do it, I was thinking that env would be available all around, thanks.