Rails 3.2 and Streaming using response_body

[Matthew Fordham]

I am working on a streaming download (CSV) from Rails 3.2 and am coming up against an issue of the initial page request taking a long time. The following controller code illustrates my issue:

      self.response_body = Enumerator.new do |y|
        10_000_000.times do
          y << "Hello World"
        end
      end 

With the above, the response does seem like its streaming (from a server than can support it... Unicorn, in my case). That said, before it starts streaming it hangs for a much longer time than I'd like. If I change it to the following, it starts much faster:

      self.response_body = Enumerator.new do |y|
        1000.times do
          y << "Hello World"
        end
      end

My understanding is that the response should begin with the first iteration of the loop, but it seems the larger loops are causing that initial load time to lengthen. If each iteration is output as it happens, shouldn't it take the same amount of time to kick off the streaming process, regardless of how many total iterations there will be???

Here is an explanation of the technique I am attempting. Maybe I am misinterpreting or missing a step?:http://facebook.stackoverflow.com/questions/3507594/ruby-on-rails-3-streaming-data-through-rails-to-client/4320399#4320399

Thanks for any insight you may have!

[Cam Allen]

Hi Matthew, 

I have built a CSV streaming response that suffers from the same problem that you've described. It seems the larger content amount will delay the response headers being sent, thus signifying the start of the stream, i.e. save dialog in browser. It seems this has to do with they web server being used and possibly buffering responses before sending the stream but haven't found a directive to fix it. 

I've tested with webrick, puma, thin and passenger+nginx and dug into the configurations to no avail. Note: only passenger and puma stream properly and they both do so with the delayed response for large streams. 

Did you every find out what was causing the delay on larger streams?

Cam

[Cam Allen]

For future reference, it turns out that Rack is calling each twice on the response_body, see 

https://groups.google.com/forum/#!topic/rack-devel/YgEzAlZd8YA

Once i added some checks to only stream once on the second call (rack closing the connection) it worked as expected! Note: if you only respond on the first call then the response will be empty.

Hopefully this saves someone some time in the future!

Cam