offloading big uploads to front-end webserver

[bluesc...@gmail.com]

I'm doing a website which has quite a few big uploads / downloads to
manage.

Since rails isn't multithreaded (yet), this means every big upload/
download will keep a mongrel/thin/whatever thread for itself doing
nothing but waiting for the transfer to finish.

Now, for public files, this of course gets handled by my frontend
nginx server.

For private files, I use the X-Accel-Redirect header, so the transfer
gets offloaded to nginx as well.

Only for uploads, I couldn't find a nice way to handle them.
Is there a way to somehow tell nginx to first receive the whole file,
then transfer it to mongrel/thin so it doesn't lock up a full
instance?
Any way to do this, while first consulting rails if this should be
allowed?

Any help is greatly appreciated.

Greetings,
Mathijs

[Peter De Berdt]

First of all, Mongrel is multithreaded and Mongrel doesn't hand off the uploaded file to Rails until after the upload itself (i.e. the upload itself doesn't block the mongrel at all). That said, after the upload the Mongrel will be locked for the duration of processing in Rails.

Depending on what you are actually doing in Rails, you have several possibilities:

1) Just saving the file to disk: just let Rails handle it using a Tempfile, see the attachment_fu source code. It's pretty fast and won't lock your Mongrel for long since the Rails request, which IS single threaded, won't take long

2) Extensive processing: offload the upload to a background process, and here you have another choice:

     - BackgroundRB (and other background processors): has access to the Rails stack and runs behind Rails, i.e. Rails will hand over the file to the background worker

     - Merb: a Rails like framework that contrary to Rails is multithreaded, it can sit behind a single mongrel and process heaps of uploads without locking the mongrel. You can plugin ActiveRecord into merb for model access. You'll also need some rewrite magic to direct uploads to the Merb application if you want the rest served through Rails.

Best regards

Peter De Berdt