Uploading Files w/ and w/o Rails

[AJ ONeal]

It seems that Rails is not the way to go for uploading or downloading files. 

(especially as the sizes grow and the regex to find the multipart tags doesn't get any faster or less memory consuming).

I'm creating a file storage area for users.

• Only an authenticated user can upload a file
• A user can only download a file he owns (or has permissions to)

The nice thing is that I get to use Rails / ActiveRecord for authentication & authorization

The ugly thing is that I'm using Rails to do uploads / downloads

Is there an elegant solution which allows me to continue to use Rails for auth, but pass file handling elsewhere? 

I prefer to not to install any special apache / nginx modules

AJ ONeal

[Blaine]

On Dec 19, 1:57 pm, AJ ONeal <Alvin.ON...@gmail.com> wrote:
> It seems that Rails is not the way to go for uploading or downloading
> files.
> (especially as the sizes grow and the regex to find the multipart tags
> doesn't get any faster or less memory consuming).
>
> I'm creating a file storage area for users.
>

>    - Only an authenticated user can upload a file
>    - A user can only download a file he owns (or has permissions to)

>
> The nice thing is that I get to use Rails / ActiveRecord for authentication
> & authorization
> The ugly thing is that I'm using Rails to do uploads / downloads
>
> Is there an elegant solution which allows me to continue to use Rails for
> auth, but pass file handling elsewhere?
>
> I prefer to not to install any special apache / nginx modules
>
> AJ ONeal

Have you considered storing the path to the files in the database?
e.g. have a table for uploads, and have uploads belongs_to users, and
users has_many uploads. in the create action for your upload
controller, save the files in a user's directory somewhere, and then
store the path to the database.

[Devlin Daley]

You've stipulated that you want to use Rails for authn and authz but not use any webserver modules. These criteria are at odds.

The most robust way that I am familiar with to perform file upload/downloads (and potentially streaming) is to take advantage of the webserver that typically sits in front of Rails.

For downloading files, let the request go through your application, but instead of sending the file directly with Rails, let the webserver (or reverse proxy) do it. Sendfile and X-Accell-Redirect are your friends here. They're usually on the list of included modules too.

For uploading, both Apache and Nginx fully buffers the request before it sends it to the Rails backend in one big gulp. Rails still has to do all the mime-parsing to extract the files (but it's not held up while the file is being uploaded, a huge win especially for clients with slow internet connections).

For more file upload awesomeness I'd look at the Nginx File Upload Module http://brainspl.at/articles/2008/07/20/nginx-upload-module and http://www.motionstandingstill.com/nginx-upload-awesomeness/2008-08-13/ . It parses the POST, writes the files to disk and then when it dispatches to your rails backend it gives you the paths to the files instead of the files themselves. Then you'd run your authorization etc. and just move files around.

Yep, it's a dependency outside of your Rails app. There's always a lot more to a Rails app than just a Rails app.

--
Utah Ruby Users Group
ur...@googlegroups.com
http://groups.google.com/group/urug/
- All meeting times and places can be found here.
 
** Please suffix your subject with "[JOB]" if your message is about job opportunities.

--
Devlin Daley
dev...@instructure.com
801-494-1964 x101

[Dave South]

How large are the files you are planning to upload?

On Dec 19, 1:57 pm, AJ ONeal <Alvin.ON...@gmail.com> wrote:

> It seems that Rails is not the way to go for uploading or downloading
> files.
> (especially as the sizes grow and the regex to find the multipart tags
> doesn't get any faster or less memory consuming).
>
> I'm creating a file storage area for users.
>

>    - Only an authenticated user can upload a file
>    - A user can only download a file he owns (or has permissions to)

[AJ ONeal]

@Devlin Thanks a lot.

You've stipulated that you want to use Rails for authn and authz but not use any webserver modules. These criteria are at odds.

I prefer not to install additional modules on my web server. However, if it simplifies things, I'm not morally opposed.

The most robust way that I am familiar with to perform file upload/downloads (and potentially streaming) is to take advantage of the webserver that typically sits in front of Rails.

I store all files in /var/webapps/MyRailsApp/private/

Apache does not have access to this directory.

My controller determines whether or not the user is allowed to get a file.

My understanding (and perhaps my ignorance) is that Apache can't serve the file.

 

For downloading files, let the request go through your application, but instead of sending the file directly with Rails, let the webserver (or reverse proxy) do it. Sendfile and X-Accell-Redirect are your friends here. They're usually on the list of included modules too.

Am I to understand that if I use X-Sendfile I can tell Apache to send a file which is not in a public space?

If that's the case, this post I found looks simple enough to follow.

http://john.guen.in/past/2007/4/17/send_files_faster_with_xsendfile/

 

For uploading, both Apache and Nginx fully buffers the request before it sends it to the Rails backend in one big gulp. Rails still has to do all the mime-parsing to extract the files (but it's not held up while the file is being uploaded, a huge win especially for clients with slow internet connections).

For more file upload awesomeness I'd look at the Nginx File Upload Module http://brainspl.at/articles/2008/07/20/nginx-upload-module and http://www.motionstandingstill.com/nginx-upload-awesomeness/2008-08-13/ . It parses the POST, writes the files to disk and then when it dispatches to your rails backend it gives you the paths to the files instead of the files themselves. Then you'd run your authorization etc. and just move files around.

I like that. I believe the same module is available for apache (or at least I came across one which was written for both).

[AJ ONeal]

How large are the files you are planning to upload?

I don't know.

I have a list of web apps that I'm working towards creating.

Right now I'm in the prototyping phase where I'm creating snippets of code that do particular tasks that I believe will be integrated into many or all of them.

One of them is a directory application. It wouldn't be unreasonable to believe that a few people could be using the application at one time and uploading 300mb worth of pictures (one jpeg can be something like 10mb nowadays on one of those new-fangled cameras, no?).

Success favors the prepared, right?

AJ ONeal

[Devlin Daley]

On Tue, Dec 22, 2009 at 4:39 PM, AJ ONeal <cool...@gmail.com> wrote:

I store all files in /var/webapps/MyRailsApp/private/

Apache does not have access to this directory.

My controller determines whether or not the user is allowed to get a file.

My understanding (and perhaps my ignorance) is that Apache can't serve the file.

 

For downloading files, let the request go through your application, but instead of sending the file directly with Rails, let the webserver (or reverse proxy) do it. Sendfile and X-Accell-Redirect are your friends here. They're usually on the list of included modules too.

Am I to understand that if I use X-Sendfile I can tell Apache to send a file which is not in a public space?

If that's the case, this post I found looks simple enough to follow.

http://john.guen.in/past/2007/4/17/send_files_faster_with_xsendfile/

That is definitely the case.

The user that apache is running as has to have read permissions on the file system to access the files. The files should not be in Apache's public path (so your authz cannot be routed around).

The plugin you linked to looks like it does the right thing, or you could just use the stuff built into Rails

http://www.therailsway.com/2009/2/22/file-downloads-done-right

— Devlin

[AJ ONeal]

Am I to understand that if I use X-Sendfile I can tell Apache to send a file which is not in a public space?

If that's the case, this post I found looks simple enough to follow.

http://john.guen.in/past/2007/4/17/send_files_faster_with_xsendfile/

That is definitely the case.

The plugin you linked to looks like it does the right thing, or you could just use the stuff built into Rails

http://www.therailsway.com/2009/2/22/file-downloads-done-right

Thanks again.

If it's that easy, why isn't that the default?

AJ ONeal

[Devlin Daley]

Probably because Rails can't make any assumptions about which webserver it is sitting behind (or not) since there are so many that it could be. 

What would be cool is if the webserver behaved like proper middleware and advertised it's functionality downstream. When a request came in, the x-send-file middleware would set a header so that the application could detect that the webserver supported it, then default to x-send-file.