GridFS Nginx module

[Michael Dirolf]

Off and on over the past couple of weeks I've been working on an Nginx
module for serving content directly from MongoDB's GridFS. I've
released the result under the Apache 2 license and it's available here:

http://github.com/mdirolf/nginx-gridfs/

Check the README for more info.

Warning / disclaimer: this isn't 10gen supported code and is probably
nowhere near production quality yet. It's something I've been working
on for fun. I wouldn't use it unless you're willing to at least deal
with and report issues, if not contribute back yourself.

- Mike

[Adrian Madrid]

Man, that sounds like fun!

Adrian Madrid
My eBiz, Developer
3082 W. Maple Loop Dr
Lehi, UT 84043
801-341-3824

[Wouter]

Great to see this in development! Very curious on the progress :-)

[Jon Hancock]

Looks good. One reason I chose not to dig into GridFS is my ruby app
processes need to return files using merb's send_file. This simply
tells nginx (or whatever proxy) to send the file, bypassing my ruby
process. I did not see how I would do this with mongo. Instead I
would have to read the bytes in from mongo through ruby and stream
them out form there.

Is this the type of use your module would work for? One feature
needed is for the files to only be available when using send_file from
the app process and not publicly available to any user hitting nginx.

Jon

[diegomsana]

i'm watching this closely, i certainly could hapily use it with my
app :)

[Mike]

Sorry for the delayed response - didn't see this message until just
now.

This module is not designed for that use case - it's designed for
serving content directly from GridFS through Nginx, bypassing your
application logic entirely.

One crazy idea you could think about for your case is to mount GridFS
as a FUSE filesystem:
http://github.com/mikejs/gridfs-fuse/tree/master

Then you could (presumably) send files from GridFS as if they were
regular files on the filesystem.

[Mike]

Cool - give it a shot and feel free to contribute!

[John Nunemaker]

Could be interesting to try out the GridFS fuse plugin which basically
makes gridfs work just like a normal file system.

[Wouter]

How exactly do you match filenames to GridFS? What about
subdirectories?

(like:
host/gridfs/images/image1.gif
host/gridfs/css/style.css)

[marc]

you have to set the filename of the gridfs object the same as you want
to access it.

e.g.

http://cdn.example.com/images/test/test.jpg
should be added with filname: images/test/test.jpg

[Wouter]

That makes sense :-)

Well I just hope development on this project continues, sounds very
promising! If only I could C :-)

[Jon Hancock]

Another reason I chose not to use GridFS is I couldn't see how I would
manage backups and file syncs efficiently. For example, I'll have
files on my dev machine and need to sync to my production server. How
would I sync just the files in my dev db to production keeping in mind
that my dev machine is just a subset of production? Also, for
backups, how would I do an "incremental" backup (only new or changed
files, preferably block changes, not the whole file)? Don't I have to
dump the entire db?

If I leave the files outside mongo, I can use well understood backup
and rsync tools.
What are the methods expected to do this with GridFS?

Jon

[Michael Dirolf]

You should use the replication that is build into MongoDB. One of the
cool things about GridFS is you get to leverage replication and
sharding.