Need feedback on a small change to the nginx-gridfs module
brahmana
I have made a small tweak to the nginx-gridfs module to serve my
requirement. I am not sure if it is generic enough to raise a pull
request. So I thought of posting it here and asking for some feedback.
Setup :
- Rails app accepts user uploaded images and stores them in GridFS,
with unique file names
- Nginx uses the nginx-gridfs module to serve these files from GridFS
directly.
Requirement
- Multiple versions (sizes) of every uploaded image needs to exist -
each address by the same (or similar) url with a query parameter
specifying the version.
- Creating versions during the upload request results in bad user
experience (too much time) and hence needs to be done by a background
process - using a queuing server. This means additional versions are
not available right away.
- The clients would not know when these versions are available and
would get 404 when they try to fetch a non-default version of the
image.
- So the nginx-gridfs module needs to be smart enough to fallback to
the default version when the requested version is not available.
i.e. Look for the specified version. If present serve it. Else fetch
the default version and serve that.
I have a very rudimentary implementation of this. It compiles and
works fine - no crashes, doesn't seem to leak memory. I haven't done
any benchmarking, but it seems do just fine under simple loads.
Here is how the different versions of the image files are stored and
addressed :
The value of the 'type' query parameter is added as a suffix to the
filename before searching.
Ex : original filename - 'images/4f4292ff1d41c80be4000013' and is
requested as 'http://hostname/files/images/4f4292ff1d41c80be4000013'
Small version of the same image will be at : 'images/
4f4292ff1d41c80be4000013_small' or 'images/4f4292ff1d41c80be4000013_s'
and is requested as 'http://hostname/files/images/
4f4292ff1d41c80be4000013?type=small' or 'http://hostname/files/images/
4f4292ff1d41c80be4000013?type=s'
If the requested version is not found, then we fallback to the default
version, i.e. the one without any suffixes (a.k.a 'images/
4f4292ff1d41c80be4000013') and serve that one.
Few notes about the changed code :
1) It assumes that the field type to be queried against on a String
type
2) It assumes that the gridfs and gridfile are reusable types - i.e
same object can be used across multiple queries.
3) It is probably not super optimized, but that is ok for now.
For these same reasons I have not raised a pull request as the use of
this change for anyone else is doubtful.
And here is the code commit on github.cm for this change
(finally.. !!) - https://github.com/brahmana/nginx-gridfs/commit/5f32a63678b9f997bcd4539b89f357dc98be65d4
It would be very helpful to get some feedback from you guys.
Regards,
brahmana
@sribrahmana - http://www.twitter.com/sribrahmana
Kyle Banker
Srirang Doddihal
Thanks for the feedback.
On Tue, Mar 6, 2012 at 1:16 AM, Kyle Banker <ky...@10gen.com> wrote:
> Hi Brahmana,
>
> Nice work. This seems to me to be a potentially useful change. Before
> submitting the pull request, please:
>
> 1.) Thoroughly document your code.
>
> 2.) Document the behavior in the README, including a couple of examples.
Done. And submitted a pull request too.
I have another question :
Every time a file is read from gridfs using this nginx-gridfs module,
an *insert* is made to system.indexes. This behavior is without my
changes. It has always been like this.
Is this something in the nginx-gridfs module or in the mongo-c-driver?
Or is it something that is fixed in the current version of the
mongo-c-driver which nginx-gridfs is yet to pull in. (Currently it
uses the mongo-c-driver commit from August 2011)
Here is a sample log showing the inserts to the system.indexes on
requesting a file in gridfs from nginx.
Tue Mar 6 10:36:32 [conn5] insert testdb_development.system.indexes 0ms
Tue Mar 6 10:36:32 [conn5] runQuery called testdb_development.$cmd {
getlasterror: 1 }
Tue Mar 6 10:36:32 [conn5] run command testdb_development.$cmd {
getlasterror: 1 }
Tue Mar 6 10:36:32 [conn5] command testdb_development.$cmd command: {
getlasterror: 1 } ntoreturn:1 reslen:67 0ms
Tue Mar 6 10:36:32 [conn5] index already exists with diff name
_files_idn { files_id: 1, n: 1 }
Tue Mar 6 10:36:32 [conn5] insert testdb_development.system.indexes 0ms
Tue Mar 6 10:36:32 [conn5] runQuery called testdb_development.$cmd {
getlasterror: 1 }
Tue Mar 6 10:36:32 [conn5] run command testdb_development.$cmd {
getlasterror: 1 }
Tue Mar 6 10:36:32 [conn5] command testdb_development.$cmd command: {
getlasterror: 1 } ntoreturn:1 reslen:67 0ms
Tue Mar 6 10:36:32 [conn5] runQuery called
testdb_development.fs.files { query: { filename:
"users/4f5337221d41c81b0f000004/picture" }, orderby: { uploadDate: -1
} }
Tue Mar 6 10:36:32 [conn5] query testdb_development.fs.files
ntoreturn:1 nreturned:1 reslen:221 0ms
Tue Mar 6 10:36:32 [conn5] runQuery called
testdb_development.fs.chunks { query: { files_id:
ObjectId('4f5337241d41c81b0f000005'), n: 0 }, orderby: { n: 1 } }
Tue Mar 6 10:36:32 [conn5] query testdb_development.fs.chunks
ntoreturn:1 nreturned:1 reslen:11322 0ms
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-dev" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-dev/-/weGOGNexoOEJ.
> To post to this group, send email to mongo...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-dev...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-dev?hl=en.
--
Regards,
Srirang G Doddihal
Brahmana.
The LIGHT shows the way.
The WISE see it.
The BRAVE walk it.
The PERSISTENT endure and complete it.
I want to do it all ALONE.