Best way to store a binary (image or audio) file using mongoose?

[Chris]

Hi, I'm using node.js (express & mongoose) as a backend API server for an iPhone client.  The iphone will upload binary data such as images and short audio clips to the server.  I'm confused about how to save these into the mongodb using Mongoose.  It seems that I should be using gridfs, but the gridfs-stream module that I found is difficult for me to understand (I'm day 5 on node, sorry).

Any help would be great!

[Arthur Weber]

Look my example of connect-multipart-gridfs on https://github.com/godu/connect-multipart-gridform/tree/master/examples

[Aaron Heckmann]

Many people store the file meta data in mongodb but put the file contents on S3 or similar. 

If the files are smallish you also could use a Buffer. Keep in mind that documents max out at 16MB.

  Schema({ file: { mime: String, bin: Buffer }})

A quick search on plugins.mongoosejs.com also turned up these options. There's probably more out there too:

  https://github.com/firebaseco/mongoose-attachments

  https://github.com/panta/mongoose-file

  https://github.com/panta/mongoose-thumbnail

On Mon, Feb 18, 2013 at 5:53 PM, Chris <chris....@gmail.com> wrote:

Hi, I'm using node.js (express & mongoose) as a backend API server for an iPhone client.  The iphone will upload binary data such as images and short audio clips to the server.  I'm confused about how to save these into the mongodb using Mongoose.  It seems that I should be using gridfs, but the gridfs-stream module that I found is difficult for me to understand (I'm day 5 on node, sorry).

Any help would be great!

--
--
http://mongoosejs.com - docs
http://plugins.mongoosejs.com - plugins search
http://github.com/learnboost/mongoose - source code
 
You received this message because you are subscribed to the Google
Groups "Mongoose Node.JS ORM" group.
To post to this group, send email to mongoo...@googlegroups.com
To unsubscribe from this group, send email to
mongoose-orm...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/mongoose-orm?hl=en
---
You received this message because you are subscribed to the Google Groups "Mongoose Node.JS ODM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongoose-orm...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Aaron

@aaronheckmann

[Chris Nohara]

Thank you for the responses!  I will give these suggestions a try.

[Marco Pantaleoni]

On Wednesday, February 20, 2013 5:48:21 PM UTC+1, Aaron Heckmann wrote:

A quick search on plugins.mongoosejs.com also turned up these options. There's probably more out there too:

  https://github.com/panta/mongoose-file

  https://github.com/panta/mongoose-thumbnail

I'm the author of the last two. The second builds on the first one to store images and providing automatic thumbnail creation. Both store the file upload metadata inside mongoose and keep the file itself on the filesystem. This is usually the most flexible and hassle-free approach.

Let me know if you need help/sample code with these.

Cheers,

Marco

[morinezumiiii]

Hi, Marco.

mongoose-file and mongoose-thumnail are very simple, awesome!

I want to use your mongoose-plugins.

but, doesn't work it my hope from express.js.

package.json:

{
    ...
    "dependencies": {
        ...
        "mongoose": "*",
        "mongoose-file": "*",
        "mongoose-thumbnail": "*"
   }
}

app.js:

var mongoose = require('mongoose');
var db = mongoose.connect('mongodb://localhost/sample');
var thumbnailPluginLib = require('mongoose-thumbnail');
var thumbnailPlugin = thumbnailPluginLib.thumbnailPlugin;

var PictureSchema = new mongoose.schema({
  title: String
});
PictureSchema.plugin(thumbnailPlugin, {
  name: "photo",
  inline: false
});
var Picture = db.model("Picture", PictureSchema);

app.post('/upload', function(req, res, next) {
  var picture = new Picture({ title: req.body.title });
  picture.set('image.file', req.files.image);
  picture.save(function(err) {
    if (err) {
      return next(err);
    }
  });
  res.redirect('/');
});

views/index.jade:

extends layout

block content
  div
    form(action='/upload', method='POST', enctype='multipart/form-data')
      input(type='file', name='image')
      input(type='submit', value='Upload')

Exec result MongoDB document:

{ "photo" : { "thumb" : { "rel" : null, "path" : null, "name" : null }, "lastModified" : null, "size" : null, "type" : null, "rel" : null, "path" : null, "name" : null }, "_id" : ObjectId("52526bd755b5711a23000001"), "__v" : 0 }

Is wrong usage carried out? 

Please help me, if there is time. 

Thanks.

morinezumiiii

[Marco Pantaleoni]

On Monday, October 7, 2013 10:25:06 AM UTC+2, morinezumiiii wrote:

Hi, Marco.

mongoose-file and mongoose-thumnail are very simple, awesome!

I want to use your mongoose-plugins.

Great :-) 

but, doesn't work it my hope from express.js.

app.js:

PictureSchema.plugin(thumbnailPlugin, {
  name: "photo",
  inline: false
});
var Picture = db.model("Picture", PictureSchema);

app.post('/upload', function(req, res, next) {
  var picture = new Picture({ title: req.body.title });
  picture.set('image.file', req.files.image);
  picture.save(function(err) {
    if (err) {
      return next(err);
    }
  });
  res.redirect('/');
});

in the code above, you are naming the image field as "photo" but later you save to "image". Either change "image.file" to "photo.file" in the '/upload' handler or change "photo" to "image" in the model definition.

I've completed your sample and uploaded it for public consumption:

http://github.com/panta/mongoose-thumbnail-example

http://npmjs.org/package/mongoose-thumbnail-example

Hope this helps.

Cheers,

Marco

[morinezumiiii]

Thanks for replay and mongoose-thumbnail-example created!

I understand name field in Model and '/upload' handler.

Rerun modified code, and then output error.

/home//node_modules/mongoose-file/lib/index.js:120
throw err;


Error: EXDEV, rename '/tmp/2330-5ezme8.jpg'

npm ERR! weird error 8
npm ERR! not ok code 0

in the mongo collection.

> use mongoose-thumbnail-example
switched to db mongoose-thumbnail-example
> show collections
pictures
system.indexes
> db.pictures.find()
{ "title" : "", "photo" : { "thumb" : { "rel" : null, "path" : null, "name" : null }, "lastModified" : null, "size" : null, "type" : null, "rel" : null, "path" : null, "name" : null }, "_id" : ObjectId("5254bf7507185d1a09000001"), "__v" : 0 }

Same error came out even mongoose-thumbnail-example.

In order to exec the mongoose-thumbnail, node_module is also required else?

Or, '/tmp' is require root permission...?

[Marco Pantaleoni]

On Wednesday, October 9, 2013 4:47:57 AM UTC+2, morinezumiiii wrote:

Thanks for replay and mongoose-thumbnail-example created!

I understand name field in Model and '/upload' handler.

Rerun modified code, and then output error.

/home//node_modules/mongoose-file/lib/index.js:120

the above path is a bit strange. Did you took out the username part or is it the real thing?

 

throw err;


Error: EXDEV, rename '/tmp/2330-5ezme8.jpg'

npm ERR! weird error 8
npm ERR! not ok code 0

in the mongo collection.

> use mongoose-thumbnail-example
switched to db mongoose-thumbnail-example
> show collections
pictures
system.indexes
> db.pictures.find()
{ "title" : "", "photo" : { "thumb" : { "rel" : null, "path" : null, "name" : null }, "lastModified" : null, "size" : null, "type" : null, "rel" : null, "path" : null, "name" : null }, "_id" : ObjectId("5254bf7507185d1a09000001"), "__v" : 0 }

Same error came out even mongoose-thumbnail-example.

In order to exec the mongoose-thumbnail, node_module is also required else?

to which 'node_module' do you refer? The example project (mongoose-thumbnail-example) expects to be used with npm dependencies installed locally (which means in node_modules in your current directory). I've just noticed that the instructions in the README were incorrect, I've fixed it.

Could you provide the exact sequence of commands you are using to install and run the example (and node and OS versions)?

 

Or, '/tmp' is require root permission...?

Absolutely no need to run as root!

Ciao,

Marco

[Andy VJ]

Pls mail me your sample code , it'll be very helpful for me!

[Marco Pantaleoni]

On Monday, October 14, 2013 10:03:02 AM UTC+2, Andy VJ wrote:

Pls mail me your sample code , it'll be very helpful for me!

No need to mail it: simply download it from github or npm:

https://github.com/panta/mongoose-thumbnail-example

https://npmjs.org/package/mongoose-thumbnail-example

The libraries are:

https://github.com/panta/mongoose-thumbnail

https://github.com/panta/mongoose-file

Cheers,

Marco

[Vijay Britto]

thats awesome. thanks really!! 

--
Documentation - http://mongoosejs.com/
Plugins - http://plugins.mongoosejs.com/
Bug Reports - http://github.com/learnboost/mongoose
Production Examples - http://mongoosejs.tumblr.com/
StackOverflow - http://stackoverflow.com/questions/tagged/mongoose
Google Groups - https://groups.google.com/forum/?fromgroups#!forum/mongoose-orm
Twitter - https://twitter.com/mongoosejs
IRC - #mongoosejs
---
You received this message because you are subscribed to a topic in the Google Groups "Mongoose Node.JS ODM" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongoose-orm/H-8JLNyivcc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongoose-orm...@googlegroups.com.

[Tony Eastwood]

Marco

I just tried the vanilla code from your site but I always get something like:

Error: EXDEV, rename 'C:\Users\teastwoo\AppData\Local\Temp\8196-1h2p967.log'

and it crashes out. It creates an object in the in MongoDB but there is no image bytes.

   Have you any idea what I'm doing wrong? - I'm calling it from a Chrome client and the code is hosted on a local tomcat server

Tony 

[Marco Pantaleoni]

On Wednesday, October 30, 2013 3:22:31 PM UTC+1, Tony Eastwood wrote:

Marco

I just tried the vanilla code from your site but I always get something like:

Error: EXDEV, rename 'C:\Users\teastwoo\AppData\Local\Temp\8196-1h2p967.log'

and it crashes out. It creates an object in the in MongoDB but there is no image bytes.

   Have you any idea what I'm doing wrong? - I'm calling it from a Chrome client and the code is hosted on a local tomcat server

Hi Tony,

I've not tried it on Windows, so I'm just guessing here. The library performs the upload to a temporary directory ("C:\Users\teastwoo\AppData\Local\Temp" in your case) and then rename the file to its final directory. The error means the rename operation wasn't successful. One reason this could happen on Windows is if the two directories are on different disks or partitions. You could try to specify a destination folder on C: and see if this changes the outcome.

If this turns out to be the case, please tell me, and I'll add some code to work around the issue (or feel free to submit a pull request :-)

Cheers,

Marco

[Trung Anh Trinh]

Appreciate your support, however, the question of this thread is : "how to save these into the mongodb using Mongoose".

As far as i understanding, your Library is just save the file's metadata to Mongo, but the file itself is stored in "upload_to"directory, this is not the answer to the question.

I think the right solution here is to store the "binary" directly into Mongodb, however, haven't found a really good way to do that with NodeJS.

Hope that this link will help you to decide: http://stackoverflow.com/questions/8515410/store-files-on-disk-or-mongodb

[Markus Lautenbach]

Hi,

I read about GridFS and it breaks Files into 16MB chunks, so you can save way bigger files.

And once saved, you receive a file id.

So it would probably make sense to the only save the file id in your schema.

GridFS didn't look that hard to use. You can stream directly into it, e. g. with a http request stream or file stream.

GridFS files can also be served as a stream.

I have to try that out myself to give a good example.

[Evaldas Buinauskas]

Hello there. I've tried using your example, but I'm receiving error everytime i try to upload anything. Something went nuts. It cannot recognize image property that is being sent to server. 

Can it be cause of Express 4.0 I'm using?

This is screenshot of error if it gives any information. 

[Valeri Karpov]

Hi Evaldas,

This looks like an Express error. The mongoose-thumbnail-example repo's package.json says it requires Express ~3.4, so if you're using Express 4.0 that is likely the issue.

FWIW, I wouldn't recommend handling image uploads yourself outside of academic exercises. Ink Filepicker does a fairly solid job of covering most basic image upload cases and I hear good things about Cloudinary as well, and you don't need any server integration beyond saving a URL.

Cheers,

Val

[Chowdhury Masood]

Hi, Aaron,

Thanks for your work on monggose, gm and gridfs-stream. I am using all of them. I am working on a project where we have to save a lot of files (around 1MB each) on mongodb and read them back. I used busboy to upload the files as streams and then save them in gridfs using gridfs-stream. But since all the files are pretty small, I would like to use Buffer and store them in a collection as binary data. I followed the schema you suggested. How do I do it using busboy or formidable, since I get a  stream? Also, I do I get the stored binary data from the collection as a steam using mongoose and pass to 'gm' for creating thumbnail. After 'gm' resizes the image to thumbnail, i would like to store it in the same collection. how to do i get buffer from 'gm' stream? please help.

Thanks,

-- shaheen