Capped Collection Support

[Nick Daugherty]

I would love to have capped collection support in Mongoose. Not only
creating capped collections (or converting existing ones), but
suppressing _id generation for things like logging.

It's less than ideal to use Mongoose for your entire app (which is
great!) then have to switch to a native driver to handle logging.

[Aaron Heckmann]

Capped collections are sooort of supported. You'd have to create the capped collection in your mongodb shell first though. The thing about them them is that they don't support all the functionality that mongoose expects which can lead to weird behavior...

• You may update the existing objects in the collection. However, the objects must not grow in size. If they do, the update will fail. (There are some possible workarounds which involve pre-padding objects; contact us in the support forums for more information, if help is needed.)
• The database does not allow deleting objects from a capped collection. Use the drop() method to remove all rows from the collection. 
  Note: After the drop you must explicitly recreate the collection.

--
http://mongoosejs.com
http://github.com/learnboost/mongoose
You received this message because you are subscribed to the Google
Groups "Mongoose Node.JS ORM" group.
To post to this group, send email to mongoo...@googlegroups.com
To unsubscribe from this group, send email to
mongoose-orm...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/mongoose-orm?hl=en

--
Aaron

[Nick Daugherty]

Yeah there are limitations to capped collections (unfortunately), but
they're still great for operations like logging. I guess the question
is 'Are there enough people using capped collections to make it worth
the effort?' There are benefits to having all your db interactions
running through the same library...and being able to create a capped
collection from Mongoose would be a plus.

As it is currently, you can work with capped collections fine (minus
deletes and most updates), but Mongoose still generates an _id for the
document, where the default in a capped collection is to not generate
an _id.

Who else is using capped collections?

On Sep 8, 11:22 pm, Aaron Heckmann <aaron.heckm...@gmail.com> wrote:
> Capped collections are sooort of supported. You'd have to create the capped
> collection in your mongodb shell first though. The thing about them them is
> that they don't support all the functionality that mongoose expects which
> can lead to weird behavior...
>

>    - You may update the existing objects in the collection. However, the

>    objects must not grow in size. If they do, the update will fail. (There are
>    some possible workarounds which involve pre-padding objects; contact us in
>    the support forums for more information, if help is needed.)

>    - The database does not allow deleting objects from a capped collection.

>    Use the drop() method to remove all rows from the collection.
>    Note: After the drop you must explicitly recreate the collection.
>

> On Thu, Sep 8, 2011 at 1:01 AM, Nick Daugherty <develo...@sentientmatter.com

[Davis Ford]

I realize this response is a bit late, but I want to use capped collections...specifically I have a bunch of hardware devices reporting data to a server -- storing all the data can become a problem -- whereas I'm really only interested historically in recent data (e.g. back one or two weeks), and new data are evaluated as they arrive, and they may trigger an event that the system processes immediately based on the value of the data.

A capped collection is perfect for this.  I don't need to update the data -- ever.  I don't even really need to index it.  It's kind of a hassle to have to drop down to the native driver to create the collection - especially since I have to create the collections on-the-fly...I can't just pre-create them.  I have one collection per hardware device, and new devices can come online at any time.

> > mongoose-orm+unsubscribe@googlegroups.com

[Christian Landgren]

Have you looked at Redis? It seems like a better fit than mongodb from what you describe?

// Christian (from cellphone)

mongoose-orm...@googlegroups.com

[Aaron Heckmann]

On Mon, Apr 23, 2012 at 11:31 PM, Davis Ford <davi...@gmail.com> wrote:

It's kind of a hassle to have to drop down to the native driver to create the collection - especially since I have to create the collections on-the-fly...I can't just pre-create them.

Yeah we should have support for that. We'll accept a pull request for that or even just open a ticket would be good: https://github.com/learnboost/mongoose/issues/new

--
Aaron

@aaronheckmann

[ekryski]

Davis,

Another way to do this would be to use a schema-less/mixed collection
for your devices (http://mongoosejs.com/docs/schematypes.html) and
then have a pre-save middleware that removes the oldest entry (or
entries) based on either some date limit or based on the count size of
your collection.

> > > > mongoose-orm...@googlegroups.com

[Aaron Heckmann]

yeah maybe https://github.com/aheckmann/mongoose-ttl could help for now.

@aaronheckmann

[Davis Ford]

All -- thanks for all the great pointers / tips.

Christian, I am using Redis w/o persistence for storing session state.  I hadn't considered it for this, b/c I do want to persist *some* history -- if the server goes down, I'd like to have something like the last two weeks worth of data.  I know Redis can do persistence, and I'm going to look into whether that will be a better fit.

mongoose-ttl and pre-save middleware may also be good options.  Thanks again for all the suggestions.

At some point, I'd like to store much more data, and have it available for statistics via m/r operations -- but I just can't justify the hardware right now.