TTL for seleted Documents

[Dylan Kirkby]

In a data collecting/processing application I'm developing, I get a big packet of data, process it, then store the original and the processed in MongoDB using Mongoose.  I want to keep a sampling of the original packets (2% of them) but recycle the space used by the other 98% of them.  TTL seems like a good way to get this thinning process done.

But the TTL index is a field in the Schema, and can't be applied to some documents and not others.

My current solution to this is to create the schema with TTL:

expiryTimestamp : { type: Date, expires: 3600 }

Then in the documents I want to expire, set expiryTimestamp to a date, and in the ones I keep set it to null

{'expiryTimestamp'      : ( keepMe ? null : new Date() )}

1. Is this the intended way to be able to do this or is there a better way to thin the database by expiring certain documents?

2. Will this be supported in future releases of Mongoose?

I am using mongoose 3.8.21 to access mongo 2.6.4

[Guilherme Souza]

Hello Dylan,

This is an intended behavior in MongoDB [1]:
"If the indexed field in a document is not a date or an array that holds a date value(s), the document will not expire. If a document does not contain the indexed field, the document will not expire"

If you want to set different expire times to each document, consider the example in the MongoDB doc [2]:
"db.log_events.createIndex( { "expireAt": 1 }, { expireAfterSeconds: 0 } )"

Just keep in mind that a TTL index does not come free of charge. MongoDB will run a thread in the background every minute to remove expired documents.

On top of that, expiring 98% of the docs is likely to result in a very fragmented data file. Maybe a compact [3] command once in a while could be useful.

Test your solution under your work load to see if this overhead is tolerable :)

[1] http://docs.mongodb.org/manual/core/index-ttl/#expiration-of-data
[2] http://docs.mongodb.org/manual/tutorial/expire-data/#expire-documents-at-a-specific-clock-time
[3] http://docs.mongodb.org/manual/reference/command/compact/

--
Documentation - http://mongoosejs.com/
Plugins - http://plugins.mongoosejs.com/
Bug Reports - http://github.com/learnboost/mongoose
Production Examples - http://mongoosejs.tumblr.com/
StackOverflow - http://stackoverflow.com/questions/tagged/mongoose
Google Groups - https://groups.google.com/forum/?fromgroups#!forum/mongoose-orm
Twitter - https://twitter.com/mongoosejs
IRC - #mongoosejs
---
You received this message because you are subscribed to the Google Groups "Mongoose Node.JS ODM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongoose-orm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Graduando em Engenharia de Controle e Automação

Universidade Estadual de Campinas

http://sitegui.com.br

[Dylan Kirkby]

Thanks for the reply and the references Guilherme!

You received this message because you are subscribed to a topic in the Google Groups "Mongoose Node.JS ODM" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongoose-orm/e2aKmufqB-U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongoose-orm...@googlegroups.com.