capped collections and 'overflow'

[barnybug]

Hi,

We're using capped collections with tailable cursors as a queue to
pass messages from one component in our system to another
asynchronously. One concern we have is if we write too quickly to the
queue it could overflow and begin overwriting queued but not yet
processed 'messages'. Increasing the capped collection size reduces
the risk, but we'd like to be able to spot it before it happens.

Is there a way of telling what the next document to be overwritten is,
so we could possibly monitor this and slow if it's not yet been
processed?

Has anyone experience with this kind of problem or suggestions how to
handle?

kind regards,

Barnaby

[Scott Hernandez]

It sounds like a problem you will get any time you use a fixed size
buffer/queue. In the case it is not large enough you will get either
roll-over on top of unprocessed or would block and error if not. In
either case you will need to address this in your application or by
increasing the capped collection.

There is no way to tell where the rollover will occur as the system
does this automatically as new data comes in.

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>

[cheetah]

You can run db.coll.stats() to get the current size and max size of the capped collection. But how do you deal with overflow is on your own implementation. For example, you can add a proxy as the single point of all capped collection writing operations. The proxy can therefore check the size and either increase the size of the capped collection or create a new capped collection. 

Wen

[barnybug]

Thanks for the response. It occurred to me after posting this that since a capped collection is always in natural order, simply doing a findOne() on the collection returns the entry about to be overwritten. This is good enough for us to check when writing, then we just need the queue consumer to update a record in another collection recording current progress, so the producer can check and throttle back if about to overflow.

 

It does add extra writes from the consumer, but since we don't expect to be able to empty the queue too quickly it's good enough to make these writes periodical updates, say every second to avoid a write for every queue operation consumed.

I think the only caveat with this method is that you assume an insert will only overwrite the first record, which seems reasonable with fixed size records as we have.

B