remove()

[David M]

What happens when you call remove() on a collection?

I have just issued a number of these commands and although they return
immediately, it seems the process is ongoing in the background and
it's block any reads to all Mongo databases.

On the console I'm seeing a lot of "key seems to have moved in the
index, refinding. found:1" following an insert (which also called
ensureIndex).

[Eliot Horowitz]

remove() does return immediately no matter what is happening on the
server and blocks.

right now it blocks all databases, but we're going to be making that
finer grained in the near future.

is it taking a lot longer than you think it should?

[David M]

I don't think it's taking too long, it's just that I'm removing a lot
of data very often (every hour). Blocking the entire DB for reads is
not good because it stops our entire service from working. Even
collection level blocking isn't great because we need to read from
them at least every 60 seconds.

This is perhaps the most critical issue (and only) we have now with
Mongo because we need to delete data to keep our disk usage
manageable. Is it something you think you might be able to get to
soon?

[Eliot Horowitz]

Did you mean blocking for remove() instead of blocking for reads()?
We are going to be working on this in the near future, but its not
going to be a super quick fix.

Have you though about using capped collections? You can make a huge
capped collection, or as much disk space as you want to use on this,
and then just never have to remove()

[David M]

When I call remove() then it's blocking reads from the entire
database. Since I have a lot of data that is being remove()'d then
that is taking a long time and so reads are being blocked for a long
time.

I have looked at capped collections but the data size is not constant
and not predictable. We have to keep data for x days and then we
remove it, and this is dependent upon the type of account the user has
(which may change). We are also keeping high sample rate data which is
processed and then removed every hour.

[nicolas_]

I second this, being able to customize the way a capped collection
deletes documents would be AWESOME.

For example, only keep the documents which have the highest values for
the key 'date'.

On 18 juil, 16:22, David M <da...@boxedice.com> wrote:
> When I call remove() then it's blocking reads from the entire
> database. Since I have a lot of data that is being remove()'d then
> that is taking a long time and so reads are being blocked for a long
> time.
>
> I have looked at capped collections but the data size is not constant
> and not predictable. We have to keep data for x days and then we
> remove it, and this is dependent upon the type of account the user has
> (which may change). We are also keeping high sample rate data which is
> processed and then removed every hour.
>
> On Jul 18, 2:50 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
>
>
>
> > Did you mean blocking for remove() instead of blocking for reads()?
> > We are going to be working on this in the near future, but its not
> > going to be a super quick fix.
>
> > Have you though about using capped collections?  You can make a huge
> > capped collection, or as much disk space as you want to use on this,
> > and then just never have to remove()
>
> > On Sat, Jul 18, 2009 at 6:18 AM, David M<da...@boxedice.com> wrote:
>
> > > I don't think it's taking too long, it's just that I'm removing a lot
> > > of data very often (every hour). Blocking the entire DB for reads is
> > > not good because it stops our entire service from working. Even
> > > collection level blocking isn't great because we need to read from
> > > them at least every 60 seconds.
>
> > > This is perhaps the most critical issue (and only) we have now with

> > > Mongo because we need todeletedata to keep our disk usage

[Nicolas Clairon]

+1

This can be done via a javascript function stored somewhere...

Is there a jira ticket fired yet ?