What happens when you call remove() on a collection?
I have just issued a number of these commands and although they return
immediately, it seems the process is ongoing in the background and
it's block any reads to all Mongo databases.
On the console I'm seeing a lot of "key seems to have moved in the
index, refinding. found:1" following an insert (which also called
ensureIndex).
On Fri, Jul 17, 2009 at 6:43 AM, David M<da...@boxedice.com> wrote:
> What happens when you call remove() on a collection?
> I have just issued a number of these commands and although they return
> immediately, it seems the process is ongoing in the background and
> it's block any reads to all Mongo databases.
> On the console I'm seeing a lot of "key seems to have moved in the
> index, refinding. found:1" following an insert (which also called
> ensureIndex).
I don't think it's taking too long, it's just that I'm removing a lot
of data very often (every hour). Blocking the entire DB for reads is
not good because it stops our entire service from working. Even
collection level blocking isn't great because we need to read from
them at least every 60 seconds.
This is perhaps the most critical issue (and only) we have now with
Mongo because we need to delete data to keep our disk usage
manageable. Is it something you think you might be able to get to
soon?
On Jul 17, 12:12 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> remove() does return immediately no matter what is happening on the
> server and blocks.
> right now it blocks all databases, but we're going to be making that
> finer grained in the near future.
> is it taking a lot longer than you think it should?
> On Fri, Jul 17, 2009 at 6:43 AM, David M<da...@boxedice.com> wrote:
> > What happens when you call remove() on a collection?
> > I have just issued a number of these commands and although they return
> > immediately, it seems the process is ongoing in the background and
> > it's block any reads to all Mongo databases.
> > On the console I'm seeing a lot of "key seems to have moved in the
> > index, refinding. found:1" following an insert (which also called
> > ensureIndex).
Did you mean blocking for remove() instead of blocking for reads()?
We are going to be working on this in the near future, but its not
going to be a super quick fix.
Have you though about using capped collections? You can make a huge
capped collection, or as much disk space as you want to use on this,
and then just never have to remove()
On Sat, Jul 18, 2009 at 6:18 AM, David M<da...@boxedice.com> wrote:
> I don't think it's taking too long, it's just that I'm removing a lot
> of data very often (every hour). Blocking the entire DB for reads is
> not good because it stops our entire service from working. Even
> collection level blocking isn't great because we need to read from
> them at least every 60 seconds.
> This is perhaps the most critical issue (and only) we have now with
> Mongo because we need to delete data to keep our disk usage
> manageable. Is it something you think you might be able to get to
> soon?
> On Jul 17, 12:12 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
>> remove() does return immediately no matter what is happening on the
>> server and blocks.
>> right now it blocks all databases, but we're going to be making that
>> finer grained in the near future.
>> is it taking a lot longer than you think it should?
>> On Fri, Jul 17, 2009 at 6:43 AM, David M<da...@boxedice.com> wrote:
>> > What happens when you call remove() on a collection?
>> > I have just issued a number of these commands and although they return
>> > immediately, it seems the process is ongoing in the background and
>> > it's block any reads to all Mongo databases.
>> > On the console I'm seeing a lot of "key seems to have moved in the
>> > index, refinding. found:1" following an insert (which also called
>> > ensureIndex).
When I call remove() then it's blocking reads from the entire
database. Since I have a lot of data that is being remove()'d then
that is taking a long time and so reads are being blocked for a long
time.
I have looked at capped collections but the data size is not constant
and not predictable. We have to keep data for x days and then we
remove it, and this is dependent upon the type of account the user has
(which may change). We are also keeping high sample rate data which is
processed and then removed every hour.
On Jul 18, 2:50 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> Did you mean blocking for remove() instead of blocking for reads()?
> We are going to be working on this in the near future, but its not
> going to be a super quick fix.
> Have you though about using capped collections? You can make a huge
> capped collection, or as much disk space as you want to use on this,
> and then just never have to remove()
> On Sat, Jul 18, 2009 at 6:18 AM, David M<da...@boxedice.com> wrote:
> > I don't think it's taking too long, it's just that I'm removing a lot
> > of data very often (every hour). Blocking the entire DB for reads is
> > not good because it stops our entire service from working. Even
> > collection level blocking isn't great because we need to read from
> > them at least every 60 seconds.
> > This is perhaps the most critical issue (and only) we have now with
> > Mongo because we need to delete data to keep our disk usage
> > manageable. Is it something you think you might be able to get to
> > soon?
> > On Jul 17, 12:12 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> >> remove() does return immediately no matter what is happening on the
> >> server and blocks.
> >> right now it blocks all databases, but we're going to be making that
> >> finer grained in the near future.
> >> is it taking a lot longer than you think it should?
> >> On Fri, Jul 17, 2009 at 6:43 AM, David M<da...@boxedice.com> wrote:
> >> > What happens when you call remove() on a collection?
> >> > I have just issued a number of these commands and although they return
> >> > immediately, it seems the process is ongoing in the background and
> >> > it's block any reads to all Mongo databases.
> >> > On the console I'm seeing a lot of "key seems to have moved in the
> >> > index, refinding. found:1" following an insert (which also called
> >> > ensureIndex).
> When I call remove() then it's blocking reads from the entire
> database. Since I have a lot of data that is being remove()'d then
> that is taking a long time and so reads are being blocked for a long
> time.
> I have looked at capped collections but the data size is not constant
> and not predictable. We have to keep data for x days and then we
> remove it, and this is dependent upon the type of account the user has
> (which may change). We are also keeping high sample rate data which is
> processed and then removed every hour.
> On Jul 18, 2:50 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> > Did you mean blocking for remove() instead of blocking for reads()?
> > We are going to be working on this in the near future, but its not
> > going to be a super quick fix.
> > Have you though about using capped collections? You can make a huge
> > capped collection, or as much disk space as you want to use on this,
> > and then just never have to remove()
> > On Sat, Jul 18, 2009 at 6:18 AM, David M<da...@boxedice.com> wrote:
> > > I don't think it's taking too long, it's just that I'm removing a lot
> > > of data very often (every hour). Blocking the entire DB for reads is
> > > not good because it stops our entire service from working. Even
> > > collection level blocking isn't great because we need to read from
> > > them at least every 60 seconds.
> > > This is perhaps the most critical issue (and only) we have now with
> > > Mongo because we need todeletedata to keep our disk usage
> > > manageable. Is it something you think you might be able to get to
> > > soon?
> > > On Jul 17, 12:12 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> > >> remove() does return immediately no matter what is happening on the
> > >> server and blocks.
> > >> right now it blocks all databases, but we're going to be making that
> > >> finer grained in the near future.
> > >> is it taking a lot longer than you think it should?
> > >> On Fri, Jul 17, 2009 at 6:43 AM, David M<da...@boxedice.com> wrote:
> > >> > What happens when you call remove() on a collection?
> > >> > I have just issued a number of these commands and although they return
> > >> > immediately, it seems the process is ongoing in the background and
> > >> > it's block any reads to all Mongo databases.
> > >> > On the console I'm seeing a lot of "key seems to have moved in the
> > >> > index, refinding. found:1" following an insert (which also called
> > >> > ensureIndex).
On Tue, Jul 21, 2009 at 1:11 PM, nicolas_<nicolas.fou...@gmail.com> wrote:
> I second this, being able to customize the way a capped collection
> deletes documents would be AWESOME.
> For example, only keep the documents which have the highest values for
> the key 'date'.
> On 18 juil, 16:22, David M <da...@boxedice.com> wrote:
>> When I call remove() then it's blocking reads from the entire
>> database. Since I have a lot of data that is being remove()'d then
>> that is taking a long time and so reads are being blocked for a long
>> time.
>> I have looked at capped collections but the data size is not constant
>> and not predictable. We have to keep data for x days and then we
>> remove it, and this is dependent upon the type of account the user has
>> (which may change). We are also keeping high sample rate data which is
>> processed and then removed every hour.
>> On Jul 18, 2:50 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
>> > Did you mean blocking for remove() instead of blocking for reads()?
>> > We are going to be working on this in the near future, but its not
>> > going to be a super quick fix.
>> > Have you though about using capped collections? You can make a huge
>> > capped collection, or as much disk space as you want to use on this,
>> > and then just never have to remove()
>> > On Sat, Jul 18, 2009 at 6:18 AM, David M<da...@boxedice.com> wrote:
>> > > I don't think it's taking too long, it's just that I'm removing a lot
>> > > of data very often (every hour). Blocking the entire DB for reads is
>> > > not good because it stops our entire service from working. Even
>> > > collection level blocking isn't great because we need to read from
>> > > them at least every 60 seconds.
>> > > This is perhaps the most critical issue (and only) we have now with
>> > > Mongo because we need todeletedata to keep our disk usage
>> > > manageable. Is it something you think you might be able to get to
>> > > soon?
>> > > On Jul 17, 12:12 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
>> > >> remove() does return immediately no matter what is happening on the
>> > >> server and blocks.
>> > >> right now it blocks all databases, but we're going to be making that
>> > >> finer grained in the near future.
>> > >> is it taking a lot longer than you think it should?
>> > >> On Fri, Jul 17, 2009 at 6:43 AM, David M<da...@boxedice.com> wrote:
>> > >> > What happens when you call remove() on a collection?
>> > >> > I have just issued a number of these commands and although they return
>> > >> > immediately, it seems the process is ongoing in the background and
>> > >> > it's block any reads to all Mongo databases.
>> > >> > On the console I'm seeing a lot of "key seems to have moved in the
>> > >> > index, refinding. found:1" following an insert (which also called
>> > >> > ensureIndex).
