If you run the compact command on a replica set secondary, the secondary will automatically demote itself to a "recovery" state until the compaction is complete. Since your compaction is failing, I think that is why it never comes back out of the state.
Why are you running a compact on the secondary, is there something wrong?
The assertion you are seeing suggests that it is expecting a particular namespace but failing to find it. If that's the case, then it sounds like this secondary might not be very healthy. Can you resync it from the master? That would actually defrag your data similar to a compaction anyway.
On Mon, May 14, 2012 at 1:26 PM, Adam C <ad...@10gen.com> wrote:
> If you run the compact command on a replica set secondary, the secondary
> will automatically demote itself to a "recovery" state until the compaction
> is complete. Since your compaction is failing, I think that is why it never
> comes back out of the state.
Yes, but it looks like a bug?
> Why are you running a compact on the secondary, is there something wrong?
I wanted to test the compact command on a test replica set of 3 nodes.
When i ran the command on the master i have this output:
PRIMARY> db.test.runCommand( "compact")
{
"errmsg" : "will not run compact on an active replica set primary as
this is a slow blocking operation. use force:true to force",
"ok" : 0
}
That's why i ran it on a secondary, as the error message suggest not
to run this command on a primary
> The assertion you are seeing suggests that it is expecting a particular
> namespace but failing to find it. If that's the case, then it sounds like
> this secondary might not be very healthy. Can you resync it from the
> master? That would actually defrag your data similar to a compaction
> anyway.
The secondary is healthy.
as i said, i only want to test the compact command, and noticed that
if by mistake i run it on a collection that doesn't exists (i
copy/paste the example in the doc without checking), then the entire
node will switch to recorery mode forever (until i restart the
process).
This is a simple bug to reproduce:
- create a replica set on 3 nodes
- log on the secondary with mongo cli
- execute db.mycollection.runCommand( "compact")
> To post to this group, send email to mongodb-user@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
Granted, if the compaction fails it should be more graceful, and the fact that it stays in recovering does look like a bug (I will look to reproduce and file one tomorrow), but if you run compact on a collection that actually exists, does it succeed?
On Monday, May 14, 2012 10:10:23 PM UTC+1, sebest wrote:
> Hi Adam
> On Mon, May 14, 2012 at 1:26 PM, Adam C <ad...@10gen.com> wrote: > > If you run the compact command on a replica set secondary, the secondary > > will automatically demote itself to a "recovery" state until the > compaction > > is complete. Since your compaction is failing, I think that is why it > never > > comes back out of the state.
> Yes, but it looks like a bug?
> > Why are you running a compact on the secondary, is there something > wrong?
> I wanted to test the compact command on a test replica set of 3 nodes.
> When i ran the command on the master i have this output: > PRIMARY> db.test.runCommand( "compact") > { > "errmsg" : "will not run compact on an active replica set primary > as > this is a slow blocking operation. use force:true to force", > "ok" : 0 > }
> That's why i ran it on a secondary, as the error message suggest not > to run this command on a primary
> > The assertion you are seeing suggests that it is expecting a particular > > namespace but failing to find it. If that's the case, then it sounds > like > > this secondary might not be very healthy. Can you resync it from the > > master? That would actually defrag your data similar to a compaction > > anyway.
> The secondary is healthy.
> as i said, i only want to test the compact command, and noticed that > if by mistake i run it on a collection that doesn't exists (i > copy/paste the example in the doc without checking), then the entire > node will switch to recorery mode forever (until i restart the > process).
> This is a simple bug to reproduce: > - create a replica set on 3 nodes > - log on the secondary with mongo cli > - execute db.mycollection.runCommand( "compact")
> > Adam
> > On Saturday, May 12, 2012 1:51:23 AM UTC+1, sebest wrote:
> > To post to this group, send email to mongodb-user@googlegroups.com. > > To unsubscribe from this group, send email to > > mongodb-user+unsubscribe@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/mongodb-user?hl=en.
On Tue, May 15, 2012 at 1:47 AM, Adam C <ad...@10gen.com> wrote:
> Quick question - does test.mycollection exist?
No, it does not.
> Granted, if the compaction fails it should be more graceful, and the fact
> that it stays in recovering does look like a bug (I will look to reproduce
That's my point.
> and file one tomorrow), but if you run compact on a collection that actually
> exists, does it succeed?
Yes, but i must use 'force' to compact from the primary.
> On Monday, May 14, 2012 10:10:23 PM UTC+1, sebest wrote:
>> Hi Adam
>> On Mon, May 14, 2012 at 1:26 PM, Adam C <ad...@10gen.com> wrote:
>> > If you run the compact command on a replica set secondary, the secondary
>> > will automatically demote itself to a "recovery" state until the
>> > compaction
>> > is complete. Since your compaction is failing, I think that is why it
>> > never
>> > comes back out of the state.
>> Yes, but it looks like a bug?
>> > Why are you running a compact on the secondary, is there something
>> > wrong?
>> I wanted to test the compact command on a test replica set of 3 nodes.
>> When i ran the command on the master i have this output:
>> PRIMARY> db.test.runCommand( "compact")
>> {
>> "errmsg" : "will not run compact on an active replica set primary
>> as
>> this is a slow blocking operation. use force:true to force",
>> "ok" : 0
>> }
>> That's why i ran it on a secondary, as the error message suggest not
>> to run this command on a primary
>> > The assertion you are seeing suggests that it is expecting a particular
>> > namespace but failing to find it. If that's the case, then it sounds
>> > like
>> > this secondary might not be very healthy. Can you resync it from the
>> > master? That would actually defrag your data similar to a compaction
>> > anyway.
>> The secondary is healthy.
>> as i said, i only want to test the compact command, and noticed that
>> if by mistake i run it on a collection that doesn't exists (i
>> copy/paste the example in the doc without checking), then the entire
>> node will switch to recorery mode forever (until i restart the
>> process).
>> This is a simple bug to reproduce:
>> - create a replica set on 3 nodes
>> - log on the secondary with mongo cli
>> - execute db.mycollection.runCommand( "compact")
>> > Adam
>> > On Saturday, May 12, 2012 1:51:23 AM UTC+1, sebest wrote:
>> > To post to this group, send email to mongodb-user@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > mongodb-user+unsubscribe@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
<sebastien.estie...@gmail.com> wrote:
> On Tue, May 15, 2012 at 1:47 AM, Adam C <ad...@10gen.com> wrote:
>> Quick question - does test.mycollection exist?
> No, it does not.
>> Granted, if the compaction fails it should be more graceful, and the fact
>> that it stays in recovering does look like a bug (I will look to reproduce
> That's my point.
>> and file one tomorrow), but if you run compact on a collection that actually
>> exists, does it succeed?
> Yes, but i must use 'force' to compact from the primary.
>> Adam.
>> On Monday, May 14, 2012 10:10:23 PM UTC+1, sebest wrote:
>>> Hi Adam
>>> On Mon, May 14, 2012 at 1:26 PM, Adam C <ad...@10gen.com> wrote:
>>> > If you run the compact command on a replica set secondary, the secondary
>>> > will automatically demote itself to a "recovery" state until the
>>> > compaction
>>> > is complete. Since your compaction is failing, I think that is why it
>>> > never
>>> > comes back out of the state.
>>> Yes, but it looks like a bug?
>>> > Why are you running a compact on the secondary, is there something
>>> > wrong?
>>> I wanted to test the compact command on a test replica set of 3 nodes.
>>> When i ran the command on the master i have this output:
>>> PRIMARY> db.test.runCommand( "compact")
>>> {
>>> "errmsg" : "will not run compact on an active replica set primary
>>> as
>>> this is a slow blocking operation. use force:true to force",
>>> "ok" : 0
>>> }
>>> That's why i ran it on a secondary, as the error message suggest not
>>> to run this command on a primary
>>> > The assertion you are seeing suggests that it is expecting a particular
>>> > namespace but failing to find it. If that's the case, then it sounds
>>> > like
>>> > this secondary might not be very healthy. Can you resync it from the
>>> > master? That would actually defrag your data similar to a compaction
>>> > anyway.
>>> The secondary is healthy.
>>> as i said, i only want to test the compact command, and noticed that
>>> if by mistake i run it on a collection that doesn't exists (i
>>> copy/paste the example in the doc without checking), then the entire
>>> node will switch to recorery mode forever (until i restart the
>>> process).
>>> This is a simple bug to reproduce:
>>> - create a replica set on 3 nodes
>>> - log on the secondary with mongo cli
>>> - execute db.mycollection.runCommand( "compact")
>>> > Adam
>>> > On Saturday, May 12, 2012 1:51:23 AM UTC+1, sebest wrote:
>>> >> this is on ubuntu precise pangolin with mongodb 2.0.4
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> > Groups
>>> > "mongodb-user" group.
>>> > To view this discussion on the web visit
>>> > https://groups.google.com/d/msg/mongodb-user/-/uBlwJTlZP6oJ.
>>> > To post to this group, send email to mongodb-user@googlegroups.com.
>>> > To unsubscribe from this group, send email to
>>> > mongodb-user+unsubscribe@googlegroups.com.
>>> > For more options, visit this group at
>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> --
>> You received this message because you are subscribed to the Google
>> Groups "mongodb-user" group.
>> To post to this group, send email to mongodb-user@googlegroups.com
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe@googlegroups.com
>> See also the IRC channel -- freenode.net#mongodb
Sorry Sebastien, I meant to post that very issue here when I found it the other day, but an urgent issue distracted me.
I also tested on 2.1.1 to see if this was still an issue, and found that it no longer happens. My intention was to post to the server issue and then update this thread, which you have now reminded me to do.
I have updated the thread with the results, assigned the issue to myself, and I intend to do a bit more testing and poking here, see where the fix to the issue lies and if it is possible to backport.
> On Tue, May 15, 2012 at 2:42 AM, Sebastien Estienne > <sebastien.estie...@gmail.com> wrote: > > On Tue, May 15, 2012 at 1:47 AM, Adam C <ad...@10gen.com> wrote: > >> Quick question - does test.mycollection exist?
> > No, it does not.
> >> Granted, if the compaction fails it should be more graceful, and the > fact > >> that it stays in recovering does look like a bug (I will look to > reproduce
> > That's my point.
> >> and file one tomorrow), but if you run compact on a collection that > actually > >> exists, does it succeed?
> > Yes, but i must use 'force' to compact from the primary.
> >> Adam.
> >> On Monday, May 14, 2012 10:10:23 PM UTC+1, sebest wrote:
> >>> Hi Adam
> >>> On Mon, May 14, 2012 at 1:26 PM, Adam C <ad...@10gen.com> wrote: > >>> > If you run the compact command on a replica set secondary, the > secondary > >>> > will automatically demote itself to a "recovery" state until the > >>> > compaction > >>> > is complete. Since your compaction is failing, I think that is why > it > >>> > never > >>> > comes back out of the state.
> >>> Yes, but it looks like a bug?
> >>> > Why are you running a compact on the secondary, is there something > >>> > wrong?
> >>> I wanted to test the compact command on a test replica set of 3 nodes.
> >>> When i ran the command on the master i have this output: > >>> PRIMARY> db.test.runCommand( "compact") > >>> { > >>> "errmsg" : "will not run compact on an active replica set > primary > >>> as > >>> this is a slow blocking operation. use force:true to force", > >>> "ok" : 0 > >>> }
> >>> That's why i ran it on a secondary, as the error message suggest not > >>> to run this command on a primary
> >>> > The assertion you are seeing suggests that it is expecting a > particular > >>> > namespace but failing to find it. If that's the case, then it > sounds > >>> > like > >>> > this secondary might not be very healthy. Can you resync it from > the > >>> > master? That would actually defrag your data similar to a > compaction > >>> > anyway.
> >>> The secondary is healthy.
> >>> as i said, i only want to test the compact command, and noticed that > >>> if by mistake i run it on a collection that doesn't exists (i > >>> copy/paste the example in the doc without checking), then the entire > >>> node will switch to recorery mode forever (until i restart the > >>> process).
> >>> This is a simple bug to reproduce: > >>> - create a replica set on 3 nodes > >>> - log on the secondary with mongo cli > >>> - execute db.mycollection.runCommand( "compact")
> >>> > Adam
> >>> > On Saturday, May 12, 2012 1:51:23 AM UTC+1, sebest wrote:
> >>> >> this is on ubuntu precise pangolin with mongodb 2.0.4
> >>> > -- > >>> > You received this message because you are subscribed to the Google > >>> > Groups > >>> > "mongodb-user" group. > >>> > To view this discussion on the web visit > >>> > https://groups.google.com/d/msg/mongodb-user/-/uBlwJTlZP6oJ.
> >>> > To post to this group, send email to mongodb-user@googlegroups.com. > >>> > To unsubscribe from this group, send email to > >>> > mongodb-user+unsubscribe@googlegroups.com. > >>> > For more options, visit this group at > >>> > http://groups.google.com/group/mongodb-user?hl=en.
> >> -- > >> You received this message because you are subscribed to the Google > >> Groups "mongodb-user" group. > >> To post to this group, send email to mongodb-user@googlegroups.com > >> To unsubscribe from this group, send email to > >> mongodb-user+unsubscribe@googlegroups.com > >> See also the IRC channel -- freenode.net#mongodb
One further update here, for those that don't visit Jira links:
I confirmed as still happening in 2.0.5, however, subsequently running a compact on an existing collection does return the secondary to its normal status. So, as a workaround, just run compact on an actual collection to resolve the issue.
On Thursday, May 17, 2012 1:14:44 PM UTC+1, Adam C wrote:
> Sorry Sebastien, I meant to post that very issue here when I found it the > other day, but an urgent issue distracted me.
> I also tested on 2.1.1 to see if this was still an issue, and found that > it no longer happens. My intention was to post to the server issue and > then update this thread, which you have now reminded me to do.
> I have updated the thread with the results, assigned the issue to myself, > and I intend to do a bit more testing and poking here, see where the fix to > the issue lies and if it is possible to backport.
> Adam
> On Thursday, May 17, 2012 12:54:10 PM UTC+1, sebest wrote:
>> On Tue, May 15, 2012 at 2:42 AM, Sebastien Estienne >> <sebastien.estie...@gmail.com> wrote: >> > On Tue, May 15, 2012 at 1:47 AM, Adam C <ad...@10gen.com> wrote: >> >> Quick question - does test.mycollection exist?
>> > No, it does not.
>> >> Granted, if the compaction fails it should be more graceful, and the >> fact >> >> that it stays in recovering does look like a bug (I will look to >> reproduce
>> > That's my point.
>> >> and file one tomorrow), but if you run compact on a collection that >> actually >> >> exists, does it succeed?
>> > Yes, but i must use 'force' to compact from the primary.
>> >> Adam.
>> >> On Monday, May 14, 2012 10:10:23 PM UTC+1, sebest wrote:
>> >>> Hi Adam
>> >>> On Mon, May 14, 2012 at 1:26 PM, Adam C <ad...@10gen.com> wrote: >> >>> > If you run the compact command on a replica set secondary, the >> secondary >> >>> > will automatically demote itself to a "recovery" state until the >> >>> > compaction >> >>> > is complete. Since your compaction is failing, I think that is why >> it >> >>> > never >> >>> > comes back out of the state.
>> >>> Yes, but it looks like a bug?
>> >>> > Why are you running a compact on the secondary, is there something >> >>> > wrong?
>> >>> I wanted to test the compact command on a test replica set of 3 >> nodes.
>> >>> When i ran the command on the master i have this output: >> >>> PRIMARY> db.test.runCommand( "compact") >> >>> { >> >>> "errmsg" : "will not run compact on an active replica set >> primary >> >>> as >> >>> this is a slow blocking operation. use force:true to force", >> >>> "ok" : 0 >> >>> }
>> >>> That's why i ran it on a secondary, as the error message suggest not >> >>> to run this command on a primary
>> >>> > The assertion you are seeing suggests that it is expecting a >> particular >> >>> > namespace but failing to find it. If that's the case, then it >> sounds >> >>> > like >> >>> > this secondary might not be very healthy. Can you resync it from >> the >> >>> > master? That would actually defrag your data similar to a >> compaction >> >>> > anyway.
>> >>> The secondary is healthy.
>> >>> as i said, i only want to test the compact command, and noticed that >> >>> if by mistake i run it on a collection that doesn't exists (i >> >>> copy/paste the example in the doc without checking), then the entire >> >>> node will switch to recorery mode forever (until i restart the >> >>> process).
>> >>> This is a simple bug to reproduce: >> >>> - create a replica set on 3 nodes >> >>> - log on the secondary with mongo cli >> >>> - execute db.mycollection.runCommand( "compact")
>> >>> > Adam
>> >>> > On Saturday, May 12, 2012 1:51:23 AM UTC+1, sebest wrote:
>> >>> >> this is on ubuntu precise pangolin with mongodb 2.0.4
>> >>> > -- >> >>> > You received this message because you are subscribed to the Google >> >>> > Groups >> >>> > "mongodb-user" group. >> >>> > To view this discussion on the web visit >> >>> > https://groups.google.com/d/msg/mongodb-user/-/uBlwJTlZP6oJ.
>> >>> > To post to this group, send email to mongodb-user@googlegroups.com.
>> >>> > To unsubscribe from this group, send email to >> >>> > mongodb-user+unsubscribe@googlegroups.com. >> >>> > For more options, visit this group at >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups "mongodb-user" group. >> >> To post to this group, send email to mongodb-user@googlegroups.com >> >> To unsubscribe from this group, send email to >> >> mongodb-user+unsubscribe@googlegroups.com >> >> See also the IRC channel -- freenode.net#mongodb
