secondary stays in recovering when issuing compact on unexistant collection
413 views
Skip to first unread message
sebest
May 11, 2012, 8:51:23 PM5/11/12
to mongod...@googlegroups.com
Hello,
MongoDB shell version: 2.0.4
connecting to: test
SECONDARY> db.mycollection.runCommand( "compact")
{
"assertion" : "namespace test.mycollection does not exist",
"assertionCode" : 13660,
"errmsg" : "db assertion failure",
"ok" : 0
}
RECOVERING>
Then the secondary never leave the RECOVERING state until i restart the mongodb
in the log:
Sat May 12 02:48:43 [conn2] replSet going into maintenance mode (0 other tasks)
Sat May 12 02:48:43 [conn2] replSet RECOVERING
Sat May 12 02:48:43 [conn2] Assertion: 13660:namespace test.mycollection does not exist
0x56de53 0x574ed5 0x89c709 0x89ce47 0x7fb909 0x7fc081 0x7dc66a 0x7df249 0x776c16 0x896ed8 0x5d228a 0x7f3c87ab1e9a 0x7f3c8672f4bd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0x56de53]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0xc5) [0x574ed5]
/usr/bin/mongod(_ZN5mongo7compactERKSsRSsbRNS_14BSONObjBuilderE+0x429) [0x89c709]
/usr/bin/mongod(_ZN5mongo10CompactCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x197) [0x89ce47]
/usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x589) [0x7fb909]
/usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x241) [0x7fc081]
/usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x3a) [0x7dc66a]
/usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xe89) [0x7df249]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xdd6) [0x776c16]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x88) [0x896ed8]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x26a) [0x5d228a]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f3c87ab1e9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f3c8672f4bd]
this is on ubuntu precise pangolin with mongodb 2.0.4
Adam C
May 14, 2012, 7:26:30 AM5/14/12
to mongod...@googlegroups.com
If you run the compact command on a replica set secondary, the secondary will automatically demote itself to a "recovery" state until the compaction is complete. Since your compaction is failing, I think that is why it never comes back out of the state.
Adam
Why are you running a compact on the secondary, is there something wrong?
The assertion you are seeing suggests that it is expecting a particular namespace but failing to find it. If that's the case, then it sounds like this secondary might not be very healthy. Can you resync it from the master? That would actually defrag your data similar to a compaction anyway.
Adam
Sebastien Estienne
May 14, 2012, 5:10:23 PM5/14/12
to mongod...@googlegroups.com
Hi Adam
On Mon, May 14, 2012 at 1:26 PM, Adam C <ad...@10gen.com> wrote:
> If you run the compact command on a replica set secondary, the secondary
> will automatically demote itself to a "recovery" state until the compaction
> is complete. Since your compaction is failing, I think that is why it never
> comes back out of the state.
Yes, but it looks like a bug?
> Why are you running a compact on the secondary, is there something wrong?
I wanted to test the compact command on a test replica set of 3 nodes.
When i ran the command on the master i have this output:
PRIMARY> db.test.runCommand( "compact")
{
"errmsg" : "will not run compact on an active replica set primary as
this is a slow blocking operation. use force:true to force",
"ok" : 0
}
That's why i ran it on a secondary, as the error message suggest not
to run this command on a primary
>
> The assertion you are seeing suggests that it is expecting a particular
> namespace but failing to find it. If that's the case, then it sounds like
> this secondary might not be very healthy. Can you resync it from the
> master? That would actually defrag your data similar to a compaction
> anyway.
The secondary is healthy.
as i said, i only want to test the compact command, and noticed that
if by mistake i run it on a collection that doesn't exists (i
copy/paste the example in the doc without checking), then the entire
node will switch to recorery mode forever (until i restart the
process).
This is a simple bug to reproduce:
- create a replica set on 3 nodes
- log on the secondary with mongo cli
- execute db.mycollection.runCommand( "compact")
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/uBlwJTlZP6oJ.
>
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
On Mon, May 14, 2012 at 1:26 PM, Adam C <ad...@10gen.com> wrote:
> If you run the compact command on a replica set secondary, the secondary
> will automatically demote itself to a "recovery" state until the compaction
> is complete. Since your compaction is failing, I think that is why it never
> comes back out of the state.
> Why are you running a compact on the secondary, is there something wrong?
When i ran the command on the master i have this output:
PRIMARY> db.test.runCommand( "compact")
{
"errmsg" : "will not run compact on an active replica set primary as
this is a slow blocking operation. use force:true to force",
"ok" : 0
}
That's why i ran it on a secondary, as the error message suggest not
to run this command on a primary
>
> The assertion you are seeing suggests that it is expecting a particular
> namespace but failing to find it. If that's the case, then it sounds like
> this secondary might not be very healthy. Can you resync it from the
> master? That would actually defrag your data similar to a compaction
> anyway.
as i said, i only want to test the compact command, and noticed that
if by mistake i run it on a collection that doesn't exists (i
copy/paste the example in the doc without checking), then the entire
node will switch to recorery mode forever (until i restart the
process).
This is a simple bug to reproduce:
- create a replica set on 3 nodes
- log on the secondary with mongo cli
- execute db.mycollection.runCommand( "compact")
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/uBlwJTlZP6oJ.
>
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
Adam C
May 14, 2012, 7:47:26 PM5/14/12
to mongod...@googlegroups.com
Quick question - does test.mycollection exist?
Granted, if the compaction fails it should be more graceful, and the fact that it stays in recovering does look like a bug (I will look to reproduce and file one tomorrow), but if you run compact on a collection that actually exists, does it succeed?
Adam.
Sebastien Estienne
May 14, 2012, 8:42:24 PM5/14/12
to mongod...@googlegroups.com
On Tue, May 15, 2012 at 1:47 AM, Adam C <ad...@10gen.com> wrote:
> Quick question - does test.mycollection exist?
No, it does not.
> Quick question - does test.mycollection exist?
>
> Granted, if the compaction fails it should be more graceful, and the fact
> that it stays in recovering does look like a bug (I will look to reproduce
> and file one tomorrow), but if you run compact on a collection that actually
> exists, does it succeed?
>
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>
>> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
Sebastien Estienne
May 17, 2012, 7:54:10 AM5/17/12
to mongod...@googlegroups.com
Hi Adam,
It seems this is the same issue as https://jira.mongodb.org/browse/SERVER-5381
Sebastien Estienne
It seems this is the same issue as https://jira.mongodb.org/browse/SERVER-5381
Sebastien Estienne
Adam C
May 17, 2012, 8:14:44 AM5/17/12
to mongod...@googlegroups.com
Sorry Sebastien, I meant to post that very issue here when I found it the other day, but an urgent issue distracted me.
Adam
I also tested on 2.1.1 to see if this was still an issue, and found that it no longer happens. My intention was to post to the server issue and then update this thread, which you have now reminded me to do.
I have updated the thread with the results, assigned the issue to myself, and I intend to do a bit more testing and poking here, see where the fix to the issue lies and if it is possible to backport.
Adam
>>> > For more options, visit this group at
>>> > http://groups.google.com/group/mongodb-user?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "mongodb-user" group.
>> To post to this group, send email to mongod...@googlegroups.com
>> To unsubscribe from this group, send email to
Adam C
May 18, 2012, 10:48:48 AM5/18/12
to mongod...@googlegroups.com
One further update here, for those that don't visit Jira links:
I confirmed as still happening in 2.0.5, however, subsequently running a compact on an existing collection does return the secondary to its normal status. So, as a workaround, just run compact on an actual collection to resolve the issue.
Adam
Reply all
Reply to author
Forward
0 new messages