repairDatabase and server-side scripting (error Assertion: 10298:can't temprelease nested write lock)

[redron]

Hi,

I'm using MongoDB 1.6.5. on debian linux x64.
I faced with need to run periodically db.repairDababase() in order to
compact space after removing data. While experimenting I put this
command to external JS file and it worked just fine, but running this
task by cron is not what I actually need.

The idea is to run repairDatabase() from application, because
application "knows" when is the proper time of doing that. Application
is supposed to suspend some threads working with MongoDB, save theirs
state and after reparation is done resume them correctly.
Application uses Java driver. If I haven't mistaken running
administration commands through java interface is not supported yet.
So I've decided to put db.repairDababase() into server-side JS and run
it by eval, but I'm getting following exception(see below exception
stack and how to reproduce it).

So, is it correct to run repairDatabase() that's way?
What might be other solution for running repair (and other
administration commands) which are synchronized to java application
logic?

Thanks,
Andrey Tkach
--------------------------------------

Mon Feb 7 16:54:10 [conn1146] Assertion: 10298:can't temprelease
nested write lock
0x540c7e 0x666719 0x6d3d78 0x6d61ad 0x73071b 0x79b5b9 0x797596
0x798538 0x5fb7e5 0x60029f 0x7074ba 0x70aaf6 0x70b5e1 0x56d938
0x557f1a 0x5e8d08 0x8d4ff3 0x8e7fed 0x8d5080 0x8d54c9
/opt/mongodb-linux-x86_64-1.6.5/bin/mongod(_ZN5mongo11msgassertedEiPKc
+0x1de) [0x540c7e]
/opt/mongodb-linux-x86_64-1.6.5/bin/
mongod(_ZN5mongo13dbtempreleaseC1Ev+0x89) [0x666719]
/opt/mongodb-linux-x86_64-1.6.5/bin/
mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x348) [0x6d3d78]
/opt/mongodb-linux-x86_64-1.6.5/bin/
mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x3d) [0x6d61ad]
/opt/mongodb-linux-x86_64-1.6.5/bin/
mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x3eb) [0x73071b]
/opt/mongodb-linux-x86_64-1.6.5/bin/
mongod(_ZN5mongo17CmdRepairDatabase3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb
+0x239) [0x79b5b9]
/opt/mongodb-linux-x86_64-1.6.5/bin/
mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb
+0xa16) [0x797596]
/opt/mongodb-linux-x86_64-1.6.5/bin/
mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi
+0x798) [0x798538]
/opt/mongodb-linux-x86_64-1.6.5/bin/
mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi
+0x35) [0x5fb7e5]
/opt/mongodb-linux-x86_64-1.6.5/bin/
mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_
+0x1bbf) [0x60029f]
/opt/mongodb-linux-x86_64-1.6.5/bin/mongod [0x7074ba]
/opt/mongodb-linux-x86_64-1.6.5/bin/
mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE
+0x14d6) [0x70aaf6]
/opt/mongodb-linux-x86_64-1.6.5/bin/
mongod(_ZN5mongo14DBDirectClient4callERNS_7MessageES2_b+0x81)
[0x70b5e1]
/opt/mongodb-linux-x86_64-1.6.5/bin/
mongod(_ZN5mongo14DBClientCursor4initEv+0x148) [0x56d938]
/opt/mongodb-linux-x86_64-1.6.5/bin/
mongod(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii
+0x32a) [0x557f1a]
/opt/mongodb-linux-x86_64-1.6.5/bin/
mongod(_ZN5mongo10mongo_findEP9JSContextP8JSObjectjPlS4_+0x3d8)
[0x5e8d08]
/opt/mongodb-linux-x86_64-1.6.5/bin/mongod(js_Invoke+0xe65)
[0x8d4ff3]
/opt/mongodb-linux-x86_64-1.6.5/bin/mongod(js_Interpret+0x11433)
[0x8e7fed]
/opt/mongodb-linux-x86_64-1.6.5/bin/mongod(js_Invoke+0xef2)
[0x8d5080]
/opt/mongodb-linux-x86_64-1.6.5/bin/mongod(js_InternalInvoke+0x189)
[0x8d54c9]

--------------------------

to reproduce this exception just run this two lines from Mongo
console.

db.system.js.save({_id: "repair", value : function()
{db.repairDatabase();}});
db.eval("repair()");

[Eliot Horowitz]

You cannot do this from eval.

The correct way is

DB.runCommand( new BasicDBObject( "repairDatabase" ) );

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[redron]

Thanks, it works for my java driver that way:

CommandResult compactionResult = db.command(new
BasicDBObject("repairDatabase", 1));

Another question related to this. I have sharded configuration. If I
run this command through mongos it does repair both shards. Is it the
possibility to send shard-addressed repair command through mongos? or
should I have separate connections to each mongod I intend to repair?
(MongoDB 1.6.5)

On Feb 7, 1:56 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> You cannot do this from eval.
>
> The correct way is
>
> DB.runCommand( new BasicDBObject( "repairDatabase" ) );
>

[Scott Hernandez]

You should connect to each mongod you wish to repair, directly.

If you are using replicasets then you should make sure the node is not
primary when you do the repair.

[redron]

Appreciate the quick reply.

You've mentioned replicasets.That's the point. I'm thinking now how to
avoid using replicas because of need of DB compaction.

In application I have threads inserting data from the stream, deleting
of old data and user may also do some queries to DB. When
repairDatabase is running at some node it's ok to exclude this data
from user queries, but not likely to stop inserts or loose some
incoming data.
General question. is it possible to mark some shard as disabled (while
reparation) to prevent mongos redirecting inserts to that shard? or if
a node is going to be offline mongos will handle it automatically?

Thanks

On Feb 15, 10:36 am, Scott Hernandez <scotthernan...@gmail.com> wrote:
> You should connect to each mongod you wish to repair, directly.
>
> If you are using replicasets then you should make sure the node is not
> primary when you do the repair.
>

[Eliot Horowitz]

If you want to repair a master node in a replica set, you should step
it down, and the repair.
If you do that, mongos will handle the switch automatically