Always draining after running removeshard

[Feifei Jia]

Hi all,

I tried to remove a shard from exsiting cluster, following the steps listed here:

http://www.mongodb.org/display/DOCS/removeshard+command

But, after running the "removeshard" command, the status is always "draining".

mongos> db.printShardingStatus()

--- Sharding Status --- 

  sharding version: { "_id" : 1, "version" : 3 }

  shards:

{  "_id" : "r1",  "host" : "r1/host1,host2" }

{  "_id" : "r2",  "draining" : true,  "host" : "r2/host3,host4" }

{  "_id" : "r3",  "host" : "r3/host5,host6" }

  databases:

{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }

{  "_id" : "test",  "partitioned" : false,  "primary" : "r1" }

{  "_id" : "foo",  "partitioned" : true,  "primary" : "r1" }

foo.t1 chunks:

r3 12

r1 12

too many chunks to print, use verbose if you want to force print

foo.t2 chunks:

r1 22

r3 21

too many chunks to print, use verbose if you want to force print

{  "_id" : "bar",  "partitioned" : false,  "primary" : "r2" }

mongos> db.runCommand({removeshard: "r2"})

{

"msg" : "draining ongoing",

"state" : "ongoing",

"remaining" : {

"chunks" : NumberLong(0),

"dbs" : NumberLong(1)

},

"ok" : 1

}

You can see that all the chunks had been moved to r1 and r3, but the state of removing sharding stayed as "ongoing" for more than 2 hours, what should I do?

Why there are remaining "dbs" in the output of "db.runCommand({removeshard: "r2"})"? The remaining db is "bar" listed in "db.printShardingStatus()"?

Thanks in advance.

[Adam C]

You have a database still on that shard, i.e. the r2 shard is the primary for the "foo" database.  So, you need to run:

db.runCommand( { movePrimary : "foo", to : "r3" } ); 

(or r1 if you prefer, of course)

Then, re-run the removeshard command and you should get the success message.

Adam.

[Feifei Jia]

Thanks Adam C, I did exactly like you said, the r2 was removed.

I thought it should be a status like "drained", then run the "movePrimary" command, is it better to have this status?

One problem though, after I removed r2, I shutdown those servers in r2. But in the log files, there are still entries saying "socket excepiton":

ReplicaSetMonitor::_checkConnection: caught exception host3 socket exception

Why does mongo still try to connect to servers in r2?

[Scott Hernandez]

This is a known issue with the replica set health watcher thread. Can
you post the full log message so we can confirm that is the cause?

Is the sharded cluster otherwise performing well?

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/a7AgOlZ34KUJ.
>
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

[Feifei Jia]

This is the mongos log:

Thu Apr 26 23:08:17 [ReplicaSetMonitorWatcher] trying reconnect to host3

Thu Apr 26 23:08:17 [ReplicaSetMonitorWatcher] reconnect host3 failed couldn't connect to server host3

Thu Apr 26 23:08:17 [ReplicaSetMonitorWatcher] trying reconnect to host4

Thu Apr 26 23:08:17 [ReplicaSetMonitorWatcher] reconnect host4 failed couldn't connect to server host4

Thu Apr 26 23:08:18 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception host3 socket exception

Thu Apr 26 23:08:18 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception host4 socket exception

It seemed the cluster worked well, so far.

How should I solve this issue then? Restart the mongos?

> mongodb-user+unsubscribe@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

> mongodb-user+unsubscribe@googlegroups.com.

[Eliot Horowitz]

You can ignore that log message or restart mongos

>> > mongodb-user...@googlegroups.com.

>> > mongodb-user...@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/mongodb-user/-/_IgvTegpViIJ.

>
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to

> mongodb-user...@googlegroups.com.

[Feifei Jia]

OK, I've just restarted mongos to get rid of this annoying messages.

I'd like to know, the "movePrimary" command, how does data move from one shard to another shard? It works like "copydb" command? What about the speed of this action?

>> > mongodb-user+unsubscribe@googlegroups.com.

>> > mongodb-user+unsubscribe@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/_IgvTegpViIJ.
>
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to

> mongodb-user+unsubscribe@googlegroups.com.

[Adam C]

Each database has a primary shard assigned (you can see this in the sharding status output).  That means that any non-sharded data will live on the desginated primary shard by default.   For example, if you had an unsharded collection it would be there, or if you used authentication on that database, the user information would be stored there.  Once you had drained it of all chunks, all the sharded data had been moved, you can think of the movePrimary command as a single shot move for the remaining non-sharded data.  If there isn't any then this is going to be very fast, since it would be little more than a meta data operation.  I don't know the exact mechanics (I'd have to look at the code), but the speed will depend on the contents of the database and the speed of your network - as always, the easiest way to figure out speed is to test - create a non-sharded collection on your cluster, add data, and issue movePrimary to gauge speed.

Adam