collection already sharded / collection is not sharded

[Dave S]

Hi

 

We have attempted to shard a large collection in mongodb 2.4.9 across 3 replica sets (rs1, rs2, rs3). At present, all data resides on rs1.

 

We have 3 config servers running and enabled sharding using:

 

sh.enableSharding("test")

 

We then selected a shard key and sharded a collection:

 

sh.shardCollection("test.fs.chunks", { files_id : 1 , n : 1 } )  

 

After that we added our additional shards:

 

sh.addShard( "rs2/mongo2:27017" )

sh.addShard( "rs3/mongo3:27017" )

 

 

However - after 4 days, all data still resides on rs1. Looking at the configuration, the database we are sharding is listed as "partitioned = true":

 

 {  "_id" : "test",  "partitioned" : true,  "primary" : "rs1" }

 

However, when we execute db.fs.chunks.getShardDistribution() we are presented with an error stating that the collection is not sharded:

 

mongos> db.fs.chunks.getShardDistribution()
Collection test.fs.chunks is not sharded.

We then tried to re-execute the shardCollection command and receive an error stating that it is already sharded:

 

mongos> sh.shardCollection("test.fs.chunks", { files_id : 1 , n : 1 } )

 

        "code" : 13449,

        "ok" : 0,

        "errmsg" : "exception: collection test.fs.chunks already sharded with 33463 chunks"

 

 

All 3 config servers are operational. The mongos logs contain a series of balancer distributed lock acquired / unlocked messages but nothing else noteworthy.

 

Does anyone have any advice on how we can troubleshoot this further?

 

Thanks

Dave

[Asya Kamsky]

As I said in SO comment - it's likely that something happened when you
were sharding the collection the first time and contents of
config.collections and config.chunks may not be in a consistent state
(there is already a server ticket to improve things so that this
doesn't happen).

Can you confirm what the entries for collection and chunks are for
this namespace?

Asya

> --
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
>
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Dave S]

Thanks Asya - I've responded on SO. Just wondering if you could point us

in the right direction to 'undo' this inconsistent state we are in, so we can

start afresh with the sharding?

[Asya Kamsky]

I would recommend keeping discussion here - SO is a terrible venue for
discussing complex issues, it's really for simple Q&A where there is
one Q and one A :)

The simplest solution is to remove the entries from "chunks"
collection for the ns that didn't get properly sharded. Every
collection that's sharded must have an entry in config.collections and
also corresponding entries in config.chunks - in your case it looks
like chunks are there but not collections entry. If you run 'use
config' and then 'db.chunks.remove({ns:"dbname.collectionname"})' <-
substituting the appropriate namespace here, it will clean up the
artifact on unsuccessful previous attempt.

Needless to say I would recommend that you back up the entire config
DB first, as well as dumping out chunks collections separately just to
be safe in case something goes wrong while attempting the cleanup.

Asya

[Dave S]

@Asya Kamsky - Thanks we will give this a shot over the weekend.

 

Am I right in assuming this will only remove entries from the config database and thus we are not at risk of removing actual data? I will take snapshots of all servers beforehand (we are using journaling so from my understanding this should suffice as a temporary backup)

 

If it doesn't work, can we simply spin up 3 brand new config servers with new databases and a new mongos instance referencing them - then start the sharding process over again from scratch? Is there anything retained within the replicaset itself that may cause an issue by approaching it this way?

[Asya Kamsky]

Dave,

While of course it's always a good idea to have backups of anything you will be changing, changing data in config servers in this case will not in any way affect your actual data.

In fact, I would recommend shutting down your shards while you do this.

If you end up using new config db then the only thing you would need to do is restart all of your mongos and mongod processes...

Asya

P.S. let us know how it all went...