server is not part of a sharded cluster or the sharding metadata is not yet initialized

[James Devine]

I have a sharded collection across two replica sets all running
mongodb 3.0.4. After the sharding migration the original replica set
did not remove the migrated documents leaving me with duplicates.

I have been running db.runCommand( {"cleanupOrphaned":
"mail.message_data"} ) on the original replica set to clean this up
which was working until yesterday when the primary for that replica
set crashed with:

2015-06-24T19:05:36.093-0600 F - [conn8055] Invalid access at
address: 0x6c65a777f884
2015-06-24T19:05:36.280-0600 F - [conn8055] Got signal: 7 (Bus error).

Now when I try to run this command I am getting 'server is not part of
a sharded cluster or the sharding metadata is not yet initialized' and
when I run db.runCommand( {"shardingState": 1} ) I am getting {
"enabled" : false, "ok" : 1 }. The daemons have been started with
shardsvr = true.

db.message_data.stats() shows both replica sets as being part of the
shard member set.

Any idea how I might go about resolving this?

[Asya Kamsky]

First, I would promote the secondary to primary and resync the
original machine from that secondary - that's the purpose of a replica
set, to give you a second copy of data to work with when there is a
problem on the first one.

Secondly, the migrations are supposed to asyncronously delete the old
copy of the data, but if you open cursors from your application with
noTimeout flag then this will *block* the deletion. You can check
db.serverStatus() while your application is running to see if you are
opening cursors this way (if so, I would recommend that you fix it
asap for several reasons, this being one, and performance issues being
other).

Third the "duplicates" are not really an issue as when you query
primary for data, it knows it's part of a sharded cluster and it
filters out these "orphan" documents that have not been cleaned up
yet. The only command that can't filter them out is a straight up
count() without a condition.

Asya

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user"
> group.
>
> For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
> ---
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAPmoJaMiSKziHc%2ByZEpti40JL4znNOO3pE3%2BsH8V_izn-92JYQ%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[James Devine]

This cluster doesn't have anything running on it, I've stopped all
access to it in favor of cleaning things up. The duplicates wouldn't
have been so much an issue except I am running out of space. 92% disk
full on these machines.

I think the bigger issue right now is the fact that the first replica
set doesn't seem to think it is a shard member anymore. I'm not sure
if it is safe to just run the enable sharding commands on the database
and collection on that replica set or if that would break things
further.

> To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJCDhgZ92dScQ-_LB%3DrCJD540Vj4fDBLyruJhw3HEQ1crw%40mail.gmail.com.

[James Devine]

What on a replica set tells it if it is part of a shard set?

[Asya Kamsky]

The sharding metadata which lives in config DB(s) has information
about what members of sharded cluster there are.

MongoS connects to config DB and then when it connects to each shard
it tells it what config servers it's using and what shard it thinks
it's connecting to.

If you brought down your mongos servers then it would explain why this
replica set since coming back up isn't aware of the sharding
configuration. Make sure all the components of sharded cluster are
up and running before checking on this again.

If you see the problem still, can you provide detailed information
about what *mongos* says it sees? sh.status() db.collection.stats()
on this sharded collection, etc.

Asya

> To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAPmoJaOhYc7s2ot_N_CGv%3DnsC3dQzqe6Ww6OX1BcawcJbckevg%40mail.gmail.com.

[James Devine]

I took down the full cluster and started again to see if maybe a reset
would fix it but now both replica sets say sharding isn't enabled.


maildumpset1:PRIMARY> db.runCommand( {"shardingState": 1} )
{ "enabled" : false, "ok" : 1 }


maildumpset2:PRIMARY> db.runCommand( {"shardingState": 1} )
{ "enabled" : false, "ok" : 1 }


mongos> sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("52bdc08bcec41b622f955288")
}
shards:
{ "_id" : "maildumpset1", "host" :
"maildumpset1/maildump1.gwtc.net:27018,maildump2.gwtc.net:27018" }
{ "_id" : "maildumpset2", "host" :
"maildumpset2/maildump5.gwtc.net:27018,maildump6.gwtc.net:27018" }
balancer:
Currently enabled: no
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
databases:
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "mail", "partitioned" : true, "primary" : "maildumpset1" }
mail.message_data
shard key: { "message_identifier" : 1 }
chunks:
maildumpset1 21583
maildumpset2 21580
too many chunks to print, use verbose if you
want to force print
{ "_id" : "system", "partitioned" : false, "primary" :
"maildumpset1" }
{ "_id" : "test", "partitioned" : false, "primary" : "maildumpset1" }


mongos> db.message_data.stats()
{
"sharded" : true,
"paddingFactorNote" : "paddingFactor is unused and
unmaintained in 3.0. It remains hard coded to 1.0 for compatibility
only.",
"userFlags" : 2,
"capped" : false,
"ns" : "mail.message_data",
"count" : 1244603433,
"numExtents" : 18323,
"size" : NumberLong("39083076010236"),
"storageSize" : NumberLong("39248852941216"),
"totalIndexSize" : 253195545344,
"indexSizes" : {
"_id_" : 48499623200,
"message_identifier_1_chunk_1" : 204695922144
},
"avgObjSize" : 31402.03134104323,
"nindexes" : 2,
"nchunks" : 43163,
"shards" : {
"maildumpset1" : {
"ns" : "mail.message_data",
"count" : 805690107,
"size" : NumberLong("25047604843804"),
"avgObjSize" : 31088,
"numExtents" : 11761,
"storageSize" : NumberLong("25204314784352"),
"lastExtentSize" : 2146426864,
"paddingFactor" : 1,
"paddingFactorNote" : "paddingFactor is unused
and unmaintained in 3.0. It remains hard coded to 1.0 for
compatibility only.",
"userFlags" : 2,
"capped" : false,
"nindexes" : 2,
"totalIndexSize" : 160857395664,
"indexSizes" : {
"_id_" : 29505066416,
"message_identifier_1_chunk_1" : 131352329248
},
"ok" : 1
},
"maildumpset2" : {
"ns" : "mail.message_data",
"count" : 438913326,
"size" : NumberLong("14035471166432"),
"avgObjSize" : 31977,
"numExtents" : 6562,
"storageSize" : NumberLong("14044538156864"),
"lastExtentSize" : 2146426864,
"paddingFactor" : 1,
"paddingFactorNote" : "paddingFactor is unused
and unmaintained in 3.0. It remains hard coded to 1.0 for
compatibility only.",
"userFlags" : 2,
"capped" : false,
"nindexes" : 2,
"totalIndexSize" : 92338149680,
"indexSizes" : {
"_id_" : 18994556784,
"message_identifier_1_chunk_1" : 73343592896
},
"ok" : 1
}
},
"ok" : 1

> To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJCitJ-%3DH286zNyqHgUmsbj7tnrwpYttH7vrGu9F3uj92w%40mail.gmail.com.

[James Devine]

Seems the fact that I had all activity shutdown was why the replica
sets weren't thinking sharding was enabled. As soon as I performed an
insert both replica sets now say sharding is enabled again.

[Asya Kamsky]

You should be going by mongos status/info in this case. They say collection is sharded then all is fine with metadata.

Asya

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAPmoJaMLc53x2TSTpuATLF%2BqQTZckiEEaGscEYWb48WA2UvP%3Dg%40mail.gmail.com.

[umiyosh umiyosh]

I have the same problem. How was this problem solved after all?

2015年7月4日土曜日 19時10分04秒 UTC+9 Asya Kamsky:

[Kevin Adistambha]

Hi

>I have the same problem. How was this problem solved after all?

Please note that you are replying to a thread that is more than a year old, and is using MongoDB 3.0.4 (the latest in the 3.0 series is currently 3.0.14)

To better understand your issue, could you please open a new thread with:

- Your MongoDB and your O/S version

- A description of your issue and what you are trying to achieve

- Relevant entries in the `mongod` or `mongos` logs

- Relevant status output (`sh.status()` or `rs.status()` if applicable)

- What you have tried to resolve the issue

Best regards,

Kevin