Getting shard error from mongos: could not initialize cursor across all shards because : can't find shard for: og-mongo01:27017

[Jorge Escobar]

Hi all,

I'm getting these errors on the log from a mongos I'm trying to initialize:

Wed Mar  7 09:52:44 [conn3] DBException in process: could not initialize cursor across all shards because : can't find shard for: og-mongo01:27017 @ ogset1/og-mongo01,og-mongo03:27017,og-mongo02:27017

You can see a more complete log here: http://pastebin.com/YppysTBa

I also get this on the mongo shell:

mongos> db.cloud_cart_log.find()

error: {

        "$err" : "could not initialize cursor across all shards because : can't find shard for: og-mongo01:27017 @ ogset1/og-mongo01,og-mongo03:27017,og-mongo02:27017",

        "code" : 14827

}

This is from a mongos instance that's running on an app server outside of the cluster, which I've initialized as all other apps I have:

sudo mongos --fork --logpath /var/log/mongo/mongod.log --configdb og-mongo01,og-mongo02,og-mongo03

I have a staging mongos that works perfectly fine with the same parameters. This is a production mongos.

I've made sure that the firewall rules on all the servers have proper permissions.

The other weird thing is that I have two app servers that are within the cluster environment and they work fine:

mongos> db.cloud_cart_log.find()

{ "_id" : ObjectId("4f484fb1c9eafb2c72000001"), "_types" : [ "CloudCartLog" ], "product_id" : 4761, "created" : ISODate("2012-02-24T22:04:17.380Z"), "offer_id" : 62, "session_id" : "67525f8ca4772569c35f326c274cad70.433713.1329256195", "every_period" : 0, "every" : 0, "_cls" : "CloudCartLog", "merchant_id" : 2, "product_attribute_id" : 0, "quantity" : 1 }

{ "_id" : ObjectId("4f485791c9eafb2c75000001"), "_types" : [ "CloudCartLog" ], "product_id" : 1674, "created" : ISODate("2012-02-24T22:08:34.769Z"), "offer_id" : 62, "session_id" : "67525f8ca4772569c35f326c274cad70.433713.1329256195", "every_period" : 3, "every" : 4, "_cls" : "CloudCartLog", "merchant_id" : 2, "product_attribute_id" : 0, "quantity" : 1 }

{ "_id" : ObjectId("4f511cfaf3a9034116000001"), "_types" : [ "CloudCartLog" ], "product_id" : 344804, "created" : ISODate("2012-03-02T14:15:19.259Z"), "offer_id" : 77, "session_id" : "7221f11093853fb9d089992fd2bdba8d.549068.1329511159", "every_period" : 0, "every" : 0, "_cls" : "CloudCartLog", "merchant_id" : 25, "product_attribute_id" : 0, "quantity" : 1 }

{ "_id" : ObjectId("4f55a463f3a90354c8000001"), "session_id" : "7221f11093853fb9d089992fd2bdba8d.514796.1330995192", "_types" : [ "CloudCartLog" ], "product_id" : 360258, "created" : ISODate("2012-03-06T00:09:44.049Z"), "offer_id" : 77, "every_period" : 3, "every" : 6, "_cls" : "CloudCartLog", "merchant_id" : 25, "product_attribute_id" : 0, "quantity" : 1 }

Any ideas where to look?

[Scott Hernandez]

Check to make sure from the failing mongos instance that you can
connect to the shards in question.

From the shell, run "mongo --host
ogset1/og-mongo01,og-mongo03,og-mongo02" and make sure you connect.

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/mreDllBoKaQJ.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

[Jorge Escobar]

Hi Scott,

Thanks for replying. That seems to work!

(v2_ordergroove-py27)[root@og-app02 v2]# mongo --host ogset1/og-mongo01,og-mongo03,og-mongo02

MongoDB shell version: 2.0.3

connecting to: ogset1/og-mongo01,og-mongo03,og-mongo02:27017/test

Wed Mar  7 10:36:23 starting new replica set monitor for replica set ogset1 with seed of og-mongo01,og-mongo03,og-mongo02:27017

Wed Mar  7 10:36:23 successfully connected to seed og-mongo01 for replica set ogset1

Wed Mar  7 10:36:23 changing hosts to { 0: "og-mongo01:27017", 1: "og-mongo03:27017", 2: "og-mongo02:27017" } from ogset1/

Wed Mar  7 10:36:23 trying to add new host og-mongo01:27017 to replica set ogset1

Wed Mar  7 10:36:23 successfully connected to new host og-mongo01:27017 in replica set ogset1

Wed Mar  7 10:36:23 trying to add new host og-mongo02:27017 to replica set ogset1

Wed Mar  7 10:36:23 successfully connected to new host og-mongo02:27017 in replica set ogset1

Wed Mar  7 10:36:23 trying to add new host og-mongo03:27017 to replica set ogset1

Wed Mar  7 10:36:23 successfully connected to new host og-mongo03:27017 in replica set ogset1

Wed Mar  7 10:36:23 replica set monitor for replica set ogset1 started, address is ogset1/og-mongo01:27017,og-mongo02:27017,og-mongo03:27017

Wed Mar  7 10:36:23 [ReplicaSetMonitorWatcher] starting

PRIMARY> use mongog

switched to db mongog

PRIMARY> db.cloud_cart.find()

{ "_id" : ObjectId("4f484fb1c9eafb2c72000000"), "_types" : [ "CloudCart" ], "product_id" : 4761, "created" : ISODate("2012-02-24T22:04:17.380Z"), "offer_id" : 62, "session_id" : "67525f8ca4772569c35f326c274cad70.433713.1329256195", "every_period" : 0, "every" : 0, "_cls" : "CloudCart", "merchant_id" : 2, "product_attribute_id" : 0, "quantity" : 1 }

{ "_id" : ObjectId("4f485791c9eafb2c75000000"), "_types" : [ "CloudCart" ], "product_id" : 1674, "created" : ISODate("2012-02-24T22:08:34.768Z"), "offer_id" : 62, "session_id" : "67525f8ca4772569c35f326c274cad70.433713.1329256195", "every_period" : 3, "every" : 4, "_cls" : "CloudCart", "merchant_id" : 2, "product_attribute_id" : 0, "quantity" : 1 }

{ "_id" : ObjectId("4f511cfaf3a9034116000000"), "_types" : [ "CloudCart" ], "product_id" : 344804, "created" : ISODate("2012-03-02T14:15:19.259Z"), "offer_id" : 77, "session_id" : "7221f11093853fb9d089992fd2bdba8d.549068.1329511159", "every_period" : 0, "every" : 0, "_cls" : "CloudCart", "merchant_id" : 25, "product_attribute_id" : 0, "quantity" : 1 }

{ "_id" : ObjectId("4f55a463f3a90354c8000000"), "session_id" : "7221f11093853fb9d089992fd2bdba8d.514796.1330995192", "_types" : [ "CloudCart" ], "product_id" : 360258, "created" : ISODate("2012-03-06T00:09:44.049Z"), "offer_id" : 77, "every_period" : 3, "every" : 6, "_cls" : "CloudCart", "merchant_id" : 25, "product_attribute_id" : 0, "quantity" : 1 }

Obviously I want to run these processes as forks, so is there anything I need to change in the way these mongos start? I'm using this (and I'm getting that error):

(v2_ordergroove-py27)[root@og-app02 v2]# mongos --fork --logpath /var/log/mongo/mongod.log --configdb og-mongo01,og-mongo02,og-mongo03

all output going to: /var/log/mongo/mongod.log

(v2_ordergroove-py27)[root@og-app02 v2]# forked process: 7132

(v2_ordergroove-py27)[root@og-app02 v2]# mongo

MongoDB shell version: 2.0.3

connecting to: test

mongos> use mongog

switched to db mongog

[Scott Hernandez]

Can you connect to mongos from the mongo javascript shell and run this:

use config
printjson(db.shards.find().toArray())

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/mongodb-user/-/uGMM1GNMPy8J.

[Jorge Escobar]

Here it comes:

(v2_ordergroove-py27)[root@og-app02 v2]# mongo

MongoDB shell version: 2.0.3

connecting to: test

mongos> use config

switched to db config

mongos> printjson(db.shards.find().toArray());

[

        {

                "_id" : "ogshard1",

                "host" : "ogset1/og-mongo01,og-mongo03:27017,og-mongo02:27017"

        },

        {

                "_id" : "ogshard2",

                "host" : "ogset2/og-mongo04,og-mongo06:27017,og-mongo05:27017"

        }

]

[Jorge Escobar]

Hi Scott,

Don't know what happened, since I replied to this, but Google Groups seems to have swallowed it.

This is the output of that:

[root@og-app01 v2]# mongo

MongoDB shell version: 2.0.3

connecting to: test

mongos> use config

switched to db config

mongos> printjson(db.shards.find().toArray())

[

        {

                "_id" : "ogshard1",

                "host" : "ogset1/og-mongo01,og-mongo03:27017,og-mongo02:27017"

        },

        {

                "_id" : "ogshard2",

                "host" : "ogset2/og-mongo04,og-mongo06:27017,og-mongo05:27017"

        }

]

I just tried shutting off the firewalls in all the mongo servers to see if it was a connectivity issue but I still get the same error:

mongos> use mongog

switched to db mongog

mongos> db.cloud_cart.find()

error: {

        "$err" : "could not initialize cursor across all shards because : can't find shard for: og-mongo01:27017 @ ogset1/og-mongo01,og-mongo03:27017,og-mongo02:27017",

        "code" : 14827

}

Why do you think this worked?

mongo --host ogset1/og-mongo01,og-mongo03,og-mongo02

Thanks for your help!

[Scott Hernandez]

Can you also run this also?

use config
printjson(db.databases.find().toArray())

Is this a test environment or production? Does this happen on more
than one mongos, or just this one?

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/mongodb-user/-/nUHzkqw__REJ.

[Jorge Escobar]

mongos> use config

switched to db config

mongos> printjson(db.databases.find().toArray())

[

        {

                "_id" : "admin",

                "partitioned" : false,

                "primary" : "config"

        },

        {

                "_id" : "mongog",

                "partitioned" : true,

                "primary" : "ogshard1"

        }

]

It's a production environment. The staging environment works perfectly fine (i.e. our staging box mongos -> staging mongo cluster).

The 6 production servers' mongos' have the same issue.

[Mathias Stearn]

Try running db.adminCommand( 'flushRouterConfig' ) and db.adminCommand('connpoolsync') on the mongos and try your query again. If that doesn't solve it, please run sh.status and include the output.

[Jorge Escobar]

mongos> use config

switched to db config

mongos> db.adminCommand( 'flushRouterConfig' );

{ "flushed" : true, "ok" : 1 }

mongos> db.adminCommand('connpoolsync');

{ "ok" : 1 }

mongos> use mongog

switched to db mongog

mongos> db.cloud_cart.find()

error: {

        "$err" : "could not initialize cursor across all shards because : can't find shard for: og-mongo01:27017 @ ogset1/og-mongo01,og-mongo03:27017,og-mongo02:27017",

        "code" : 14827

}

mongos> sh.status()

--- Sharding Status --- 

  sharding version: { "_id" : 1, "version" : 3 }

  shards:

        {  "_id" : "ogshard1",  "host" : "ogset1/og-mongo01,og-mongo03:27017,og-mongo02:27017" }

        {  "_id" : "ogshard2",  "host" : "ogset2/og-mongo04,og-mongo06:27017,og-mongo05:27017" }

  databases:

        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }

        {  "_id" : "mongog",  "partitioned" : true,  "primary" : "ogshard1" }

                mongog.cloud_cart chunks:

                                ogshard1        1

                        { "session_id" : { $minKey : 1 } } -->> { "session_id" : { $maxKey : 1 } } on : ogshard1 { "t" : 1000, "i" : 0 }

                mongog.cloud_cart_log chunks:

                                ogshard1        1

                        { "session_id" : { $minKey : 1 } } -->> { "session_id" : { $maxKey : 1 } } on : ogshard1 { "t" : 1000, "i" : 0 }

                mongog.cloud_log chunks:

                                ogshard1        1

                        { "session_id" : { $minKey : 1 } } -->> { "session_id" : { $maxKey : 1 } } on : ogshard1 { "t" : 1000, "i" : 0 }

[Jorge Escobar]

Hi Mathias and Scott,

I just noticed something. It looks I have an accidentally created "mongo" database on the cluster:

mongos> sh.status()

--- Sharding Status --- 

  sharding version: { "_id" : 1, "version" : 3 }

  shards:

        {  "_id" : "ogshard1",  "host" : "ogset1/og-mongo01,og-mongo03:27017,og-mongo02:27017" }

        {  "_id" : "ogshard2",  "host" : "ogset2/og-mongo04,og-mongo06:27017,og-mongo05:27017" }

  databases:

        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }

        {  "_id" : "mongog",  "partitioned" : true,  "primary" : "ogshard1" }

                mongog.cloud_cart chunks:

                                ogshard1        1

                        { "session_id" : { $minKey : 1 } } -->> { "session_id" : { $maxKey : 1 } } on : ogshard1 { "t" : 1000, "i" : 0 }

                mongog.cloud_cart_log chunks:

                                ogshard1        1

                        { "session_id" : { $minKey : 1 } } -->> { "session_id" : { $maxKey : 1 } } on : ogshard1 { "t" : 1000, "i" : 0 }

                mongog.cloud_log chunks:

                                ogshard1        1

                        { "session_id" : { $minKey : 1 } } -->> { "session_id" : { $maxKey : 1 } } on : ogshard1 { "t" : 1000, "i" : 0 }

        {  "_id" : "mongo",  "partitioned" : false,  "primary" : "ogshard2" }

I want to drop this database. So I went to the mongos that does work and I tried this:

mongos> use admin

switched to db admin

mongos> db.mongo.drop()

Wed Mar  7 16:50:55 uncaught exception: drop failed: {

        "assertion" : "write $cmd failed on a node: { \"errmsg\" : \"ns not found\", \"ok\" : 0 } og-mongo01:27019 ns: admin.$cmd cmd: { drop: \"mongo\" }",

        "assertionCode" : 13105,

        "errmsg" : "db assertion failure",

        "ok" : 0

}

Is this database what's creating the issue?

I'm okay nuking all the data in these mongodbs, if that is necessary, as we haven't launched any clients on production yet.

Thanks so much for your help!

[Mathias Stearn]

Is there any reason you used a different name for the shards than their replset names? I generally wouldn't suggest doing that.

[Mathias Stearn]

Before you change anything, could you run db.adminCommand('getShardMap'). Then I'd suggest rebuilding the cluster without setting the shard _id different from the replset name

[Mathias Stearn]

Also, please make sure to list the port numbers for all hosts in each shard when adding them.

[Jorge Escobar]

Hi Mathias,

Here are the results:

mongos> use config

switched to db config

mongos> db.adminCommand('getShardMap')

{

        "map" : {

                "config" : "og-mongo01:27019,og-mongo02:27019,og-mongo03:27019",

                "og-mongo01" : "ogset1/og-mongo01,og-mongo03:27017,og-mongo02:27017",

                "og-mongo02:27017" : "ogset1/og-mongo01,og-mongo03:27017,og-mongo02:27017",

                "og-mongo03:27017" : "ogset1/og-mongo01,og-mongo03:27017,og-mongo02:27017",

                "og-mongo04" : "ogset2/og-mongo04,og-mongo06:27017,og-mongo05:27017",

                "og-mongo05:27017" : "ogset2/og-mongo04,og-mongo06:27017,og-mongo05:27017",

                "og-mongo06:27017" : "ogset2/og-mongo04,og-mongo06:27017,og-mongo05:27017",

                "ogset1" : "ogset1/og-mongo01,og-mongo03:27017,og-mongo02:27017",

                "ogset1/og-mongo01,og-mongo03:27017,og-mongo02:27017" : "ogset1/og-mongo01,og-mongo03:27017,og-mongo02:27017",

                "ogset2" : "ogset2/og-mongo04,og-mongo06:27017,og-mongo05:27017",

                "ogset2/og-mongo04,og-mongo06:27017,og-mongo05:27017" : "ogset2/og-mongo04,og-mongo06:27017,og-mongo05:27017",

                "ogshard1" : "ogset1/og-mongo01,og-mongo03:27017,og-mongo02:27017",

                "ogshard2" : "ogset2/og-mongo04,og-mongo06:27017,og-mongo05:27017"

        },

        "ok" : 1

}

Let me know what you suggest I do next.

Thanks!

[Jorge Escobar]

Hi guys,

Can you give me some feedback on the command's results? It looks like it's all over the place to me.

Should I just restart the cluster again, naming the shards properly and putting the proper ports?

Thanks for your help!

[Mathias Stearn]

I think that the issue is that you didn't list the ports for og-mongo01 and og-mongo04. This means that it tries to use the shard port (27018) which is the default when you start mongod with --shardsvr. Therefore you should always list explicit port numbers in your shard connection strings.

[Jorge Escobar]

You are absolutely right. I redid the whole thing, making sure not to name the shard, so that the set and the shard had the same name, and explicitly calling out the ports in all the fork commands.

Everything is working perfectly now.

Thanks so much and looking forward to kick the tires with Mongo!