Can't connect to mongos after IP address changed

[Flash Gorman]

Working to set up my first sharded collection.  Configured my replica set:

rs.initiate({
    _id: "events",
    configsvr: true,
    members: [{_id: 0, host: "9.80.193.174:27019"}]
})
{ "ok" : 1 }

Started mongos:

mongos --configdb events/9.80.193.174:27019 --port 27030

Connected to mongos and registered my mongod process as the first shard:

mongos> sh.addShard("9.80.193.174:27017")
{ "shardAdded" : "shard0000", "ok" : 1 }

And enabled sharding:

mongos> sh.enableSharding("test")
{ "ok" : 1 }

As it turns out, that IP address I was using - 9.80.193.174 - is a dynamic IP address.  Today, I am assigned a new IP address.  (I know that is problematic but I'm only testing things out at this point.)  No problem, I restarted all processes (mongod shard, mongod config server, and mongos) and gave mongos the new IP address but it has all sorts of trouble and never seems to start listening on port 27030 which, of course, means my mongo shell can't connect to it, but that means I can't execute any commands to try and fix things.

So how do I fix mongos configuration when I can't connect to it?

For your reference, here is the output of mongos:

2016-04-09T05:55:29.098-0400 W SHARDING [main] Running a sharded cluster with fewer than 3 config servers should only be done for testing purposes and is not recommended for production.
2016-04-09T05:55:29.108-0400 I SHARDING [mongosMain] MongoS version 3.2.4 starting: pid=77634 port=27030 64-bit host=gormanm.local (--help for usage)
2016-04-09T05:55:29.109-0400 I CONTROL  [mongosMain] db version v3.2.4
2016-04-09T05:55:29.109-0400 I CONTROL  [mongosMain] git version: e2ee9ffcf9f5a94fad76802e28cc978718bb7a30
2016-04-09T05:55:29.109-0400 I CONTROL  [mongosMain] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2016-04-09T05:55:29.109-0400 I CONTROL  [mongosMain] allocator: system
2016-04-09T05:55:29.109-0400 I CONTROL  [mongosMain] modules: none
2016-04-09T05:55:29.109-0400 I CONTROL  [mongosMain] build environment:
2016-04-09T05:55:29.109-0400 I CONTROL  [mongosMain]     distarch: x86_64
2016-04-09T05:55:29.109-0400 I CONTROL  [mongosMain]     target_arch: x86_64
2016-04-09T05:55:29.109-0400 I CONTROL  [mongosMain] options: { net: { port: 27030 }, sharding: { configDB: "events/9.80.210.175:27019" } }
2016-04-09T05:55:29.109-0400 I SHARDING [mongosMain] Updating config server connection string to: events/9.80.210.175:27019
2016-04-09T05:55:29.109-0400 I NETWORK  [mongosMain] Starting new replica set monitor for events/9.80.210.175:27019
2016-04-09T05:55:29.109-0400 I NETWORK  [ReplicaSetMonitorWatcher] starting
2016-04-09T05:55:29.117-0400 I SHARDING [thread1] creating distributed lock ping thread for process gormanm.local:27030:1460195729:-633143006 (sleeping for 30000ms)
2016-04-09T05:55:34.121-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:55:34.122-0400 W NETWORK  [mongosMain] No primary detected for set events
2016-04-09T05:55:39.627-0400 W NETWORK  [replSetDistLockPinger] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:55:39.627-0400 W NETWORK  [replSetDistLockPinger] No primary detected for set events
2016-04-09T05:55:39.628-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set events
2016-04-09T05:55:39.628-0400 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set events
2016-04-09T05:55:46.633-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:55:46.633-0400 W NETWORK  [mongosMain] No primary detected for set events
2016-04-09T05:55:52.140-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:55:52.140-0400 W NETWORK  [mongosMain] No primary detected for set events
2016-04-09T05:55:52.140-0400 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set events
2016-04-09T05:55:59.149-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:55:59.149-0400 W NETWORK  [mongosMain] No primary detected for set events
2016-04-09T05:56:04.657-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:56:04.657-0400 W NETWORK  [mongosMain] No primary detected for set events
2016-04-09T05:56:04.657-0400 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set events
2016-04-09T05:56:11.664-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:56:11.664-0400 W NETWORK  [mongosMain] No primary detected for set events
2016-04-09T05:56:17.173-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:56:17.173-0400 W NETWORK  [mongosMain] No primary detected for set events
2016-04-09T05:56:17.173-0400 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set events
2016-04-09T05:56:22.682-0400 W NETWORK  [replSetDistLockPinger] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:56:22.682-0400 W NETWORK  [replSetDistLockPinger] No primary detected for set events
2016-04-09T05:56:22.682-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set events
2016-04-09T05:56:28.189-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:56:28.189-0400 W NETWORK  [mongosMain] No primary detected for set events
2016-04-09T05:56:33.693-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:56:33.693-0400 W NETWORK  [mongosMain] No primary detected for set events
2016-04-09T05:56:33.694-0400 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set events
2016-04-09T05:56:40.699-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:56:40.699-0400 W NETWORK  [mongosMain] No primary detected for set events
2016-04-09T05:56:46.206-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:56:46.206-0400 W NETWORK  [mongosMain] No primary detected for set events
2016-04-09T05:56:46.206-0400 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set events
2016-04-09T05:56:53.214-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:56:53.214-0400 W NETWORK  [mongosMain] No primary detected for set events
2016-04-09T05:56:58.720-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:56:58.720-0400 W NETWORK  [mongosMain] No primary detected for set events
2016-04-09T05:56:58.721-0400 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set events
2016-04-09T05:57:04.224-0400 W NETWORK  [replSetDistLockPinger] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:57:04.224-0400 W NETWORK  [replSetDistLockPinger] No primary detected for set events
2016-04-09T05:57:04.224-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set events
2016-04-09T05:57:09.733-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:57:09.734-0400 W NETWORK  [mongosMain] No primary detected for set events
2016-04-09T05:57:15.239-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:57:15.240-0400 W NETWORK  [mongosMain] No primary detected for set events
2016-04-09T05:57:15.240-0400 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set events
2016-04-09T05:57:22.248-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:57:22.249-0400 W NETWORK  [mongosMain] No primary detected for set events
2016-04-09T05:57:27.755-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:57:27.755-0400 W NETWORK  [mongosMain] No primary detected for set events
2016-04-09T05:57:27.755-0400 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set events
2016-04-09T05:57:34.759-0400 W NETWORK  [mongosMain] Failed to connect to 9.80.193.174:27019 after 5000 milliseconds, giving up.
2016-04-09T05:57:34.760-0400 W NETWORK  [mongosMain] No primary detected for set events

[Wan Bachtiar]

gave mongos the new IP address but it has all sorts of trouble and never seems to start listening on port 27030

Hi Flash,

The issue originates with your replica set config servers, not your mongos. The log messages show that the config servers has no primary:

[replSetDistLockPinger] Failed to connect to 9.80.193.174:27019

 after 5000 milliseconds, giving up.
[replSetDistLockPinger] No primary detected for set events
[replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set events

The config server has a record of the previous IP address as a member of the set, which could no longer be reached. You would need to update the config server replica set members to point to the new IP address.

So how do I fix mongos configuration when I can’t connect to it?

To update the replica set config server member, connect to the primary of the replica set

1. Connect via mongo shell to the primary of the replica set config server.
2. Store the current configuration of the replica set into a variable: configdb_conf = rs.conf()
3. Update the IP address of the member (In your case there is only one to update) : configdb_conf.members[0].host = "<new ip>:<port>"
4. Apply the update using rs.reconfig() : rs.reconfig(configdb_conf, {force:true})
5. Confirm the status of your replica set config servers with rs.status().

See also :

• Reconfigure a Replica Set with Unavailable Members.
• Change Hostnames in a Replica Set

Worth mentioning that for any system that will run in production and require minimal interruption, the use of hostnames over IP addresses is certainly recommended. As hostnames give you a layer of abstraction when your IP addresses change.

As you are testing a sharded cluster deployment, you may find the following useful:

• Sharded Cluster Deployment
• Sharded Cluster Diagnostics

Best regards,

Wan.

​

[Flash Gorman]

Thank you.