Replica sets connected via SSH port forwarding failing with "no replset config has been received" on secondaries

1,176 views

Skip to first unread message

Andrew Featherstone

unread,

Jul 21, 2016, 4:19:55 PM7/21/16

to mongodb-user

Hi Everyone,

I'm trying to establish a replica set with three nodes (one of which is an arbiter). For technical reasons the nodes must access each other using SSH tunnelling. I am reasonably confident this is configured correctly as on all the mongodb host's I am able to connect to the other nodes using mongo by providing the relevant --host and --port parameters. When I initiate the replica set on what I'd like to be the primary, the logs show the "initiator" successfully connecting to the other members:

REPL [ReplicationExecutor] transition to RECOVERING

REPL [ReplicationExecutor] transition to SECONDARY

REPL [ReplicationExecutor] Member 10.x.x.2:27017 is now in state STARTUP

REPL [ReplicationExecutor] Member 10.x.x.3:27017 is now in state STARTUP

REPL [ReplicationExecutor] Starting an election, since we've seen no PRIMARY in the past 10000ms

REPL [ReplicationExecutor] conducting a dry run election to see if we could be elected

However, the other members refuse to vote as they don't have any configuration for the replica set:

REPL [ReplicationExecutor] VoteRequester: Got error processing response with status: BadValue: Unexpected field info in ReplSetRequestVotes, resp:{ info: "run rs.initiate(...) if not yet done for the set", ok: 0.0, errmsg: "no replset config has been received", code: 94 }

REPL [ReplicationExecutor] VoteRequester: Got no vote from 10.x.x.2:27017 because: , resp:{ info: "run rs.initiate(...) if not yet done for the set", ok: 0.0, errmsg: "no replset config has been received", code: 94 }

REPL [ReplicationExecutor] VoteRequester: Got no vote from 10.x.x.3:27017 because: , resp:{ info: "run rs.initiate(...) if not yet done for the set", ok: 0.0, errmsg: "no replset config has been received", code: 94 }

REPL [ReplicationExecutor] not running for primary, we received insufficient votes

This "dry run election" process repeats every electionTimeoutMillis (set to the default of 10s).

The other piece of information that might be relevant is that the intiator of the replica set appears to never receive any heartbeats from the other nodes.

I'm not really sure what's cause and what's effect here. Does the initiator only send config on receipt of a heartbeat from a node? What can be done to get "replset config" to be send to the other nodes? Has anyone else successfully managed to create a replica set using SSH tunnelling?

Thanks in advance,

Andrew

Kevin Adistambha

unread,

Jul 25, 2016, 1:12:34 AM7/25/16

to mongodb-user

Hi Andrew,

I’m trying to establish a replica set with three nodes (one of which is an arbiter). For technical reasons the nodes must access each other using SSH tunnelling. I am reasonably confident this is configured correctly as on all the mongodb host’s I am able to connect to the other nodes using mongo by providing the relevant —host and —port parameters. When I initiate the replica set on what I’d like to be the primary, the logs show the “initiator” successfully connecting to the other members:

I’m not really sure what’s cause and what’s effect here. Does the initiator only send config on receipt of a heartbeat from a node? What can be done to get “replset config” to be send to the other nodes? Has anyone else successfully managed to create a replica set using SSH tunnelling?

I managed to create a localhost-based replica set using SSH tunnels by first starting all the mongod servers and set up the SSH tunnels, then initiated the replica set using a configuration where the members’ addresses are using the SSH tunnel’s endpoint (in may case, it was localhost). So it is possible to create a replica set using SSH tunneling.

However, this is not a supported setup if you’re looking for transport-level security. This is because if you setup all tunnel endpoints on a single machine, then this machine is a single point of failure. If this machine went offline, then all tunnels are inaccessible, and the replica set could not failover as designed. This complicates your deployment, and will compromise the operation of the replica set’s high availability functions.

A better approach for transport-level security is to use MongoDB’s built-in transport encryption features, which utilizes TLS/SSL for transport level encryption, and also identity verification using SSL certificates. These security functionalities are baked into MongoDB and are more robust to failure scenarios compared to using SSH tunnels.

For more information, please see:

Best regards,
Kevin

Reply all

Reply to author

Forward

0 new messages