Arbiter is not reachable/healthy (on EC2)
Scarlett Zuo
testThreeNode:PRIMARY> rs.status()
{
"set" : "testThreeNode",
"date" : ISODate("2017-01-17T09:54:40.923Z"),
"myState" : 1,
"term" : NumberLong(13),
"heartbeatIntervalMillis" : NumberLong(2000),
"members" : [
{
"_id" : 0,
"name" : "ec2-XX-XX-XX-XX.eu-central-1.compute.amazonaws.com:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 465,
"optime" : {
"ts" : Timestamp(1484646826, 1),
"t" : NumberLong(13)
},
"optimeDate" : ISODate("2017-01-17T09:53:46Z"),
"electionTime" : Timestamp(1484646516, 1),
"electionDate" : ISODate("2017-01-17T09:48:36Z"),
"configVersion" : 178490,
"self" : true
},
{
"_id" : 1,
"name" : "replica:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 371,
"optime" : {
"ts" : Timestamp(1484646826, 1),
"t" : NumberLong(13)
},
"optimeDate" : ISODate("2017-01-17T09:53:46Z"),
"lastHeartbeat" : ISODate("2017-01-17T09:54:40.271Z"),
"lastHeartbeatRecv" : ISODate("2017-01-17T09:54:39.303Z"),
"pingMs" : NumberLong(0),
"syncingTo" : "ec2-XX-XX-XX-XX.eu-central-1.compute.amazonaws.com:27017",
"configVersion" : 178490
},
{
"_id" : 2,
"name" : "arbiter:30000",
"health" : 0,
"state" : 8,
"stateStr" : "(not reachable/healthy)",
"uptime" : 0,
"lastHeartbeat" : ISODate("2017-01-17T09:54:32.238Z"),
"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
"pingMs" : NumberLong(0),
"lastHeartbeatMessage" : "Couldn't get a connection within the time limit",
"configVersion" : -1
}
],
"ok" : 1
}
[ec2-user@mongo1 ~]$ mongo --host arbiter --port 30000
MongoDB shell version: 3.2.11
connecting to: arbiter:30000/test
2017-01-17T09:58:53.696+0000 W NETWORK [thread1] Failed to connect to XX.XX.XX.XX:30000 after 5000 milliseconds, giving up.
2017-01-17T09:58:53.696+0000 E QUERY [thread1] Error: couldn't connect to server arbiter:30000, connection attempt failed :
connect@src/mongo/shell/mongo.js:231:14
@(connect):1:6
exception: connect failed
The primary can reach the replica just fine, and the arbiter can reach both primary and replica as well. But neither primary nor replica can reach the arbiter.
I followed the MongoDB manual:
https://docs.mongodb.com/v3.2/tutorial/add-replica-set-arbiter/ and
http://www.thegeekstuff.com/2014/02/mongodb-replication
I've spent 2, 3 days on this, setting it up and reset it, but nothing seems to work. Here is a list of things that I tried:
with different ports, with /ect/hosts or with simply ec2 ips, and I've tried to start both with config file and command line.
I do have authentication set up on all nodes, and I keep a key file the same relative location on each instance.
Thanks in advance for helping!
Scarlett