Error configuring replicaset for Ops Manager DB

[Mark Falcione]

Hello. I'm trying to configure a 3 instance replicaset for the Ops Manager application database. The 3 databases are in separate VMs and created successfully, I installed manually using the tarball. I can connect to each through the mongo shell. In db1 I created a root administrator so I could run rs.initiate. The initiate command looks like this:

rs.initiate(
  {
    _id : "opsMgrRs",
    members: [
      { _id : 0, host : "testmongoops1.db.com:3000" },
      { _id : 1, host : "testmongoops2.db.com:3000" },
      { _id : 2, host : "testmongoops3.db.com:3000" }
    ]
  }
)

The settings in the config files look like this:

systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongodb1/mongod.log

# Where and how to store data.
storage:
  dbPath: /data/mongodb/mongodb1
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# how the process runs
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile
  timeZoneInfo: /usr/share/zoneinfo

# network interfaces
net:
  port: 3000
  bindIp: 127.0.0.1 testmongoops1.qsroute66.com testmongoops2.qsroute66.com testmongoops3.qsroute66.
com

security:
#    authorization: "enabled"
    keyFile: "/etc/mongodb/opsMgrDb.kf"

replication:
  replSetName: "opsMgrRs"
#sharding:

All hosts are reachable and hostnames are resolvable via DNS. The first error I was getting was:

        "errmsg" : "No host described in new configuration 1 for replica set opsMgrRs maps to this node",
        "code" : 93,
        "codeName" : "InvalidReplicaSetConfig"

Then I added testmongoops1 and testmongoops1.db.com to the hosts file's 127.0.0.1 line and the above error went away and I got a new error:

        "errmsg" : "replSetInitiate quorum check failed because not all proposed set members responded affirmatively: testmongoops2.qsroute66.com:40000 failed with Error connecting to testmongoops2.qsroute66.com:40000 (10.252.216.254:40000) :: caused by :: No route to host, testmongoops3.qsroute66.com:40000 failed with Error connecting to testmongoops3.qsroute66.com:40000 (10.252.216.238:40000) :: caused by :: No route to host",
        "code" : 74,
        "codeName" : "NodeNotFound"

Some questions:

Why would I have to add the hostname to /etc/hosts when the names are resolvable via DNS?

Do I need to create the root user on instances 2 and 3?

Do I need to add all of the hostnames to all 3 host files?

Any help will be greatly appreciated.

Thanks,

Mark

[Mark Falcione]

Update on this. I got it working by disabling the linux firewall on the 3 servers which brought another error which i was able to get past by setting the bindIp=0.0.0.0. Neither one of these fixes is acceptable long term however. I'll need to figure out what needs to be allowed through the firewall but I'm not sure how to fix bindIp. I couldn't get it to work with anyone of the following formats:

bindIp: 127.0.0.1 testmongoops1.qsroute66.com,testmongoops2.qsroute66.com,testmongoops3.qsroute66.com

bindIp: 127.0.0.1 [testmongoops1.qsroute66.com,testmongoops2.qsroute66.com,testmongoops3.qsroute66.com]

bindIp: 127.0.0.1 testmongoops1.qsroute66.com testmongoops2.qsroute66.com testmongoops3.qsroute66.com

The first two formats prevented the database from even starting. The third one let the database come up but I was getting connection refused type errors after disabling the firewall. Can anyone tell me the mongodb v4.0.3 format for this property?

Also, the other questions I asked in my initial post resolved themselves once I got things working. I removed the /etc/hosts entries and once I got the replicaset up the root user replicated to the other instances.