Re: [mongodb-user] Replica Set with Virtual IP Addresses

[Scott Hernandez]

The names in replica set configuration must be resolvable and
accessible from your application machines.

This is just a name resolution issue.

Please post an example of the names/ips and mapping if this doesn't
make sense so we can give you an example using your data.

On Thu, Sep 6, 2012 at 6:37 PM, David Taylor <davidse...@gmail.com> wrote:
> We have configured a replica set behind a firewall, using Virtual IPs to map
> hostnames, made available to our app (and Mongo Java driver), from the VIP
> names to actual Mongo server hostnames in another subnet. This protected
> subnet is where all of our replica sets are running. So our app connects via
> the VIP hostnames, configured in the mongo driver as a replica set list of
> hostnames and ports.
>
> The exceptions we are seeing are listed below. If we take the firewall out,
> and connect directly to the mongo servers, we do not see these errors. I
> believe the servers actually communicate back to the driver with their
> hostnames, when the driver makes calls to the mongo servers to determine
> things like "are you the master". I have seen (in a debugger) the driver
> receiving back subnet hostnames, not the VIP addresses. The replica set is
> configured to use all protected, subnet hostnames so they can communicate
> together.
>
> If anyone has experience with this kind of virtual IP configuration with
> replica sets and could recommend a solution, much appreciated. Thanks,
>
> Exceptions, lots of these:
>
> WARNING: Server seen down: mongodb1-prod-i2:80
>
> and also:
>
> Caused by: com.mongodb.MongoException: Rare case where master=null, probably
> all servers are down
> at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:366)
> at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:212)
> at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:305)
> at com.mongodb.DB.getCollectionNames(DB.java:284)
>
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb

[David Taylor]

We are in the process of testing some changes to our firewall rules. Basically, what we are thinking, is that we need to open up our firewall.

Here is our network topology, my apologies for the text drawing....

-------------------------------------10.100.8.x mongo subnet ---------------------

   mongo-node1:6801      mongo-node2:6801  mongo-node3:6801

                |                                |                          |

       ==========  F5-VIP ===========================

               mongodb-production:80

--------------------------------------------------------------------------------------------------

                |  initial connection to a well known name via JNDI: mongodb-production:80

                |

-------------------------------------10.100.6.x application subnet ---------------------

              JVM-1                   JVM-2           ......

  

--------------------------------------------------------------------------------------------------

            

The "mongodb-production:80 connection we never want to change. If I understand how the Mongo driver works, I can give it one address in the replica set, and it will return all nodes in the replica set as well which node is the primary. The driver then connects directly to the nodes. So we originally setup the nodes in our mongo rs collection as mongo-node1:6801, mongo-node2:6801, mongo-node3:6801, and then via the driver, connected to mongodb-production:80. So when the driver tries to connect to mongo-node1:6801 for example, it was getting blocked. We believe that is what we need to open up.

I was wondering if you could verify these observations.... I've been stepping through the debugger on the Mongo Java driver, and I see calls made between the Mongo driver and the replication set nodes. Calls made like in ConnectionStatus.UpdatableNode.update():

                res = _port.runCommand(_mongo.getDB("admin"), isMasterCmd);

If I recall correctly, the CommandResult returned contains node
addresses of the format hostname/ipaddress:port. In the case of the above
topology, isMasterCmd would return the address of the primary of the 3 nodes,
such as mongo-node1/10.100.8.11:6800. I believe the driver is using this information

to submit subsequent commands (the root of our problems). I was wondering if the IP address is ever used, or is the hostname always used

I hope I explained that well enough... thanks for your help, will respond back to the list once the firewall is reconfigured, might be Monday

--

David

[Barrie]

David,

Did you reconfigure your firewall?

Barrie

[David Taylor]

Yes, we reconfigured the firewall and the replica set now starts up without errors. Basically we had to open access to all logical DNS names to the mongo servers from the servers running the drivers, and make sure that the replica set configuration had the same DNS names

Now that its up and running, the servers are crashing when processing our requests. Is it possible that our Solaris version wasn't built correctly?

Is it recommended to run on Solaris? 

Thu Sep 13 13:55:22 [rsMgr] replSet error self not present in the repl set configuration:

Thu Sep 13 13:55:22 [rsMgr] { _id: "mongodb1", version: 7, members: [ { _id: 1, host: "dev-app6-solz2:6802" } ] }

Thu Sep 13 13:55:22 [rsMgr] replset msgReceivedNewConfig version: version: 7

Thu Sep 13 13:55:22 [rsMgr] replSet info saving a newer config version to local.system.replset

ld.so.1: mongod: fatal: relocation error: file /sharedvol-perfdev/marketlive5/services/datasvcs/version/mongodb-sunos5-x86_64-2.0.6/bin/mongod: symbol nanosleep: referenced symbol not found