Java-Driver: ReplicaSet Failover not working

[Stephan]

Hi everybody,

I found some older discussions regaring this, but nothing recent. I'm
testing the failover mechanism of the Mongodb Replicaset now. Mongodb
itself does the failover rather smoothly, the new master is elected
right away. But the JAva driver still tries to connect to the "old"
master (which is still down).
Some info:
- MongoDb replicaSet, 3 Nodes, one of them as Arbiter
- MongoDbDriver is configurerd with the two addresses of the "real"
nodes
- Failover works fine in Mongo - new Primary is elected after a couple
of seconds
- When trying to read or write from the replicaset, the driver ends up
in some endless loop repeating the following message every coulple of
seconds
=> Failover not working! Do you know, what's wrong? Did I configure
something wrong here?
2011-09-29 08:57:05,030 ERROR [STDERR] (Thread-22)
com.mongodb.MongoException$Network: can't say something
2011-09-29 08:57:05,031 ERROR [STDERR] (Thread-22) at
com.mongodb.DBTCPConnector.say(DBTCPConnector.java:159)
2011-09-29 08:57:05,031 ERROR [STDERR] (Thread-22) at
com.mongodb.DBTCPConnector.say(DBTCPConnector.java:132)
2011-09-29 08:57:05,031 ERROR [STDERR] (Thread-22) at
com.mongodb.DBApiLayer$MyCollection.update(DBApiLayer.java:343)
2011-09-29 08:57:05,031 ERROR [STDERR] (Thread-22) at
com.mongodb.DBCollection.save(DBCollection.java:641)
2011-09-29 08:57:05,031 ERROR [STDERR] (Thread-22) at
com.mongodb.DBCollection.save(DBCollection.java:608)
2011-09-29 08:57:05,031 ERROR [STDERR] (Thread-22) at
de.holidayinsider.mongo.Layer.storeNoCache(Layer.java:761)
2011-09-29 08:57:05,031 ERROR [STDERR] (Thread-22) at
de.holidayinsider.mongo.Layer.access$1(Layer.java:758)
2011-09-29 08:57:05,031 ERROR [STDERR] (Thread-22) at
de.holidayinsider.mongo.Layer$WriteCacheWorker.run(Layer.java:900)
2011-09-29 08:57:05,031 ERROR [STDERR] (Thread-22) Caused by:
java.io.IOException: couldn't connect to
[integration.holidayinsider.de/109.75.180.5:27017]
bc:java.net.ConnectException: Verbindungsaufbau abgelehnt
2011-09-29 08:57:05,031 ERROR [STDERR] (Thread-22) at
com.mongodb.DBPort._open(DBPort.java:224)
2011-09-29 08:57:05,031 ERROR [STDERR] (Thread-22) at
com.mongodb.DBPort.go(DBPort.java:101)
2011-09-29 08:57:05,032 ERROR [STDERR] (Thread-22) at
com.mongodb.DBPort.go(DBPort.java:82)
2011-09-29 08:57:05,032 ERROR [STDERR] (Thread-22) at
com.mongodb.DBPort.say(DBPort.java:77)
2011-09-29 08:57:05,032 ERROR [STDERR] (Thread-22) at
com.mongodb.DBTCPConnector.say(DBTCPConnector.java:146)
2011-09-29 08:57:05,032 ERROR [STDERR] (Thread-22) ... 7 more
2011-09-29 08:57:06,537 INFO [com.mongodb.port.localhost:27017]
(http-0.0.0.0-9090-7) connect fail to : MASTER_NODE:27017
java.net.ConnectException: Verbindungsaufbau abgelehnt
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:
327)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:
193)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:
180)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
at java.net.Socket.connect(Socket.java:546)
at com.mongodb.DBPort._open(DBPort.java:214)
at com.mongodb.DBPort.go(DBPort.java:101)
at com.mongodb.DBPort.go(DBPort.java:82)
at com.mongodb.DBPort.call(DBPort.java:72)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:202)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:303)
at com.mongodb.DBCursor._check(DBCursor.java:360)
at com.mongodb.DBCursor._hasNext(DBCursor.java:490)
at com.mongodb.DBCursor.hasNext(DBCursor.java:515)

2011-09-29 08:57:06,537 INFO [com.mongodb.port.localhost:27017]
(http-0.0.0.0-9090-7) connect fail to : integration.holidayinsider.de/
109.75.180.5:27017
java.net.ConnectException: Verbindungsaufbau abgelehnt
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:
327)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:
193)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:
180)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
at java.net.Socket.connect(Socket.java:546)
at com.mongodb.DBPort._open(DBPort.java:214)
at com.mongodb.DBPort.go(DBPort.java:101)
at com.mongodb.DBPort.go(DBPort.java:82)
at com.mongodb.DBPort.call(DBPort.java:72)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:202)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:303)
at com.mongodb.DBCursor._check(DBCursor.java:360)
at com.mongodb.DBCursor._hasNext(DBCursor.java:490)
at com.mongodb.DBCursor.hasNext(DBCursor.java:515)

2011-09-29 08:57:06,538 SEVERE [com.mongodb.port.localhost:27017]
(http-0.0.0.0-9090-7) going to sleep and retry. total sleep time
after = 6218ms this time:3200ms
2011-09-29 08:57:06,538 ERROR [com.mongodb.port.localhost:27017]
(http-0.0.0.0-9090-7) going to sleep and retry. total sleep time
after = 6218ms this time:3200ms

[Scott Hernandez]

You must construct the Mongo instance using the constructor which
takes a list for it to work in replica set mode; otherwise you are
just connecting to a single mongo instance directly.

Also, what version of the driver are you using?

Can you post your code which creates the Mongo instance?

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[Stephan]

Hi,

this is exactly, what I do. I use the Mongo-Constructor with the list
of nodes. And: it's the latest Version 2.6.5
Code:

String host[] = hosts.split(",");
String port[] = ports.split(",");
for (int i = 0; i < host.length; i++) {
int pp = 27017;
if (port.length > i) {
pp = Integer.parseInt(port[i].trim());
}
try {
cfg.addAddress(host[i].trim(), pp);
} catch (UnknownHostException e) {
HiLogger.get().error(e.getMessage(), e);
}
}
MongoOptions o = new MongoOptions();
o.autoConnectRetry = true;
o.fsync = true;
o.connectTimeout = 2500;
o.connectionsPerHost = config.getMaxConnections();
o.socketKeepAlive = true;
o.threadsAllowedToBlockForConnectionMultiplier = 5;
o.safe = false;
o.slaveOk = true;
mongo = new Mongo(config.getAdr(), o);

unfortunately it does not work... getAdr() returns a list of in this
case two nodes of the ReplicaSet (not the Arbiter).

Kind Regards,

Stephan

On 29 Sep., 14:14, Scott Hernandez <scotthernan...@gmail.com> wrote:
> You must construct the Mongo instance using the constructor which
> takes a list for it to work in replica set mode; otherwise you are
> just connecting to a single mongo instance directly.
>
> Also, what version of the driver are you using?
>

[Scott Hernandez]

You will get exceptions in your application as the driver detects the
new primary. It takes a bit of time for the servers to elect a new
primary, and then for the driver to detect which one it is.

How long are you waiting for the driver to stop throwing errors during
the failover? Also, are all the addresses in the replica set config
resolvable and available to the host the java driver is running from?
The configured hostname:port must be accessible from all clients. The
server list you are passing into Mongo is just a seed list which will
be used to get the real configuration from the replica set
configuration.

[Stephan]

Thanks for the fast answer...
I was waiting about 30 minutes, Log flooded with Exceptions.

The addresses in the configuration are ok, i can connect to each node
with the mongo-shell.

So, i think i configurerd it correctly...
:-(

[Scott Hernandez]

The errors in the logs are fine as there is a background task which
checks each member in the replicaset (so while one is down you can get
exceptions/errors about it), but you should not get exceptions in your
application. Is this just a logging issue?

[Stephan]

I whish it was - the application is not working. As I don't get
answers from Mongo, I end up with a lot of NullpointerExceptions.
Anyway: as soon as the failover switch took place, the Exceptions
should stop, right?

> ...
>
> Erfahren Sie mehr »

[Stephan]

I just tried again - The exceptions are mainly concerning the write.
Unfortunately the exception is thrown up to the application when
calling save on DBCollection
2011-09-29 16:55:04,754 ERROR [STDERR] (Thread-23)

com.mongodb.MongoException$Network: can't say something

2011-09-29 16:55:04,755 ERROR [STDERR] (Thread-23) at
com.mongodb.DBTCPConnector.say(DBTCPConnector.java:159)
2011-09-29 16:55:04,755 ERROR [STDERR] (Thread-23) at
com.mongodb.DBTCPConnector.say(DBTCPConnector.java:132)
2011-09-29 16:55:04,755 ERROR [STDERR] (Thread-23) at
com.mongodb.DBApiLayer$MyCollection.update(DBApiLayer.java:343)
2011-09-29 16:55:04,755 ERROR [STDERR] (Thread-23) at
com.mongodb.DBCollection.save(DBCollection.java:641)
2011-09-29 16:55:04,755 ERROR [STDERR] (Thread-23) at
com.mongodb.DBCollection.save(DBCollection.java:608)
2011-09-29 16:55:04,755 ERROR [STDERR] (Thread-23) at
de.holidayinsider.myApp.store(App.java:161)
2011-09-29 16:55:04,755 ERROR [STDERR] (Thread-23) Caused by:
java.io.IOException: couldn't connect to [MASTER_SERVER]
bc:java.net.ConnectException: Verbindungsaufbau abgelehnt
2011-09-29 16:55:04,755 ERROR [STDERR] (Thread-23) at
com.mongodb.DBPort._open(DBPort.java:224)
2011-09-29 16:55:04,755 ERROR [STDERR] (Thread-23) at
com.mongodb.DBPort.go(DBPort.java:101)
2011-09-29 16:55:04,756 ERROR [STDERR] (Thread-23) at
com.mongodb.DBPort.go(DBPort.java:82)
2011-09-29 16:55:04,756 ERROR [STDERR] (Thread-23) at
com.mongodb.DBPort.say(DBPort.java:77)
2011-09-29 16:55:04,756 ERROR [STDERR] (Thread-23) at
com.mongodb.DBTCPConnector.say(DBTCPConnector.java:146)
2011-09-29 16:55:04,756 ERROR [STDERR] (Thread-23) ... 7 more

> ...
>
> Erfahren Sie mehr »

[Scott Hernandez]

On Thu, Sep 29, 2011 at 10:48 AM, Stephan <sboes...@googlemail.com> wrote:
> I whish it was - the application is not working. As I don't get
> answers from Mongo, I end up with a lot of NullpointerExceptions.
> Anyway: as soon as the failover switch took place, the Exceptions
> should stop, right?

Yes.

There are many people using the java driver with replica sets and
fail-over so this sounds like an isolated case. No to figure out what
is different about your config.

Is there anything else you can tell me about your system?

Are the error logs and exceptions related to the current primary, or
the old one?

[Stephan Bösebeck]

no, I did not see any additional errors or exceptions.
Let me explain this a bit more:
We're using JBoss as application Server and created some kind of "Layer" for the access to mongo. All accesses are tunnels through that layer. Everything works fine, as long as the Master-node is available. Switching off one of the secondarys does not cause any error.
When the primary node is down, the exceptions occur as soon as there is some kind of access to mongo (both read & write cause errors) which are then passed on to the application. Then, the MongoDB-Driver seems to try to reconnect every other second, which causes again the exceptions to be put into the log. There is no sign whatsoever that would indicate, that there is _any_ try to connect to the secondary node.

As soon as the primary goes up again, the connection is reestablished and everything works like charm, as soon as the syncing took place.
We use MongoDB 2.0.0 and the "primary" node has a priority of 2 (all others of 1) so that it will become master again, as soon as it reconnects.

I hope that makes all a bit clearer…

How long does it usually take to do the failover?

Kind Regards,

Stephan

[Stephan Bösebeck]

ouch - this is embarrassing.
I found the error - it's not the Mongo-DB driver! It works like charm now.
Explanation: There was an error in the deployment script, so that the wrong configuration was taken for production and integration taking only one server into the list.

Thanks for your help! This brought me to the right idea…

Kind Regards,

Stephan

Am 29.09.2011 um 17:31 schrieb Scott Hernandez: