Problem when using different mongo instances within a jvm

[Michael W.]

Hi,
I'm using the mongodb java driver 2.5.3 in a spring application. We have a mongodb replica set consisting of a mongodb master and one slave (both of version 1.8.0). Within a junit test i start many threads performing mainly some 'findAndModify' operations. To connect to this replica set each thread creates and uses its own Mongo instance.

Each Mongo instances uses the following options:
autoConnectRetry=false
connectionsPerHost=5
threadsAllowedToBlockForConnectionMultiplier=20
connectTimeout=2000
maxWaitTime=2000
socketTimeout=2000
socketKeepAlive=false
slaveOk=false
safe=true
w=1
wtimeout=0
fsync=false
replicaSet=192.168.0.141:27017,192.168.0.136:27017

... and is instantiated with: new Mongo(replicaSetAsServerAddressList, mongoOptions);

The more i increase the number of threads the higher is the probability that exceptions of the following type occur:
14.04.2011 13:30:29 com.mongodb.ReplicaSetStatus$Node update
FATAL: can't update node: 192.168.0.141:27017
java.lang.IllegalArgumentException: need a connector: admin
    at com.mongodb.DBApiLayer.<init>(DBApiLayer.java:86)
    at com.mongodb.Mongo.getDB(Mongo.java:319)
    at com.mongodb.ReplicaSetStatus$Node.update(ReplicaSetStatus.java:149)
    at com.mongodb.ReplicaSetStatus.updateAll(ReplicaSetStatus.java:314)
    at com.mongodb.ReplicaSetStatus$Updater.run(ReplicaSetStatus.java:263)

The problem is still present when i play with different MongoOptions.

When i use only one mongo instance for all threads no problems occur. When i use no replica set but only a single mongo master no problems occur even if i use different Mongo instances (each created with: new Mongo(serverAddress, MongoOptions)).

The Question is - is this expected behavior? Is it in my case not allowed to use different Mongo instances within a jvm?

Best Regards,
Michael

[Michael W.]

Update: Sometimes also the following exception occurs:

14.04.2011 14:12:42 com.mongodb.ReplicaSetStatus$Updater run
WARNING: couldn't do update pass
java.lang.NullPointerException
    at com.mongodb.ReplicaSetStatus$Updater.run(ReplicaSetStatus.java:275)

We you debug this issue the connector is null: (ReplicaSetStatus.java:275:)  _mongo.getConnector().checkMaster(true, false);

[Antoine Girbal]

Most likely it is because your client application is running out of
sockets / file descriptors.
Each mongo instance creates its own connections, and there will be
more if you have a slave.
Make sure that:
- your tests are sequential, not in parallel
- you call mongo.close() at the end of test, otherwise connections are
not closed until GC'd.

[Shi Shei]

Why do you suggest sequential tests? Also, to test real world
conditions and for higher performance it should be possible to execute
in parallel, shouldn't it?

We simply increased the limit from 1024 (default) to 4096 by
executing:
ulimit -n 4096

That to solved the problem.

[Antoine Girbal]

If you use multiple threads, you should share the single mongo
instance across all threads.
The mongo instance is designed to be used by many threads and run
operations in parallel.
If you create many instances of Mongo (not advised) and dont close
them when done, then there are many connections left open that will
not be closed until GC kicks in, which may take a while.

[Michael W.]

Thank you for your answers.

Even if i close the Mongo instances after their usage the discribed exception occur. I think there are despite of the closing too many Mongo instances present at a time. The suggestion with "ulimit" i couldn't test so far.

[Shi Shei]

Thanks Antoine for the insights.
Of course, connections need to be closed when done.
However, when the test application wants to simulate n client
applications in parallel to stress mongo, it should open n mongo
instances because in real world, each client application would have
also its own mongo instance since every client application is running
in their own environment.

So I'd say, if the mongo server hit the limit of open files, just
increment it.