Issue while connecting a spring application to a mongo replica set, running on docker

[jbo...@ekito.fr]

Hi all,

I'm struggling with deploying a spring application connected to a mongo replicaset running on docker.

The replicaset is deployed using mongodb-automation image found on https://registry.hub.docker.com/u/mcascallares/mongodb-automation/

It works pretty well with MMS and I can connect to the replicaset from my localhost :

{

        "set" : "rsname",

        "date" : ISODate("2015-01-14T14:03:55Z"),

        "myState" : 2,

        "syncingTo" : "mongod2.mongodb-automation.dev.docker:27000",

        "members" : [

                {

                        "_id" : 0,

                        "name" : "mongod1.mongodb-automation.dev.docker:27000",

                        "health" : 1,

                        "state" : 2,

                        "stateStr" : "SECONDARY",

                        "uptime" : 74557,

                        "optime" : Timestamp(1421243583, 53),

                        "optimeDate" : ISODate("2015-01-14T13:53:03Z"),

                        "self" : true

                },

                {

                        "_id" : 1,

                        "name" : "mongod2.mongodb-automation.dev.docker:27000",

                        "health" : 1,

                        "state" : 1,

                        "stateStr" : "PRIMARY",

                        "uptime" : 804,

                        "optime" : Timestamp(1421243583, 53),

                        "optimeDate" : ISODate("2015-01-14T13:53:03Z"),

                        "lastHeartbeat" : ISODate("2015-01-14T14:03:55Z"),

                        "lastHeartbeatRecv" : ISODate("2015-01-13T21:08:49Z"),

                        "pingMs" : 0,

                        "electionTime" : Timestamp(1421243479, 1),

                        "electionDate" : ISODate("2015-01-14T13:51:19Z")

                },

                {

                        "_id" : 2,

                        "name" : "mongod3.mongodb-automation.dev.docker:27000",

                        "health" : 1,

                        "state" : 2,

                        "stateStr" : "SECONDARY",

                        "uptime" : 766,

                        "optime" : Timestamp(1421243583, 53),

                        "optimeDate" : ISODate("2015-01-14T13:53:03Z"),

                        "lastHeartbeat" : ISODate("2015-01-14T14:03:55Z"),

                        "lastHeartbeatRecv" : ISODate("2015-01-13T21:08:51Z"),

                        "pingMs" : 0,

                        "syncingTo" : "mongod2.mongodb-automation.dev.docker:27000"

                }

        ],

        "ok" : 1

}

As you can see in the previous extract, each mongo server run on port 27000 in their docker vm.

For each docker container, the port 27000 is binded to 27017, 27018, 27019.

Now, I want to connect to the mongo replicaset using Spring Data and the mongo-java-driver-2.12.2.

My server configuration use a list of ServerAddr with the following parameters : 

• mongod1.mongodb-automation.dev.docker:27017
• mongod2.mongodb-automation.dev.docker:27018
• mongod3.mongodb-automation.dev.docker:27019
  

When I start the application, I get the following error : 

org.springframework.dao.InvalidDataAccessResourceUsageException: Timed out while waiting for a server that matches {serverSelectors=[ReadPreferenceServerSelector{readPreference=primary}, LatencyMinimizingServerSelector{acceptableLatencyDifference=15 ms}]} after 4 ms; nested exception is com.mongodb.MongoTimeoutException: Timed out while waiting for a server that matches {serverSelectors=[ReadPreferenceServerSelector{readPreference=primary}, LatencyMinimizingServerSelector{acceptableLatencyDifference=15 ms}]} after 4 ms
 at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:69)
 at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:1918)
 at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1801)
 at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1612)
 at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1596)
 at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:535)
 at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:526)
 at com.group3s.falcon.repository.AirportScheduleItemRepositoryImpl.findByArrivalDepartureBetweenDatesAndArrivalNumberAndDepartureNumber(AirportScheduleItemRepositoryImpl.java:72)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.executeMethodOn(RepositoryFactorySupport.java:405)
 at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:380)
 at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:344)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
 at com.sun.proxy.$Proxy66.findByArrivalDepartureBetweenDatesAndArrivalNumberAndDepartureNumber(Unknown Source)
 at com.group3s.falcon.service.PlannetSynchronisationServiceImpl.synchronise(PlannetSynchronisationServiceImpl.java:89)
 at com.group3s.falcon.job.PlannetInterfaceScheduledTask.reportCurrentTime(PlannetInterfaceScheduledTask.java:28)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
 at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)
Caused by: com.mongodb.MongoTimeoutException: Timed out while waiting for a server that matches {serverSelectors=[ReadPreferenceServerSelector{readPreference=primary}, LatencyMinimizingServerSelector{acceptableLatencyDifference=15 ms}]} after 4 ms
 at com.mongodb.BaseCluster.getServer(BaseCluster.java:87)
 at com.mongodb.DBTCPConnector.getServer(DBTCPConnector.java:654)
 at com.mongodb.DBTCPConnector.access$300(DBTCPConnector.java:39)
 at com.mongodb.DBTCPConnector$MyPort.getConnection(DBTCPConnector.java:503)
 at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:451)
 at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:286)
 at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:271)
 at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:84)
 at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:66)
 at com.mongodb.DBCursor._check(DBCursor.java:458)
 at com.mongodb.DBCursor._hasNext(DBCursor.java:546)
 at com.mongodb.DBCursor.hasNext(DBCursor.java:571)
 at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1787)
 ... 30 common frames omitted
 

If I debug the java driver, I can see that it will use the internal mongo configuration, instead of my own config :

{
        "_id" : "rsname",
        "version" : 1,
        "members" : [
                {
                        "_id" : 0,
                        "host" : "mongod1.mongodb-automation.dev.docker:27000"
                },
                {
                        "_id" : 1,
                        "host" : "mongod2.mongodb-automation.dev.docker:27000"
                },
                {
                        "_id" : 2,
                        "host" : "mongod3.mongodb-automation.dev.docker:27000"
                }
        ]
}

The servers can not be reached on port 27000 as it is the port used internally in the docker container.

Is there a driver configuration to force using the ServerAddr provided to the driver instead of the internal rs.config ?

Thanks

Some related post :

https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/320

https://groups.google.com/forum/#!topic/mongodb-user/FKjchobMmSw

[Matias Cascallares]

Hi,

List of ServerAddr is used as a seed by the driver to discover the replica set topology and nodes. The Java driver will pull the replica set configuration and that's why when you debug driver information you see:

{
        "_id" : "rsname",
        "version" : 1,
        "members" : [
                {
                        "_id" : 0,
                        "host" : "mongod1.mongodb-automation.dev.docker:27000"
                },
                {
                        "_id" : 1,
                        "host" : "mongod2.mongodb-automation.dev.docker:27000"
                },
                {
                        "_id" : 2,
                        "host" : "mongod3.mongodb-automation.dev.docker:27000"
                }
        ]
}

That behaviour cannot be changed.

Therefore the issue comes from the fact that hosts and ports from replica set configuration don't match host and ports used by the application. The application should rely on name resolution to use exactly the same hosts and ports specified in replica set configuration. In the example there is a container called 'skydns' that runs a DNS service and containers are automatically registered when they start by another container called 'skydock'. You must configure the host/container where the app is executed to use skydns container as DNS service. If your application is executed inside another container managed by the same docker process then it's automatic and your ServerAddr list should be:

• mongod1.mongodb-automation.dev.docker:27000
• mongod2.mongodb-automation.dev.docker:27000
• mongod3.mongodb-automation.dev.docker:27000

If your application runs somewhere else then you must configure that host to resolve using skydns container (check resolv.conf) and do some port-forwarding to avoid overlapping. Another approach is to use ambassador pattern for cross-host linking, check Docker documentation.

I hope this clarifies your question.

Matias