Debezium Docker compose file mongo replia set unreachable hostname

[Hossein Bakhtiari]

i have this docker compose file:

version: '2.2'

services:

   # genomics-app:

   #     image: genomics

   #     environment:

   #         - SPRING_PROFILES_ACTIVE=prod,swagger

   #         - SPRING_CLOUD_CONSUL_HOST=consul

   #         - SPRING_CLOUD_CONSUL_PORT=8500

   #         - SPRING_DATASOURCE_URL=jdbc:postgresql://genomics-postgresql:5432/genomics

   #         - SPRING_DATA_NEO4J_URI=bolt://genomics-neo4j:7687

   #         - SPRING_DATA_NEO4J_USERNAME=neo4j

   #         - SPRING_DATA_NEO4J_PASSWORD=qwerty

   #         - SPRING_DATA_MONGODB_URI=mongodb://genomics_user:another@mongodb-primary:27017,mongodb-secondary:27017/genomics?replicaSet=rs0

   #         # - SPRING_DATA_MONGODB_DATABASE=genomics

   #         - JHIPSTER_SLEEP=100 # gives time for the database to boot before the application

   #         - SPRING_CLOUD_STREAM_KAFKA_BINDER_BROKERS=kafka

   #         - SPRING_CLOUD_STREAM_KAFKA_BINDER_ZK_NODES=zookeeper

   #     ports:

   #         - 8080:8080

   genomics-postgresql:

       extends:

           file: postgresql.yml

           service: genomics-postgresql

   genomics-neo4j:

       extends:

           file: neo4j.yml

           service: genomics-neo4j

   

   mongodb-primary:

       hostname: mongodb-primary

       image: 'bitnami/mongodb:latest'

       ports:

           - 27017:27017

       environment:

           - MONGODB_REPLICA_SET_NAME=rs0

           - MONGODB_REPLICA_SET_MODE=primary

           - MONGODB_ROOT_PASSWORD=another_root

           - MONGODB_DATABASE=genomics

           - MONGODB_REPLICA_SET_KEY=replicasetkey123

           - MONGODB_USERNAME=genomics_user

           - MONGODB_PASSWORD=another

           - MONGODB_ADVERTISED_HOSTNAME=mongodb-primary

       volumes:

           - 'mongodb_master_data:/bitnami'

    mongodb-secondary:

       hostname: mongodb-secondary

       image: 'bitnami/mongodb:latest'

       depends_on:

           - mongodb-primary

       environment:

           - MONGODB_REPLICA_SET_NAME=rs0

           - MONGODB_REPLICA_SET_MODE=secondary

           - MONGODB_PRIMARY_HOST=mongodb-primary

           - MONGODB_PRIMARY_PORT_NUMBER=27017

           - MONGODB_PRIMARY_ROOT_PASSWORD=another_root

           - MONGODB_REPLICA_SET_KEY=replicasetkey123

           - MONGODB_USERNAME=genomics_user

           - MONGODB_PASSWORD=another

           - MONGODB_DATABASE=genomics

           - MONGODB_ADVERTISED_HOSTNAME=mongodb-secondary

    mongodb-arbiter:

       hostname: mongodb-arbiter

       image: 'bitnami/mongodb:latest'

       depends_on:

           - mongodb-primary

       environment:

           - MONGODB_REPLICA_SET_NAME=rs0

           - MONGODB_REPLICA_SET_MODE=arbiter

           - MONGODB_PRIMARY_HOST=mongodb-primary

           - MONGODB_PRIMARY_PORT_NUMBER=27017

           - MONGODB_PRIMARY_ROOT_PASSWORD=another_root

           - MONGODB_REPLICA_SET_KEY=replicasetkey123

           - MONGODB_USERNAME=genomics_user

           - MONGODB_PASSWORD=another

           - MONGODB_DATABASE=genomics

           - MONGODB_ADVERTISED_HOSTNAME=mongodb-arbiter

    debezium_connect:

       image: debezium/connect:latest

       ports:

           - 8083:8083

       environment:

           - BOOTSTRAP_SERVERS=kafka:9092

           - GROUP_ID=1

           - CONFIG_STORAGE_TOPIC=my_connect_configs

           - OFFSET_STORAGE_TOPIC=my_connect_offsets

       links:

           - kafka

           - mongodb-primary

   kafka:

       image: debezium/kafka:latest

       environment:

           KAFKA_ADVERTISED_HOST_NAME: kafka

           KAFKA_ADVERTISED_PORT: 9092

           KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

           ZOOKEEPER_CONNECT: zookeeper:2181

           KAFKA_CREATE_TOPICS: "topic-jhipster:1:1"

       ports:

           - 9092:9092

       links:

           - zookeeper

   zookeeper:

       image: debezium/zookeeper:latest

       ports:

           - 2181:2181

           - 2888:2888

           - 3888:3888

   consul:

       image: consul:0.9.3

       command: consul agent -dev -ui -client 0.0.0.0

       ports:

           - 8300:8300

           - 8500:8500

           - 8600:8600

   consul-config-loader:

       image: jhipster/consul-config-loader:v0.2.2

       volumes:

           - ./central-server-config:/config

       environment:

           - INIT_SLEEP_SECONDS=5

           - CONSUL_URL=consul

           - CONSUL_PORT=8500

       # Uncomment to load configuration into Consul from a Git repository

       # as configured in central-server-config/git2consul.json

       # Also set SPRING_CLOUD_CONSUL_CONFIG_FORMAT=files on your apps

       #    - CONFIG_MODE=git

volumes:

   mongodb_master_data:

       driver: local

# networks:

#     app-tier:

#         driver: bridge

and after starting it send this:

{

   "name": "run-connector",

   "config": {

       "connector.class" : "io.debezium.connector.mongodb.MongoDbConnector",

       "tasks.max" : "1",

       "mongodb.hosts" : "rs0/mongodb-primary:27017",

       "mongodb.name" : "genomics",

       "mongodb.user" : "genomics_user",

       "mongodb.password" : "another",

       "database.whitelist" : "inventory",

       "database.history.kafka.bootstrap.servers" : "kafka:9092"

   }

}

for debezium kafka connect.

but this error showed up in debezium_connect:

debezium_connect_1      | 2018-06-05 23:56:48,758 ERROR  MongoDB|genomics|disc  Error while reading the 'shards' collection in the 'config' database: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=UNKNOWN, servers=[{address=mongodb-primary:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSecurityException: Exception authenticating MongoCredential{mechanism=null, userName='genomics_user', source='admin', password=<hidden>, mechanismProperties={}}}, caused by {com.mongodb.MongoCommandException: Command failed with error 18: 'Authentication failed.' on server mongodb-primary:27017. The full response is { "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18, "codeName" : "AuthenticationFailed", "operationTime" : { "$timestamp" : { "t" : 1528243002, "i" : 1 } }, "$clusterTime" : { "clusterTime" : { "$timestamp" : { "t" : 1528243002, "i" : 1 } }, "signature" : { "hash" : { "$binary" : "ixuk6xkzy0HTsYTV6Sq1oGMdC0U=", "$type" : "00" }, "keyId" : { "$numberLong" : "6563606422322413569" } } } }}}]   [io.debezium.connector.mongodb.ReplicaSetDiscovery]

debezium_connect_1      | com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=UNKNOWN, servers=[{address=mongodb-primary:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSecurityException: Exception authenticating MongoCredential{mechanism=null, userName='genomics_user', source='admin', password=<hidden>, mechanismProperties={}}}, caused by {com.mongodb.MongoCommandException: Command failed with error 18: 'Authentication failed.' on server mongodb-primary:27017. The full response is { "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18, "codeName" : "AuthenticationFailed", "operationTime" : { "$timestamp" : { "t" : 1528243002, "i" : 1 } }, "$clusterTime" : { "clusterTime" : { "$timestamp" : { "t" : 1528243002, "i" : 1 } }, "signature" : { "hash" : { "$binary" : "ixuk6xkzy0HTsYTV6Sq1oGMdC0U=", "$type" : "00" }, "keyId" : { "$numberLong" : "6563606422322413569" } } } }}}]

debezium_connect_1      | at com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:377)

debezium_connect_1      | at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:104)

debezium_connect_1      | at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:75)

debezium_connect_1      | at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:71)

debezium_connect_1      | at com.mongodb.binding.ClusterBinding.getReadConnectionSource(ClusterBinding.java:63)

debezium_connect_1      | at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:402)

debezium_connect_1      | at com.mongodb.operation.ListDatabasesOperation.execute(ListDatabasesOperation.java:102)

debezium_connect_1      | at com.mongodb.operation.ListDatabasesOperation.execute(ListDatabasesOperation.java:54)

debezium_connect_1      | at com.mongodb.Mongo.execute(Mongo.java:836)

debezium_connect_1      | at com.mongodb.Mongo$2.execute(Mongo.java:823)

debezium_connect_1      | at com.mongodb.OperationIterable.iterator(OperationIterable.java:47)

debezium_connect_1      | at com.mongodb.ListDatabasesIterableImpl.iterator(ListDatabasesIterableImpl.java:57)

debezium_connect_1      | at com.mongodb.MappingIterable.iterator(MappingIterable.java:36)

debezium_connect_1      | at io.debezium.connector.mongodb.MongoUtil.contains(MongoUtil.java:181)

debezium_connect_1      | at io.debezium.connector.mongodb.MongoUtil.contains(MongoUtil.java:170)

debezium_connect_1      | at io.debezium.connector.mongodb.MongoUtil.onDatabase(MongoUtil.java:114)

debezium_connect_1      | at io.debezium.connector.mongodb.MongoUtil.onCollection(MongoUtil.java:129)

debezium_connect_1      | at io.debezium.connector.mongodb.MongoUtil.onCollectionDocuments(MongoUtil.java:148)

debezium_connect_1      | at io.debezium.connector.mongodb.ReplicaSetDiscovery.getReplicaSets(ReplicaSetDiscovery.java:67)

debezium_connect_1      | at io.debezium.connector.mongodb.ReplicaSetMonitorThread.run(ReplicaSetMonitorThread.java:63)

debezium_connect_1      | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

debezium_connect_1      | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

debezium_connect_1      | at java.lang.Thread.run(Thread.java:748)

debezium_connect_1      | 2018-06-05 23:56:48,759 INFO   MongoDB|genomics|disc  Cluster description not yet available. Waiting for 30000 ms before timing out   [org.mongodb.driver.cluster]

i don't know why problem occures,, but i'm sure other containers can see the mongodb replicaset and can authenticate without problem,

any idea?

[Jiri Pechanec]

Hi,

could you please doble check that the security settings of the user you are using to connect conforms to http://debezium.io/docs/connectors/mongodb/#setting-up-mongodb ?

Thanks

J.

[Hossein Bakhtiari]

Hi:),

Thanks, i checked authentication many times but i couldn't set it correctly. so i just disabled mongodb auth system, and it just worked.

at first thought that mongo connector can't see the mongodb in it's network, but now i know that i was wrong. but anyway i don't know why couldn't authenticate into mongodb. i checked auth information many times.

[Jiri Pechanec]

Hi,

could you please post the MongoDB logs for first start container? Also please try our tutorial to see if it works for you https://github.com/debezium/debezium-examples/tree/master/tutorial#using-mongodb

J.