Is there a way to get all events from multiple mongodb collections to a single pubsub topic?

[Sudharsan K]

Hi,
My requirement is to connect Mongodb to Pubsub to capture CDC events. I'm currently creating new topic for each collection.
Is there a way to send all events to a single pubsub topic?
Say, I have some 50 collections, and I want all the events to be published to a single topic "debez.database".

This is my configuration.

``` 

debezium.sink.type=pubsub
debezium.sink.pubsub.project.id=project-id
debezium.sink.pubsub.ordering.enabled=true
debezium.source.topic.prefix=doc_stream

debezium.sink.pubsub.batch.delay.threshold.ms=100
debezium.sink.pubsub.batch.element.count.threshold=100
debezium.source.connector.class=io.debezium.connector.mongodb.MongoDbConnector
debezium.source.snapshot.mode=initial
debezium.source.poll.interval.ms=1000
debezium.source.errors.max.retries=3
debezium.source.tasks.max=100
debezium.source.mongodb.connection.string=mongohost
debezium.source.mongodb.user=user
debezium.source.mongodb.password=pwd
debezium.source.database.include.list=dbname

debezium.format.key=json
debezium.format.value=json
debezium.format.value.schemas.enable=false
quarkus.log.console.json=true
quarkus.log.file.enable=false
quarkus.log.level=INFO
debezium.source.offset.storage.file.filename=/tmp/offsets.dat 

```

Any help will be appreciated.

Thanks
Sudharsan K

[jiri.p...@gmail.com]

Hi,

Debezium Server supports SMTs. You can use arbitrary SMT (like RegexRouter) to route messages to an arbitrary destination.

Jiri

[Sudharsan K]

Hi Jiri, 
Thanks for the reply.
I tried adding 

transforms=Reroute
transforms.Reroute.type=io.debezium.transforms.ByLogicalTableRouter
transforms.Reroute.topic.regex=(.*)
transforms.Reroute.topic.replacement=all_topics

and created a pubsub topic named "all_topics", yet I'm not getting any messages in it if any event occurs. 

[Nathan Smit]

Have you tried it with the debezium prefix added?  The version I do is  on Debezium 1.9 so it has some older config items (I believe 
debezium.source.database.server.name  is now replaced by  debezium.source.topic.prefix  ), but this is how I do the config so messages are split between two topics (messages to dbz-ingestion and schemas to dbz-schema).

debezium.transforms=RerouteData,RerouteSchema
debezium.transforms.RerouteSchema.type=io.debezium.transforms.ByLogicalTableRouter
debezium.transforms.RerouteSchema.topic.regex=debezium_server
debezium.transforms.RerouteSchema.topic.replacement=dbz-schema
debezium.transforms.RerouteData.type=io.debezium.transforms.ByLogicalTableRouter
debezium.transforms.RerouteData.topic.regex=debezium_server.([^.]+)\.([^.]+)
debezium.transforms.RerouteData.topic.replacement=dbz-ingestion