Using the MySQL connector without Kafka

[Ido Shamun]

I want to use Debezium Server with MySQL connector but it requires to a Kafka server for storing the schema changes. Is it possible to disable it?

I want to use Google Pub/Sub as sink and don't have Kafka in place.

Thanks!

[Gunnar Morling]

Hi,

You cannot disable the schema history, as the connector needs to recover the schema after a restart, in order to be able to decode incoming messages.

It doesn't have to be backed by a Kafka topic necessarily, though. You could develop your own history implementation by implementing  io.debezium.relational.history.(Abstract)DatabaseHistory. Besides the Kafka one, there's a file-based one, and there's a pending PR for adding a history implementation backed by JDBC: https://github.com/debezium/debezium/pull/2543. Should you create one based on Pub/Sub, perhaps you'd be interested in adding it to the debezium-examples repo?

Hth,

--Gunnar

[Ido Shamun]

I think I replied only to you so I'm sending it again here.

I'm not sure PuSub has database capabilities as Kafka does. The file-based history sounds great. Any example or docs that I can follow?

Thanks a lot.

Ido

[Gunnar Morling]

Here's an example showing it being used with the embedded engine:

    https://debezium.io/documentation/reference/development/engine.html#_in_the_code

You can apply the same configuration when using Debezium Server.

--Gunnar

--
You received this message because you are subscribed to a topic in the Google Groups "debezium" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/debezium/_QwOhV85kHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/5611046c-0fec-45a3-9bd9-458eb1ceafc3n%40googlegroups.com.

[Ido Shamun]

Cool! Gonna try it now

[Ido Shamun]

It still expects to get the Kafka configuration.

Here's the `application.properties` file:

```

debezium.source.connector.class=io.debezium.connector.mysql.MySqlConnector
debezium.source.offset.storage.file.filename=data/offsets.dat
debezium.source.offset.flush.interval.ms=0
debezium.source.database.hostname=%hostname%
debezium.source.database.port=3306
debezium.source.database.user=%database_user%
debezium.source.database.password=%database_pass%
debezium.source.database.dbname=%database_dbname%
debezium.source.database.server.name=gateway
debezium.source.table.include.list=gateway.users
debezium.source.heartbeat.interval.ms=60000
debezium.source.include.schema.changes=false
database.history=io.debezium.relational.history.FileDatabaseHistory
database.history.file.filename=data/dbhistory.dat

debezium.transforms=Reroute
debezium.transforms.Reroute.type=io.debezium.transforms.ByLogicalTableRouter
debezium.transforms.Reroute.topic.regex=(.*)
debezium.transforms.Reroute.topic.replacement=%topic%

debezium.sink.type=pubsub

```

Logs:

```

2021-09-15 08:54:54,063 INFO  [io.deb.ser.DebeziumServer] (main) Engine executor started

2021-09-15 08:54:54,070 INFO  [org.apa.kaf.con.sto.FileOffsetBackingStore] (pool-7-thread-1) Starting FileOffsetBackingStore with file data/offsets.dat

2021-09-15 08:54:54,355 ERROR [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) The 'database.history.kafka.bootstrap.servers' value is invalid: A value is required

2021-09-15 08:54:54,356 ERROR [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) The 'database.history.kafka.topic' value is invalid: A value is required

2021-09-15 08:54:54,366 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) Stopping down connector

2021-09-15 08:54:54,370 INFO  [org.apa.kaf.con.sto.FileOffsetBackingStore] (pool-7-thread-1) Stopped FileOffsetBackingStore

```

[Gunnar Morling]

With Debezium Server, all the property keys need to be prefixed with "debezium.". Can you update your config accordingly?

--Gunnar

[Ido Shamun]

Yap, sorry. I also figured it out after I sent the message.

It seems to work. I have another error but it's unrelated.

Thanks a lot! 🙏