Using debezium for non log-based cdc

[Nathan Smit]

Hi there,

I wanted to ask about a use case we have in our organisation and whether this could be something Debezium might cater for in the future.

We've deployed Debezium on a few databases and our plan is to replace our existing ingestion processes with it over time.  Currently we use Streamsets along with a trigger-based solution to perform change data capture on most of our databases.

Something that would be useful would be if it were possible to configure Debezium to be able to to use a LAST_UPDATE_TIME or some similar incrementing column instead of log-based change data capture.  This would allow us to transition to using Debezium in our existing environments without having to make changes on the source system.

In other words, it would be great to have something that works similar to Apache Nifi's QueryDatabaseTable where you can specify a column that is incremented. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.17.0/org.apache.nifi.processors.standard.QueryDatabaseTable/index.html .

It'd be useful to be able to have this as a potential config for the various connectors (or maybe it'd be a generic standalone jdbc connector?).  I imagine the mechanism would be similar to the sort of polling done for SQL Servers CDC.  I know this goes against the principle of Debezium using log-based replication and so I wanted to find out if this was something Debezium would consider implementing?

[jiri.p...@gmail.com]

Hi,

unfortunately no. We don't have anything like that on our roadmap.

J.

[Nathan Smit]

Hey Jiri,  Thanks for the response.  I guess I'm wondering if it's something that, in principle, you'd be willing to include if e.g. someone added this as a contribution or if it kinda fundamentally goes against what you're looking to do with Debezium.

[jiri.p...@gmail.com]

Hi,

we would definitely not dedicate any cycles but the contribution idea is definitely interesting. Still it would be good to sit first and discuss what would be the differentiatior from JDBC Source connector.

J.

[Nathan Smit]

Hey Jira, are you referring here to the JDBC Source Connector for Kafka Connect?  I guess from our side 1) if you're running Debezium Server you're likely running a Kafka-less environment, 2) By using Debezium Server we have the same structure then for both our log-based and non-log-based cdc solutions which allows us to use the same downstream code for both.

I would be interested in working on something like that.  How would I go about it though?  Would I need to work on a provisional pull request?

[jiri.p...@gmail.com]

Hi,

yes that sounds logical. Still I am not convinced about the effort needed? How about trying something slightly different? Could you try to get JDBC source connector running in Debezium Server? If this would be achievable then any source connector would be usable. And in that case it would take just a simple SMT written and applied to deliver messages that are in the same format as those emitted by Debezium.

WDYT?

J.

[Nathan Smit]

Hey Jiri (sorry for the typo on your name in my last message), that sounds interesting but having some trouble visualising how that would work.  So would debezium-server have a kafka-connect-jdbc as a kinda standalone source?  I'm trying to think through what all that would entail :)

[Nathan Smit]

To add to this, I guess I'm wondering how the source-specific functionality like polling the source would be handled as I imagine Debezium Server doesn't currently do any of that heavy lifting?

[jiri.p...@gmail.com]

Debezium Server is just encapsulated Debezium Engine. So the first step might be to write a super simple app using Debezium Engine and using non-Debezium connector. IIRC we already have a test in our test suite that uses FileStreamSourceConnector - see EmbeddedEngineTest. So I'd recommend to try using this with JDBC source connector. And if it works we might think about how to do the support in DebeziumServer. It is possible it might be just enough to add JDBC Source connector to Debeziu Server's classpath and properly configure it.

J.

[George H]

Hi there,

My name is George form a recruitment consultancy in the UK.

We are recruiting for a Debezium Consultant for a remote working contract.

If you are interested or know anyone who has excellent Debezium experience please mail geo...@digisourced.com and rebecca...@recann.co.uk