Resetting Kafka Connect JDBC for the sake of debugging and testing

[Lim Kim Tee]

Hi,

I was testing kafka connect and I deployed a standalone. Each time I am doing testing loading different data, I have to change the name of kafka connect jdbc source properties file in order for the data to be ingested into kafka connect.

A similar question has been asked here but that is for Kafka Consumer.

So my question is is there a reset button such that I can re-ingest the data everytime I load a different data for the purpose of testing and debugging?

below is the properties file of the source-jdbc.properties

name=WORKER_ID # I have to change everytime in order to do debugging and testing as this is the only way to get Kafka to ingest data from MSSQL to Kafka

connector.class=io.confluent.connect.jdbc.JdbcSourceConnector

tasks.max=1

connection.url=MY_JDBC_CONNECTION_STRING

connection.user=USERNAME

connection.password=PASSWORD

table.whitelist=MY_TABLE

mode=timestamp+incrementing

timestamp.column.name=UpdateDate

incrementing.column.name=ID

topic.prefix=TOPIC_PREFIX_

Thanks.

Regards,

Kim Tee

[Ewen Cheslack-Postava]

There isn't a tool for this at the moment. Kafka Streams has a reset tool and https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling proposes adding one for the consumer. We're aware of the need and have it in our list of features that users are looking for. Doing just the offset reset is likely the first step. Doing a full cleanup is quite a bit more challenging since connectors can write to potentially any topic, so also cleaning out the data from the topics requires tracking all the topics that have been written to.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/18f643b9-d6e2-4166-a2a2-68d2ee4fdd9b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.