Is there a SQL Server CDC connector for Kafka Connect?

[Todd Bernhard]

Hello,

We are trying to replicate a MSSQL database into Kafka with Connect. I've seen MySQL BinLog and Postgres connectors, but haven't found one for MSSQL's CDC. Before we begin writing it ourselves, is anyone aware of an available CDC connector?

Thanks,

Todd

Platform Engineer

Athlinks

[Ben Davison]

Hi Todd,

We have just gone through the exact same scenario. We didn't find a connector at the time (there might be one now) . The way we solved it is to have Kafka connect calling a stored proc with all the needed cdc "stuff" contained in it and throw that into Kafka.

Our use case was a little different because we wanted to have "messages" and not just raw rows. E.g. An artist is updated, so we also wanted info from the artist table and the tracks table combined on one row.

Thanks,

Ben

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/90e4688a-d743-4a81-8e5c-462067e64f8e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

This email, including attachments, is private and confidential. If you have received this email in error please notify the sender and delete it from your system. Emails are not secure and may contain viruses. No liability can be accepted for viruses that might be transferred by this email or any attachment. Any unauthorised copying of this message or unauthorised distribution and publication of the information contained herein are prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.

[Ewen Cheslack-Postava]

I'm not aware of a connector implementing SQL Server CDC yet, but given the pretty awesome support SQL Server provides (https://msdn.microsoft.com/en-us/library/cc645937.aspx) I think it's probably one of the easier CDC connectors to implement.

-Ewen

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CA%2BvjBRhLgCV1vq2WFsJHGnoFKvd_E6MQUjz2OTOYBK1EB_iB%3Dg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

[Vincent Murphy]

http://www.striim.com/microsoft-sql-server-cdc-to-kafka/

Found at http://www.confluent.io/product/connectors

[Sumit Arora]

We are currently in the process of writing a SQL Server CDC connector. Development is almost complete and it should soon be deployed to our test environment. See a related post here:

https://groups.google.com/forum/#!topic/confluent-platform/MX54z7ouIqE

thanks!

[Sumit Arora]

To provide a little more detail:

We are using JDBC connector with a query to fetch data from SQL Server CDC tables and push to Kafka. Our SQL server connector then picks up the messages from Kafka topic and pushes them to a different SQL server.

[AjeethKumar]

Hello Ben,

Is it possible to poll a custom CDC stored procedure from Kafka connect. When I was reading the documents  the configs had either table names or SQL selects.

Thanks,

Ajeeth

On Saturday, June 4, 2016 at 12:00:59 PM UTC+1, Ben Davison wrote:

Hi Todd,

We have just gone through the exact same scenario. We didn't find a connector at the time (there might be one now) . The way we solved it is to have Kafka connect calling a stored proc with all the needed cdc "stuff" contained in it and throw that into Kafka.

Our use case was a little different because we wanted to have "messages" and not just raw rows. E.g. An artist is updated, so we also wanted info from the artist table and the tracks table combined on one row.

Thanks,

Ben

On Friday, 3 June 2016, Todd Bernhard <bernha...@gmail.com> wrote:

Hello,

We are trying to replicate a MSSQL database into Kafka with Connect. I've seen MySQL BinLog and Postgres connectors, but haven't found one for MSSQL's CDC. Before we begin writing it ourselves, is anyone aware of an available CDC connector?

Thanks,

Todd

Platform Engineer

Athlinks

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/90e4688a-d743-4a81-8e5c-462067e64f8e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Davison]

Hi Ajeeth,

Yup it's possible, as we just called "EXEC storedProc;" as the SQL statement.

Thanks,

Ben

On Fri, Jul 22, 2016 at 12:10 PM, AjeethKumar <ajee...@gmail.com> wrote:

Hello Ben,

Is it possible to poll a custom CDC stored procedure from Kafka connect. When I was reading the documents  the configs had either table names or SQL selects.

Thanks,

Ajeeth

On Saturday, June 4, 2016 at 12:00:59 PM UTC+1, Ben Davison wrote:

Hi Todd,

We have just gone through the exact same scenario. We didn't find a connector at the time (there might be one now) . The way we solved it is to have Kafka connect calling a stored proc with all the needed cdc "stuff" contained in it and throw that into Kafka.

Our use case was a little different because we wanted to have "messages" and not just raw rows. E.g. An artist is updated, so we also wanted info from the artist table and the tracks table combined on one row.

Thanks,

Ben

On Friday, 3 June 2016, Todd Bernhard <bernha...@gmail.com> wrote:

Hello,

We are trying to replicate a MSSQL database into Kafka with Connect. I've seen MySQL BinLog and Postgres connectors, but haven't found one for MSSQL's CDC. Before we begin writing it ourselves, is anyone aware of an available CDC connector?

Thanks,

Todd

Platform Engineer

Athlinks

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/90e4688a-d743-4a81-8e5c-462067e64f8e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

This email, including attachments, is private and confidential. If you have received this email in error please notify the sender and delete it from your system. Emails are not secure and may contain viruses. No liability can be accepted for viruses that might be transferred by this email or any attachment. Any unauthorised copying of this message or unauthorised distribution and publication of the information contained herein are prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/61a31701-a384-4cd8-9ac1-cbcb73b485d6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[pramach]

Hello Sumit, 

We are as well looking for a SQL server connector for one time bulk load and then delta CDC for around 100 different tables. I am afraid having many connector configuration will be a maintenance nightmare. I also saw your other post on monitoring. Would like to hear from you on how you guys went about these 2 things. 

Thanks

[Carlos Elói Barros]

Hi Ben,

Are you using JDBC connector to call an SP? Or a Custom Connector? Can you share you properties file where you call an sp? 

Best regards.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/90e4688a-d743-4a81-8e5c-462067e64f8e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

This email, including attachments, is private and confidential. If you have received this email in error please notify the sender and delete it from your system. Emails are not secure and may contain viruses. No liability can be accepted for viruses that might be transferred by this email or any attachment. Any unauthorised copying of this message or unauthorised distribution and publication of the information contained herein are prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/61a31701-a384-4cd8-9ac1-cbcb73b485d6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Puneet Jindal]

Hello Sumit,

Need your help on this.
My connection to Sumit is not successful now.Please let me know how we can connect. I am stuck here and in urgency otherwise I would have to fall back on Talend tool based approach.
If would be really great of all if anybody can help me here

Thanks,
Puneet
+91-783-886-0769
my email-id - puneetjin...@gmail.com
https://www.linkedin.com/in/puneet-jindal-2493a216/

[Zeeshan Anwar]

I download the project and try to build it but not able too and got following error. Can anyone help ?

FAILURE: Build failed with an exception.

* Where:

Build file '/Users/admin/Downloads/kafka-connect-mssqlserver-cdc-master/build.gradle' line: 19

* What went wrong:

An exception occurred applying plugin request [id: 'org.ajoberstar.release-opinion', version: '1.7.2']

> Failed to apply plugin [id 'org.ajoberstar.release-base']

   > Could not get unknown property 'grgit' for root project 'kafka-connect-mssqlserver-cdc' of type org.gradle.api.Project.

On Saturday, September 23, 2017 at 12:09:05 AM UTC-4, John Palomo wrote:

Came across this thread and wanted to just let you all know that I currently have beta version of a SQL Server CDC connector implemented.  It resembles the Confluent JDBC source connection but uses SQL Server specific functions to query the underlying tables and does not require  custom stored procedure/query.  I am currently in the process of writing system tests to confirm functionality and should be done with this in the near future.  Please check it and feel free to contribute or provide suggestions or feedback.

https://github.com/jpalomo/kafka-connect-mssqlserver-cdc

John 

[Zeeshan Anwar]

Hi John -

Thanks, this  is very helpful. Do you have any document on how to use it?

On Thursday, December 21, 2017 at 11:54:39 AM UTC-5, John Palomo wrote:

Zeeshan,

It looks like you downloaded the project as a zip which by default has not .git folder.  The error you are receiving happens when the release-opinion plugin cannot find the git info it needs to infer a release version (I used this plugin to tag my releases).  I can probably add a fix for this in the near future but your best bet right now would be to clone the project directly and then build as follows:

1.  git clone https://github.com/jpalomo/kafka-connect-mssqlserver-cdc.git

2.  gradlew.bat connect:build

* There is a gradle sub project for integrations test, but this will only work if you have an instance of SQL server installed and configured locally.  I will try to add more clearer steps and directions on how to build and test the project sometime soon.  Hope that helps