CDC vs Kafka Connect JDBC

1,746 views
Skip to first unread message

Unmesh Joshi

unread,
Apr 10, 2018, 2:03:22 AM4/10/18
to Confluent Platform
Hi,

On our current project we want to sync data from an Oracle database of existing system to new system. We are evaluating two mechanisms. 

1. Change data capture with Oracle Golden Gate 
2. Kafka Connect JDBC.

I am curious to know what the community prefers and if there are any lessons learnt from CDC implementations?

Thanks,
Unmesh 

Ewen Cheslack-Postava

unread,
Apr 11, 2018, 7:08:58 PM4/11/18
to Confluent Platform
CDC is generally better as it is more direct, efficient, and correctly captures things like deletes that the JDBC connector has no way to capture. The tradeoff is that you need a different connector implementation per database, details of data format and output may vary depending on the connector, and in the case of Oracle you're probably paying additional money to license a product that is able to pull CDC data.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/d1d2848e-ca43-4c9a-b4af-d0f6105a2acd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Unmesh Joshi

unread,
Apr 12, 2018, 11:52:35 PM4/12/18
to Confluent Platform
Hi EWen,

One of the other aspects of CDC we were looking at was how to capture transactional updates atomically. As I understand CDC will typically have a topic per table in RDBMS schema, so if we need to build a transactional aggregate from different topics, its tricky. I see a blog post on debezium blog http://debezium.io/blog/2018/03/08/creating-ddd-aggregates-with-debezium-and-kafka-streams/ .
Kafka connect JDBC will probably give u control to write custom read query to form these aggregates?  The trade off as you said it to get information about deletes etc. which is not possible with kafka connect.

Thanks,
Unmesh  


On Thursday, 12 April 2018 04:38:58 UTC+5:30, Ewen Cheslack-Postava wrote:
CDC is generally better as it is more direct, efficient, and correctly captures things like deletes that the JDBC connector has no way to capture. The tradeoff is that you need a different connector implementation per database, details of data format and output may vary depending on the connector, and in the case of Oracle you're probably paying additional money to license a product that is able to pull CDC data.

-Ewen
On Mon, Apr 9, 2018 at 11:03 PM, Unmesh Joshi <unmes...@gmail.com> wrote:
Hi,

On our current project we want to sync data from an Oracle database of existing system to new system. We are evaluating two mechanisms. 

1. Change data capture with Oracle Golden Gate 
2. Kafka Connect JDBC.

I am curious to know what the community prefers and if there are any lessons learnt from CDC implementations?

Thanks,
Unmesh 

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

Robin Moffatt

unread,
Apr 16, 2018, 6:20:31 AM4/16/18
to confluent...@googlegroups.com
You may have seen this post already, but in it I discuss some of the pros & cons of the two mechanisms: https://www.confluent.io/blog/no-more-silos-how-to-integrate-your-databases-with-apache-kafka-and-cdc. 
With regards to capturing transactions atomically, you'll definitely need log-based CDC for this. For example, in GG you have the tx mode option (https://docs.oracle.com/goldengate/bd123110/gg-bd/GADBD/using-kafka-handler.htm#GUID-E43DB743-4A2A-4C2F-97E8-50BB22CACCE3)


--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.

Rohit Chowdhary

unread,
Oct 15, 2018, 4:03:03 PM10/15/18
to Confluent Platform
Hi,

Is there a standard pattern/sink-connector to use the OGG CDC messages from Kafka topics to insert into another database like Postgres?
I am getting all the Insert/Update/Delete AVRO messages, so to consume these, does everyone writes Java code to do the inserts into the target db?
Or is there a generic/smarter way to do so?

Thanks



On Monday, April 16, 2018 at 6:20:31 AM UTC-4, Robin Moffatt wrote:
You may have seen this post already, but in it I discuss some of the pros & cons of the two mechanisms: https://www.confluent.io/blog/no-more-silos-how-to-integrate-your-databases-with-apache-kafka-and-cdc. 
With regards to capturing transactions atomically, you'll definitely need log-based CDC for this. For example, in GG you have the tx mode option (https://docs.oracle.com/goldengate/bd123110/gg-bd/GADBD/using-kafka-handler.htm#GUID-E43DB743-4A2A-4C2F-97E8-50BB22CACCE3)
On 10 April 2018 at 07:03, Unmesh Joshi <unmes...@gmail.com> wrote:
Hi,

On our current project we want to sync data from an Oracle database of existing system to new system. We are evaluating two mechanisms. 

1. Change data capture with Oracle Golden Gate 
2. Kafka Connect JDBC.

I am curious to know what the community prefers and if there are any lessons learnt from CDC implementations?

Thanks,
Unmesh 

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

Mayur Mahadeswar

unread,
Oct 15, 2018, 4:13:03 PM10/15/18
to confluent...@googlegroups.com
There is a jdbc sink connector to insert messages from kafka topic into Postgres. 
On Mon, Oct 15, 2018 at 4:03 PM Rohit Chowdhary <rohit.c...@gmail.com> wrote:
Hi,

Is there a standard pattern/sink-connector to use the OGG CDC messages from Kafka topics to insert into another database like Postgres?
I am getting all the Insert/Update/Delete AVRO messages, so to consume these, does everyone writes Java code to do the inserts into the target db?
Or is there a generic/smarter way to do so?

Thanks



On Monday, April 16, 2018 at 6:20:31 AM UTC-4, Robin Moffatt wrote:
You may have seen this post already, but in it I discuss some of the pros & cons of the two mechanisms: https://www.confluent.io/blog/no-more-silos-how-to-integrate-your-databases-with-apache-kafka-and-cdc. 
With regards to capturing transactions atomically, you'll definitely need log-based CDC for this. For example, in GG you have the tx mode option (https://docs.oracle.com/goldengate/bd123110/gg-bd/GADBD/using-kafka-handler.htm#GUID-E43DB743-4A2A-4C2F-97E8-50BB22CACCE3)
On 10 April 2018 at 07:03, Unmesh Joshi <unmes...@gmail.com> wrote:
Hi,

On our current project we want to sync data from an Oracle database of existing system to new system. We are evaluating two mechanisms. 

1. Change data capture with Oracle Golden Gate 
2. Kafka Connect JDBC.

I am curious to know what the community prefers and if there are any lessons learnt from CDC implementations?

Thanks,
Unmesh 

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


--
Regards, 
Mayur
Systems Engineer



Somasundaram Sekar

unread,
Oct 16, 2018, 3:01:09 AM10/16/18
to confluent...@googlegroups.com
Debezium connector has an SMT that will flatten the CDC AVRO record to a format that can directly be fed into JDBC sink connector(through a topic of course), maybe you can explore that.

On Mon, Oct 15, 2018 at 10:03 PM Rohit Chowdhary <rohit.c...@gmail.com> wrote:
Hi,

Is there a standard pattern/sink-connector to use the OGG CDC messages from Kafka topics to insert into another database like Postgres?
I am getting all the Insert/Update/Delete AVRO messages, so to consume these, does everyone writes Java code to do the inserts into the target db?
Or is there a generic/smarter way to do so?

Thanks



On Monday, April 16, 2018 at 6:20:31 AM UTC-4, Robin Moffatt wrote:
You may have seen this post already, but in it I discuss some of the pros & cons of the two mechanisms: https://www.confluent.io/blog/no-more-silos-how-to-integrate-your-databases-with-apache-kafka-and-cdc. 
With regards to capturing transactions atomically, you'll definitely need log-based CDC for this. For example, in GG you have the tx mode option (https://docs.oracle.com/goldengate/bd123110/gg-bd/GADBD/using-kafka-handler.htm#GUID-E43DB743-4A2A-4C2F-97E8-50BB22CACCE3)
On 10 April 2018 at 07:03, Unmesh Joshi <unmes...@gmail.com> wrote:
Hi,

On our current project we want to sync data from an Oracle database of existing system to new system. We are evaluating two mechanisms. 

1. Change data capture with Oracle Golden Gate 
2. Kafka Connect JDBC.

I am curious to know what the community prefers and if there are any lessons learnt from CDC implementations?

Thanks,
Unmesh 

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/a183b660-1390-44bb-9661-c2e6af86d356%40googlegroups.com.

Rohit Chowdhary

unread,
Oct 26, 2018, 8:02:01 AM10/26/18
to Confluent Platform
Thanks Mayur and Sekar for your response, very helpful. 

Regards
Rohit


On Tuesday, October 16, 2018 at 3:01:09 AM UTC-4, Somasundaram Sekar wrote:
Debezium connector has an SMT that will flatten the CDC AVRO record to a format that can directly be fed into JDBC sink connector(through a topic of course), maybe you can explore that.

On Mon, Oct 15, 2018 at 10:03 PM Rohit Chowdhary <rohit.c...@gmail.com> wrote:
Hi,

Is there a standard pattern/sink-connector to use the OGG CDC messages from Kafka topics to insert into another database like Postgres?
I am getting all the Insert/Update/Delete AVRO messages, so to consume these, does everyone writes Java code to do the inserts into the target db?
Or is there a generic/smarter way to do so?

Thanks



On Monday, April 16, 2018 at 6:20:31 AM UTC-4, Robin Moffatt wrote:
You may have seen this post already, but in it I discuss some of the pros & cons of the two mechanisms: https://www.confluent.io/blog/no-more-silos-how-to-integrate-your-databases-with-apache-kafka-and-cdc. 
With regards to capturing transactions atomically, you'll definitely need log-based CDC for this. For example, in GG you have the tx mode option (https://docs.oracle.com/goldengate/bd123110/gg-bd/GADBD/using-kafka-handler.htm#GUID-E43DB743-4A2A-4C2F-97E8-50BB22CACCE3)
On 10 April 2018 at 07:03, Unmesh Joshi <unmes...@gmail.com> wrote:
Hi,

On our current project we want to sync data from an Oracle database of existing system to new system. We are evaluating two mechanisms. 

1. Change data capture with Oracle Golden Gate 
2. Kafka Connect JDBC.

I am curious to know what the community prefers and if there are any lessons learnt from CDC implementations?

Thanks,
Unmesh 

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages