Cassandra CDC feature doesn't work well for Debezium Cassandra Connector

5 views
Skip to first unread message

Bingqin Zhou

unread,
May 3, 2021, 1:40:32 PMMay 3
to debe...@googlegroups.com
Hey,

For people who have been using or testing with Debezium Cassandra Connector, does the CDC feature in Cassandra work well for you? 

In our case, CommitLog files don't get flushed into cdc_raw very often, and the flush frequency can be as low as a few months. May I know how you deal with this issue and manage to make Commitlog files flushed more frequently into cdc_raw directory please?

Thank you,
Bingqin Zhou

Ahmed Eljami

unread,
May 3, 2021, 2:23:43 PMMay 3
to debe...@googlegroups.com
Hi Bingqin,

Not yet in production but yes it works well :)

In our case, tables with cdc enabled have a low write limit so memtables are rarely flushed ... To deal with this, we trigger an explicit nodetool flush every X minutes, commitLogs are transferred well into cdc_raw and are consumed by the connector.

You can  also deal with the following commitLogs parameters to, low setting could speed up the flush and transfer of commitLog into cdc_raw:
commitlog_total_space_in_mb, default 8G
commitlog_segment_size_in_mb, default 32mb

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/CABYXsiHsbmtq-eJ0BX5joCvqyZvU4r9W%2BY53F0CsxsokRZrP8Q%40mail.gmail.com.


--
Cordialement;

Ahmed ELJAMI

Ahmed Eljami

unread,
May 3, 2021, 3:45:41 PMMay 3
to debe...@googlegroups.com
And one another thing, we will enable cdc only on stats datacenter that we use it for only  analytical jobs to ensure that the explicit flush does not impact the live users requests.

Cheers,


Bingqin Zhou

unread,
May 3, 2021, 6:23:22 PMMay 3
to debe...@googlegroups.com
I see :) Thank you for the insights and suggestions Ahmed!

Bingqin Zhou

Ahmed Eljami

unread,
May 4, 2021, 7:40:55 PMMay 4
to debe...@googlegroups.com
Hi Bingqin,

 I would also be interested in the work that you are doing on your side to deal with the commitLogs flush. Could you share it when you are finishing with it please :)

Also, is there a plan to add support for Cassandra-4? As the GA version will be released soon, the new version of cdc could help us reading the commitLog without waiting for it to be flushed.

Cheers,
Ahmed

Bingqin Zhou

unread,
May 5, 2021, 1:43:29 PMMay 5
to debe...@googlegroups.com
Hi Ahmed,

Sure, will keep you updated once we figure this out :)

We're planning to support Cassandra 4.0 as soon as it's officially released.

Thanks,
Bingqin Zhou

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.

Ahmed Eljami

unread,
May 5, 2021, 3:40:59 PMMay 5
to debe...@googlegroups.com
Good news!

Thanks a lot Bingqin :)


Reply all
Reply to author
Forward
0 new messages