Cassandra CDC feature doesn't work well for Debezium Cassandra Connector

[Bingqin Zhou]

Hey,

For people who have been using or testing with Debezium Cassandra Connector, does the CDC feature in Cassandra work well for you? 

In our case, CommitLog files don't get flushed into cdc_raw very often, and the flush frequency can be as low as a few months. May I know how you deal with this issue and manage to make Commitlog files flushed more frequently into cdc_raw directory please?

Thank you,

Bingqin Zhou

[Ahmed Eljami]

Hi Bingqin,

Not yet in production but yes it works well :)

In our case, tables with cdc enabled have a low write limit so memtables are rarely flushed ... To deal with this, we trigger an explicit nodetool flush every X minutes, commitLogs are transferred well into cdc_raw and are consumed by the connector.

You can  also deal with the following commitLogs parameters to, low setting could speed up the flush and transfer of commitLog into cdc_raw:

commitlog_total_space_in_mb, default 8G

commitlog_segment_size_in_mb, default 32mb

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/CABYXsiHsbmtq-eJ0BX5joCvqyZvU4r9W%2BY53F0CsxsokRZrP8Q%40mail.gmail.com.

--

Cordialement;

Ahmed ELJAMI

[Ahmed Eljami]

And one another thing, we will enable cdc only on stats datacenter that we use it for only  analytical jobs to ensure that the explicit flush does not impact the live users requests.

Cheers,

[Bingqin Zhou]

I see :) Thank you for the insights and suggestions Ahmed!

Bingqin Zhou

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/CAJjuABYf47kXbtehhkDM%2Bu4CNan%2B09e2Q1AuRajdWBhyQvnXOw%40mail.gmail.com.

[Ahmed Eljami]

Hi Bingqin,

 I would also be interested in the work that you are doing on your side to deal with the commitLogs flush. Could you share it when you are finishing with it please :)

Also, is there a plan to add support for Cassandra-4? As the GA version will be released soon, the new version of cdc could help us reading the commitLog without waiting for it to be flushed.

Cheers,

Ahmed

[Bingqin Zhou]

Hi Ahmed,

Sure, will keep you updated once we figure this out :)

We're planning to support Cassandra 4.0 as soon as it's officially released.

Thanks,

Bingqin Zhou

--

You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/CAJjuABad4avT-N7ErcoQqceLx2bQbysutY4sEabG-bHGnfknZA%40mail.gmail.com.

[Ahmed Eljami]

Good news!

Thanks a lot Bingqin :)

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/CABYXsiF0ocqoKaZjXkfmNJ0HS-7vq5QEne1jdKKaEdzveDS4iw%40mail.gmail.com.

[Ahmed Eljami]

Hi  Bingqin,

Now that cassandra 4.0 GA has been released, you think that we will get a version that supports Cassandra 4.0 soon ?

Thanks a lot,

Ahmed

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/CABYXsiF0ocqoKaZjXkfmNJ0HS-7vq5QEne1jdKKaEdzveDS4iw%40mail.gmail.com.

--

Cordialement;

Ahmed ELJAMI