Hazelcast Jet + Debezium CDC

[Ivan Belfinger]

Hey everyone!

We are piloting Hazelcast Jet + Debezium CDC in one of our apps. Hazelcast is running as embedded into the java app. When running in this configuration we have noticed that WAL changes are coming in but Postgres Replication Slot doesn't advance LSN at all. Running debezium Embedded engine on it's own works properly.

Questions:

1. Is this a proper forum to ask Debezium + Jet questions?
2. What can I do not to block out replication slot (server has run out of disk space once already.

Thanks!

[Neil Stevenson]

Hi,

 (1) definitely the right place to ask

 Can you confirm what versions of Jet, Debezium and Postgres you're using please?

 Jet 4.3, Debezium 1.3 Postgres 12.... etc

Neil

[Ivan Belfinger]

Hi!

I was able to reproduce the same behaviour using this example:

https://jet-start.sh/docs/tutorials/cdc-postgres

But instead of doing "4. Start Hazelcast Jet", I start JetJob.main() through IDE. In app console I see events coming in but replication slot is parked

postgres@localhost:postgres> select * from pg_replication_slots;

+-------------+-------------+-------------+----------+------------+-------------+----------+--------------+--------+----------------+---------------+-----------------------+

| slot_name   | plugin      | slot_type   | datoid   | database   | temporary   | active   | active_pid   | xmin   | catalog_xmin   | restart_lsn   | confirmed_flush_lsn   |

|-------------+-------------+-------------+----------+------------+-------------+----------+--------------+--------+----------------+---------------+-----------------------|

| debezium    | decoderbufs | logical     | 13067    | postgres   | False       | True     | 90           | <null> | 601            | 0/207DFE8     | 0/207E038             |

+-------------+-------------+-------------+----------+------------+-------------+----------+--------------+--------+----------------+---------------+-----------------------+

SELECT 1

Time: 0.008s

postgres@localhost:postgres> insert into inventory.customers(id, first_name, last_name, email) values (1006, 'Ivan', 'Belfinger', 'ivan.l...@gmail.com')

INSERT 0 1

Time: 0.008s

postgres@localhost:postgres> insert into inventory.customers(id, first_name, last_name, email) values (10061, 'Ivan', 'Belfinger', 'ivan....@gmail.com')

INSERT 0 1

Time: 0.006s

postgres@localhost:postgres> insert into inventory.customers(id, first_name, last_name, email) values (101061, 'Ivan', 'Belfinger', 'ivan.l...@gmail.com')

INSERT 0 1

Time: 0.006s

postgres@localhost:postgres> select * from pg_replication_slots;

+-------------+-------------+-------------+----------+------------+-------------+----------+--------------+--------+----------------+---------------+-----------------------+

| slot_name   | plugin      | slot_type   | datoid   | database   | temporary   | active   | active_pid   | xmin   | catalog_xmin   | restart_lsn   | confirmed_flush_lsn   |

|-------------+-------------+-------------+----------+------------+-------------+----------+--------------+--------+----------------+---------------+-----------------------|

| debezium    | decoderbufs | logical     | 13067    | postgres   | False       | True     | 90           | <null> | 601            | 0/207DFE8     | 0/207E038             |

+-------------+-------------+-------------+----------+------------+-------------+----------+--------------+--------+----------------+---------------+-----------------------+

[Ivan Belfinger]

And yeah, the setup and code is very similar to what we have. Did we do something wrong?

[Ivan Belfinger]

Tried with the full example, WAL still isn't flushed. Even on Jet server shutdown

[Jozsef Bartok]

Hi. 

We have looked into this and turns out it's indeed a piece of missing functionality in our CDC source implementation. 

Our testing was focused on MySQL and we have missed it... 

Sorry for the inconvenience.

This is the issue that has been created to track the fix: https://github.com/hazelcast/hazelcast-jet/issues/2841

It's too late to sneak the fix into our upcoming 4.4 release, but we will treat the problem with the highest priority and we will try to release a fix ASAP.

I will notify you in this thread too, once the fix is available.

Best regards,

Jozsef

This message contains confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. -Hazelcast

[Ivan Belfinger]

Hi!

Thanks for the quick answer and thanks for checking it out! Won't be an issue, we have an workaround already in place.

In rough estimate, what do you think, when will fix be available?

[Jozsef Bartok]

I hope I will be able to fix it next week and provide you with some way of trying it out. When the fix will get into a proper release, that I don’t know. Will update you, when I do.

Just curious: what workaround are you using for it?

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/a28a7976-7529-49bc-97e5-91bca8a06104n%40googlegroups.com.

--

Bartók József Jet Team Engineer       hazelcast®     +4 0722 300 315 | jet-start.sh

[Ivan Belfinger]

Sounds good! 

We are running Debezium Engine in Embedded mode, and filling in a buffer. We also wrote a custom StreamSource that than drains that buffer. Pretty much inspired by other connectors: https://github.com/hazelcast/hazelcast-jet-contrib/tree/master/http. Probably doesn't cover all edge cases but it's good enough for our pilot.

Let me know if you need any help with coding/reviewing/testing.

[Jozsef Bartok]

Update: fix is done, just a bit more testing needed and then to find a way to get it to you, so you can try it sooner then waiting for it to be properly released.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/e2bca716-ca1e-4f9a-a9e7-afee4e2ea382n%40googlegroups.com.

[Jozsef Bartok]

Hi. 

I have finally managed to finish and test the fix. In the meantime we have released Jet 4.4 (which does NOT include the fix) 

and the fix is on that version's maintenance branch. Switching to this new version shouldn't affect you, it does not change

anything else related to CDC.

Run following commands and those should yield a Jet distribution archive, which you can use just like the released one you are using now.

Pls. try it out and let me know if it solves your problem or not. 

Also feel free to contact me if there are any issues and I'll try to help out.

The command I mentioned:

git clone https://github.com/hazelcast/hazelcast-jet.git --branch 4.4-maintenance --single-branch

cd hazelcast-jet

mvn package -DskipTests

ls hazelcast-jet-distribution/target/hazelcast-jet-4.4.1-SNAPSHOT.tar.gz

Hopefully I will be able to get this fix into a patch release for 4.4, but first things first, let's see if it works.

Rgds,

Jozsef