Debezium Server - Memory Consumption

39 views
Skip to first unread message

Adam Whitmore

unread,
Jan 9, 2022, 8:28:05 AMJan 9
to debezium
Is it expected that memory consumption would differ when changing snapshot mode from initial to schema_only?

We are running Debezium Server 1.6.2.Final with SQL Server connector and seeing memory consumption climb when setting snapshot mode to schema_only, particularly when processing large transactions.

I've run tests with one particular table, inserting 1.5M records in one transaction, which resulted in no increase in memory consumption when snapshot mode is set to initial.  Running that same transaction on an instance with snapshot mode set to schema_only results in an increase in memory consumption of around 200 MiB.

The increase in memory consumption is there with smaller transactions,  too, but to a lesser degree.  The increase was 70 MiB when inserting 1.5M records in 10 separate transactions of 150K records.

Thanks,
Adam

jiri.p...@gmail.com

unread,
Jan 10, 2022, 4:45:03 AMJan 10
to debezium
Hi,

thise should not be the case. The snapshot phase should consume more memory for initial snapshot but then the consumption should be more or less same. Could you please share some charts containing multiple GC runs?

Thanks

J.

Adam Whitmore

unread,
Jan 10, 2022, 5:12:47 AMJan 10
to debezium
Here's what I have from the runs I've recorded so far:
snapshot_mode_tests.png

Adam Whitmore

unread,
Jan 11, 2022, 12:50:08 PMJan 11
to debezium
Adding to this - also ran a test with v1.8.0.Final today in schema_only mode, had a similar increase from 1.52 GiB to 1.73 GiB when processing a 1.5M record insert transaction.
Reply all
Reply to author
Forward
0 new messages