Memory usage on Mysq 5.7 with 200k tables

[AR]

Hello,

I'm using mysql 5.7 and debezium 1.6

In my case I have mysql shard server with 540 schemas and 276153 tables in all schemas summary.

Using:

snapshot.locking.mode: minimal_percona
snapshot.mode: schema_only

database.exclude.list: information_schema,mysql,performance_schema,sys

transforms: AddPrefix

transforms.Reroute.topic.replacement  to route all events in 1 topic.

DDL of all tables is the same. 

Data changes in mysql happens very rarely (about 10-50 per second).

Snapshot takes about 10-20 minutes.

Than events goes ok and kafka target topic fills with right data.

But memory usage is +8gb after  start and do not decrease.

After some profiling with jcmd heap_dump found, that memory used by

io.debezium.relational.Tables

io.debezium.relational.ddl.DdlChanges

The question is:

Is it possible to decrease memory usage after initial snapshot?

May be with some configuration option, assuming that DDL will not be changed never, and is the same for all tables.

[Gunnar Morling]

Sorry for the late reply; do you think you could take a heap dump an analyze what's consuming most of the memory? Assuming the internal metadata model is the most impacting factor, we do have a task for Debezium 1.8 for exploring and hopefully following through on improving the situation there: https://issues.redhat.com/browse/DBZ-4130.

--Gunnar