PostgreSQL data delay

[Miquel Andreu]

Hi everybody,

After a few months using debezium for PostgreSQL we saw a really strange and bad thing.

When we compare the value for payload.source.ts_ms with the value for payload.ts_ms, we saw a lag from more than 3 weeks and is not a single case. Does anybody know what's happening here or any way to debug this, the connector is fine and works with any failure since starts.

payload.source.ts_ms  this value is a few weeks before the payload.ts_ms and this should be the time when the event was processed by debezium. So we see a huge difference between postgresql and debezium

This is the configuration used:

{
    "name": "xxxxxxx.DEBEZIUM_POSTGRESQL_CONNECTOR",
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "tasks.max": "1",
    "key.converter": "io.confluent.connect.avro.AvroConverter",
    "key.converter.schema.registry.url": "xxxxxx:23411",
    "key.converter.basic.auth.credentials.source": "xxxxx",
    "key.converter.basic.auth.user.info": "xxxxx",
    "value.converter": "io.confluent.connect.avro.AvroConverter",
    "value.converter.schema.registry.url": "xxxxxx:23411",
    "value.converter.basic.auth.credentials.source": "xxxxx",
    "value.converter.basic.auth.user.info": "xxxxxxb",
    "config.action.reload": "restart",
    "errors.tolerance": "none",
    "errors.log.enable": "true",
    "errors.log.include.messages": "true",
    "database.server.name": "cps-db000",
    "database.hostname": "xxxxxx",
    "database.port": "5432",
    "database.user": "xxxxx",
    "database.password": "xxxx",
    "database.dbname": "csi",
    "plugin.name": "wal2json",
    "decimal.handling.mode": "double",
    "snapshot.mode": "exported",
    "schema.refresh.mode": "columns_diff_exclude_unchanged_toast",
    "tombstones.on.delete": "false",
    "table.whitelist": "public.users, public.channels, public.user_silos, public.user_channels"
}

Thanks

[Gunnar Morling]

Hey,

Which version is this? Please try 1.4.0.Final, I remember some issue where incorrect lag values were exposed via metrics and which got recently fixed, but perhaps that's yet another thing. To be sure, do you observe wrong time stamp values, or do you actually *see* those change events only after three weeks in Debezium and Kafka? In case of the latter, you'd encounter a huge size of the replication slot in the DB, too, which I think is something you'd have noticed. In terms of debugging, hard to tell without knowing more details. If you have the option, you could run Connect in debug mode and make it halt via a conditional breakpoint for this particular situation.

--Gunnar