debezium postgres connector change events getting missed
227 views
Skip to first unread message
Mithilesh kumar
Oct 20, 2021, 9:51:24 AM10/20/21
to debezium
Hi Team,
We are using debezium version 1.2.4.Final and postgres version aws rds 13.2
plugin.name - pgoutput
some time may be at the time when there is bulk insertion in rds, we are getting this logs in debezium continuously (received LSN > requested LSN)
2021-10-19 22:08:52,689 INFO Postgres|postgres-mf_tracking-connector|postgres-connector-task Streaming requested from LSN LSN{5CD/70C100E8}, received LSN LSN{5CE/874C7110} identified as already processed [io.debezium.connector.postgresql.connection.AbstractMessageDecoder]
2021-10-19 22:08:52,689 INFO Postgres|postgres-mf_tracking-connector|postgres-connector-task Streaming requested from LSN LSN{5CD/70C100E8}, received LSN LSN{5CE/874C7248} identified as already processed [io.debezium.connector.postgresql.connection.AbstractMessageDecoder]
2021-10-19 22:08:52,689 INFO Postgres|postgres-mf_tracking-connector|postgres-connector-task Streaming requested from LSN LSN{5CD/70C100E8}, received LSN LSN{5CE/874C7380} identified as already processed [io.debezium.connector.postgresql.connection.AbstractMessageDecoder]
2021-10-19 22:08:52,689 INFO Postgres|postgres-mf_tracking-connector|postgres-connector-task Streaming requested from LSN LSN{5CD/70C100E8}, received LSN LSN{5CE/874C74B8} identified as already processed [io.debezium.connector.postgresql.connection.AbstractMessageDecoder]
After that get ERROR
2021-10-19 22:08:52,691 ERROR Postgres|postgres-mf_tracking-connector|postgres-connector-task Producer failure [io.debezium.pipeline.ErrorHandler]
org.postgresql.util.PSQLException: Database connection failed when reading from copy at org.postgresql.core.v3.QueryExecutorImpl.readFromCopy(QueryExecutorImpl.java:1102) at org.postgresql.core.v3.CopyDualImpl.readFromCopy(CopyDualImpl.java:42) at org.postgresql.core.v3.replication.V3PGReplicationStream.receiveNextData(V3PGReplicationStream.java:158)
After that connector got restarted and start picking changes from new LSN and we got miss some events.
Can you please help here?
We are using debezium version 1.2.4.Final and postgres version aws rds 13.2
plugin.name - pgoutput
some time may be at the time when there is bulk insertion in rds, we are getting this logs in debezium continuously (received LSN > requested LSN)
2021-10-19 22:08:52,689 INFO Postgres|postgres-mf_tracking-connector|postgres-connector-task Streaming requested from LSN LSN{5CD/70C100E8}, received LSN LSN{5CE/874C7110} identified as already processed [io.debezium.connector.postgresql.connection.AbstractMessageDecoder]
2021-10-19 22:08:52,689 INFO Postgres|postgres-mf_tracking-connector|postgres-connector-task Streaming requested from LSN LSN{5CD/70C100E8}, received LSN LSN{5CE/874C7248} identified as already processed [io.debezium.connector.postgresql.connection.AbstractMessageDecoder]
2021-10-19 22:08:52,689 INFO Postgres|postgres-mf_tracking-connector|postgres-connector-task Streaming requested from LSN LSN{5CD/70C100E8}, received LSN LSN{5CE/874C7380} identified as already processed [io.debezium.connector.postgresql.connection.AbstractMessageDecoder]
2021-10-19 22:08:52,689 INFO Postgres|postgres-mf_tracking-connector|postgres-connector-task Streaming requested from LSN LSN{5CD/70C100E8}, received LSN LSN{5CE/874C74B8} identified as already processed [io.debezium.connector.postgresql.connection.AbstractMessageDecoder]
After that get ERROR
2021-10-19 22:08:52,691 ERROR Postgres|postgres-mf_tracking-connector|postgres-connector-task Producer failure [io.debezium.pipeline.ErrorHandler]
org.postgresql.util.PSQLException: Database connection failed when reading from copy at org.postgresql.core.v3.QueryExecutorImpl.readFromCopy(QueryExecutorImpl.java:1102) at org.postgresql.core.v3.CopyDualImpl.readFromCopy(CopyDualImpl.java:42) at org.postgresql.core.v3.replication.V3PGReplicationStream.receiveNextData(V3PGReplicationStream.java:158)
After that connector got restarted and start picking changes from new LSN and we got miss some events.
Can you please help here?
Reply all
Reply to author
Forward
0 new messages