Reconnect debezium-connector to PostgreSQL db (AWS RDS) after a db restart (task is being killed and will not recover until manually restarted)

[George Tsopouridis]

Any ideas about the following unrecoverable exception? Thanks in advance.

2025-05-28 05:08:30,140 INFO [debezium-connector-postgres|task-0|offsets] WorkerSourceTask{id=debezium-connector-postgres-0} Committing offsets for 24 acknowledged messages (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1] 2025-05-28 05:08:36,769 INFO 127.0.0.6 - - [28/May/2025:05:08:36 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.31+" 0 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp365255349-30] 2025-05-28 05:08:36,769 INFO 127.0.0.6 - - [28/May/2025:05:08:36 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.31+" 0 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp365255349-27] 2025-05-28 05:08:46,769 INFO 127.0.0.6 - - [28/May/2025:05:08:46 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.31+" 1 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp365255349-23] 2025-05-28 05:08:46,769 INFO 127.0.0.6 - - [28/May/2025:05:08:46 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.31+" 1 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp365255349-30] 2025-05-28 05:08:51,433 ERROR [debezium-connector-postgres|task-0] Producer failure (io.debezium.pipeline.ErrorHandler) [debezium-postgresconnector-debezium-change-event-source-coordinator] org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2713) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2401) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:368) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:498) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:415) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:190) at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:134) at io.debezium.jdbc.JdbcConnection.prepareQuery(JdbcConnection.java:721) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.probeConnectionIfNeeded(PostgresStreamingChangeEventSource.java:383) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.processMessages(PostgresStreamingChangeEventSource.java:234) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:183) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:42) at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:324) at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:203) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:143) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) 2025-05-28 05:08:51,437 WARN [debezium-connector-postgres|task-0] Retry 1 of unlimited retries will be attempted (io.debezium.pipeline.ErrorHandler) [debezium-postgresconnector-debezium-change-event-source-coordinator] 2025-05-28 05:08:51,462 INFO [debezium-connector-postgres|task-0] Finished streaming (io.debezium.pipeline.ChangeEventSourceCoordinator) [debezium-postgresconnector-debezium-change-event-source-coordinator] 2025-05-28 05:08:51,462 INFO [debezium-connector-postgres|task-0] Connected metrics set to 'false' (io.debezium.pipeline.ChangeEventSourceCoordinator) [debezium-postgresconnector-debezium-change-event-source-coordinator] 2025-05-28 05:08:51,708 WARN [debezium-connector-postgres|task-0] Going to restart connector after 10 sec. after a retriable exception (io.debezium.connector.common.BaseSourceTask) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:08:51,711 INFO [debezium-connector-postgres|task-0] SignalProcessor stopped (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:08:51,712 INFO [debezium-connector-postgres|task-0] Connection gracefully closed (io.debezium.jdbc.JdbcConnection) [pool-16-thread-1] 2025-05-28 05:08:51,712 INFO [debezium-connector-postgres|task-0] Debezium ServiceRegistry stopped. (io.debezium.service.DefaultServiceRegistry) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:08:51,713 INFO [debezium-connector-postgres|task-0] Connection gracefully closed (io.debezium.jdbc.JdbcConnection) [pool-17-thread-1] 2025-05-28 05:08:51,714 INFO [debezium-connector-postgres|task-0] Connection gracefully closed (io.debezium.jdbc.JdbcConnection) [pool-18-thread-1] 2025-05-28 05:08:51,716 INFO [debezium-connector-postgres|task-0] Connection gracefully closed (io.debezium.jdbc.JdbcConnection) [pool-19-thread-1] 2025-05-28 05:08:51,716 WARN [debezium-connector-postgres|task-0] WorkerSourceTask{id=debezium-connector-postgres-0} failed to poll records from SourceTask. Will retry operation. (org.apache.kafka.connect.runtime.AbstractWorkerSourceTask) [task-thread-debezium-connector-postgres-0] org.apache.kafka.connect.errors.RetriableException: An exception occurred in the change event producer. This connector will be restarted. at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:63) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:186) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:42) at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:324) at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:203) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:143) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2713) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2401) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:368) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:498) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:415) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:190) at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:134) at io.debezium.jdbc.JdbcConnection.prepareQuery(JdbcConnection.java:721) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.probeConnectionIfNeeded(PostgresStreamingChangeEventSource.java:383) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.processMessages(PostgresStreamingChangeEventSource.java:234) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:183) ... 9 more 2025-05-28 05:08:51,717 INFO [debezium-connector-postgres|task-0] Awaiting end of restart backoff period after a retriable error (io.debezium.connector.common.BaseSourceTask) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:08:56,769 INFO 127.0.0.6 - - [28/May/2025:05:08:56 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.31+" 1 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp365255349-23] 2025-05-28 05:08:56,769 INFO 127.0.0.6 - - [28/May/2025:05:08:56 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.31+" 1 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp365255349-26] 2025-05-28 05:09:01,717 INFO [debezium-connector-postgres|task-0] Attempting to restart task. (io.debezium.connector.common.BaseSourceTask) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,717 INFO [debezium-connector-postgres|task-0] Loading the custom source info struct maker plugin: io.debezium.connector.postgresql.PostgresSourceInfoStructMaker (io.debezium.config.CommonConnectorConfig) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,718 INFO [debezium-connector-postgres|task-0] Loading the custom topic naming strategy plugin: io.debezium.schema.SchemaTopicNamingStrategy (io.debezium.config.CommonConnectorConfig) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,721 INFO [debezium-connector-postgres|task-0] WorkerSourceTask{id=debezium-connector-postgres-0} Committing offsets for 8 acknowledged messages (org.apache.kafka.connect.runtime.WorkerSourceTask) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,724 ERROR [debezium-connector-postgres|task-0] WorkerSourceTask{id=debezium-connector-postgres-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-debezium-connector-postgres-0] io.debezium.DebeziumException: Couldn't obtain encoding for database picflow at io.debezium.connector.postgresql.connection.PostgresConnection.getDatabaseCharset(PostgresConnection.java:599) at io.debezium.connector.postgresql.PostgresConnectorTask.start(PostgresConnectorTask.java:91) at io.debezium.connector.common.BaseSourceTask.startIfNeededAndPossible(BaseSourceTask.java:403) at io.debezium.connector.common.BaseSourceTask.poll(BaseSourceTask.java:313) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.poll(AbstractWorkerSourceTask.java:465) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:353) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:225) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:280) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:78) at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:237) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: org.postgresql.util.PSQLException: The connection attempt failed. at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:354) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:263) at org.postgresql.Driver.makeConnection(Driver.java:443) at org.postgresql.Driver.connect(Driver.java:297) at io.debezium.jdbc.JdbcConnection.lambda$patternBasedFactory$1(JdbcConnection.java:246) at io.debezium.jdbc.JdbcConnection$ConnectionFactoryDecorator.connect(JdbcConnection.java:132) at io.debezium.jdbc.JdbcConnection.establishConnection(JdbcConnection.java:920) at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:904) at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:898) at io.debezium.connector.postgresql.connection.PostgresConnection.getDatabaseCharset(PostgresConnection.java:596) ... 14 more Caused by: java.net.SocketException: Connection reset at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:328) at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355) at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808) at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966) at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:128) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:113) at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73) at org.postgresql.core.PGStream.receiveChar(PGStream.java:465) at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:589) at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:191) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:258) ... 24 more 2025-05-28 05:09:01,725 INFO [debezium-connector-postgres|task-0] Stopping down connector (io.debezium.connector.common.BaseSourceTask) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,726 INFO [debezium-connector-postgres|task-0] [Producer clientId=connector-producer-debezium-connector-postgres-0] Closing the Kafka producer with timeoutMillis = 30000 ms. (org.apache.kafka.clients.producer.KafkaProducer) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,729 INFO [debezium-connector-postgres|task-0] Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,729 INFO [debezium-connector-postgres|task-0] Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,729 INFO [debezium-connector-postgres|task-0] Closing reporter org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter (org.apache.kafka.common.metrics.Metrics) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,729 INFO [debezium-connector-postgres|task-0] Metrics reporters closed (org.apache.kafka.common.metrics.Metrics) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,730 INFO [debezium-connector-postgres|task-0] App info kafka.producer for connector-producer-debezium-connector-postgres-0 unregistered (org.apache.kafka.common.utils.AppInfoParser) [task-thread-debezium-connector-postgres-0]

[Chris Cranford]

This is something that would be expected when the database server is restarted. Typically restarts will force close any active connections that are not terminated before the database server stops.

-cc

On 5/29/25 2:30 AM, 'George Tsopouridis' via debezium wrote:

Any ideas about the following unrecoverable exception? Thanks in advance.

2025-05-28 05:08:30,140 INFO [debezium-connector-postgres|task-0|offsets] WorkerSourceTask{id=debezium-connector-postgres-0} Committing offsets for 24 acknowledged messages (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1] 2025-05-28 05:08:36,769 INFO 127.0.0.6 - - [28/May/2025:05:08:36 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.31+" 0 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp365255349-30] 2025-05-28 05:08:36,769 INFO 127.0.0.6 - - [28/May/2025:05:08:36 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.31+" 0 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp365255349-27] 2025-05-28 05:08:46,769 INFO 127.0.0.6 - - [28/May/2025:05:08:46 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.31+" 1 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp365255349-23] 2025-05-28 05:08:46,769 INFO 127.0.0.6 - - [28/May/2025:05:08:46 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.31+" 1 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp365255349-30] 2025-05-28 05:08:51,433 ERROR [debezium-connector-postgres|task-0] Producer failure (io.debezium.pipeline.ErrorHandler) [debezium-postgresconnector-debezium-change-event-source-coordinator] org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2713) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2401) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:368) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:498) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:415) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:190) at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:134) at io.debezium.jdbc.JdbcConnection.prepareQuery(JdbcConnection.java:721) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.probeConnectionIfNeeded(PostgresStreamingChangeEventSource.java:383) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.processMessages(PostgresStreamingChangeEventSource.java:234) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:183) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:42) at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:324) at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:203) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:143) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) 2025-05-28 05:08:51,437 WARN [debezium-connector-postgres|task-0] Retry 1 of unlimited retries will be attempted (io.debezium.pipeline.ErrorHandler) [debezium-postgresconnector-debezium-change-event-source-coordinator] 2025-05-28 05:08:51,462 INFO [debezium-connector-postgres|task-0] Finished streaming (io.debezium.pipeline.ChangeEventSourceCoordinator) [debezium-postgresconnector-debezium-change-event-source-coordinator] 2025-05-28 05:08:51,462 INFO [debezium-connector-postgres|task-0] Connected metrics set to 'false' (io.debezium.pipeline.ChangeEventSourceCoordinator) [debezium-postgresconnector-debezium-change-event-source-coordinator] 2025-05-28 05:08:51,708 WARN [debezium-connector-postgres|task-0] Going to restart connector after 10 sec. after a retriable exception (io.debezium.connector.common.BaseSourceTask) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:08:51,711 INFO [debezium-connector-postgres|task-0] SignalProcessor stopped (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:08:51,712 INFO [debezium-connector-postgres|task-0] Connection gracefully closed (io.debezium.jdbc.JdbcConnection) [pool-16-thread-1] 2025-05-28 05:08:51,712 INFO [debezium-connector-postgres|task-0] Debezium ServiceRegistry stopped. (io.debezium.service.DefaultServiceRegistry) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:08:51,713 INFO [debezium-connector-postgres|task-0] Connection gracefully closed (io.debezium.jdbc.JdbcConnection) [pool-17-thread-1] 2025-05-28 05:08:51,714 INFO [debezium-connector-postgres|task-0] Connection gracefully closed (io.debezium.jdbc.JdbcConnection) [pool-18-thread-1] 2025-05-28 05:08:51,716 INFO [debezium-connector-postgres|task-0] Connection gracefully closed (io.debezium.jdbc.JdbcConnection) [pool-19-thread-1] 2025-05-28 05:08:51,716 WARN [debezium-connector-postgres|task-0] WorkerSourceTask{id=debezium-connector-postgres-0} failed to poll records from SourceTask. Will retry operation. (org.apache.kafka.connect.runtime.AbstractWorkerSourceTask) [task-thread-debezium-connector-postgres-0] org.apache.kafka.connect.errors.RetriableException: An exception occurred in the change event producer. This connector will be restarted. at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:63) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:186) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:42) at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:324) at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:203) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:143) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2713) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2401) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:368) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:498) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:415) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:190) at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:134) at io.debezium.jdbc.JdbcConnection.prepareQuery(JdbcConnection.java:721) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.probeConnectionIfNeeded(PostgresStreamingChangeEventSource.java:383) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.processMessages(PostgresStreamingChangeEventSource.java:234) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:183) ... 9 more 2025-05-28 05:08:51,717 INFO [debezium-connector-postgres|task-0] Awaiting end of restart backoff period after a retriable error (io.debezium.connector.common.BaseSourceTask) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:08:56,769 INFO 127.0.0.6 - - [28/May/2025:05:08:56 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.31+" 1 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp365255349-23] 2025-05-28 05:08:56,769 INFO 127.0.0.6 - - [28/May/2025:05:08:56 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.31+" 1 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp365255349-26] 2025-05-28 05:09:01,717 INFO [debezium-connector-postgres|task-0] Attempting to restart task. (io.debezium.connector.common.BaseSourceTask) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,717 INFO [debezium-connector-postgres|task-0] Loading the custom source info struct maker plugin: io.debezium.connector.postgresql.PostgresSourceInfoStructMaker (io.debezium.config.CommonConnectorConfig) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,718 INFO [debezium-connector-postgres|task-0] Loading the custom topic naming strategy plugin: io.debezium.schema.SchemaTopicNamingStrategy (io.debezium.config.CommonConnectorConfig) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,721 INFO [debezium-connector-postgres|task-0] WorkerSourceTask{id=debezium-connector-postgres-0} Committing offsets for 8 acknowledged messages (org.apache.kafka.connect.runtime.WorkerSourceTask) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,724 ERROR [debezium-connector-postgres|task-0] WorkerSourceTask{id=debezium-connector-postgres-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-debezium-connector-postgres-0] io.debezium.DebeziumException: Couldn't obtain encoding for database picflow at io.debezium.connector.postgresql.connection.PostgresConnection.getDatabaseCharset(PostgresConnection.java:599) at io.debezium.connector.postgresql.PostgresConnectorTask.start(PostgresConnectorTask.java:91) at io.debezium.connector.common.BaseSourceTask.startIfNeededAndPossible(BaseSourceTask.java:403) at io.debezium.connector.common.BaseSourceTask.poll(BaseSourceTask.java:313) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.poll(AbstractWorkerSourceTask.java:465) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:353) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:225) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:280) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:78) at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:237) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: org.postgresql.util.PSQLException: The connection attempt failed. at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:354) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:263) at org.postgresql.Driver.makeConnection(Driver.java:443) at org.postgresql.Driver.connect(Driver.java:297) at io.debezium.jdbc.JdbcConnection.lambda$patternBasedFactory$1(JdbcConnection.java:246) at io.debezium.jdbc.JdbcConnection$ConnectionFactoryDecorator.connect(JdbcConnection.java:132) at io.debezium.jdbc.JdbcConnection.establishConnection(JdbcConnection.java:920) at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:904) at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:898) at io.debezium.connector.postgresql.connection.PostgresConnection.getDatabaseCharset(PostgresConnection.java:596) ... 14 more Caused by: java.net.SocketException: Connection reset at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:328) at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355) at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808) at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966) at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:128) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:113) at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73) at org.postgresql.core.PGStream.receiveChar(PGStream.java:465) at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:589) at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:191) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:258) ... 24 more 2025-05-28 05:09:01,725 INFO [debezium-connector-postgres|task-0] Stopping down connector (io.debezium.connector.common.BaseSourceTask) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,726 INFO [debezium-connector-postgres|task-0] [Producer clientId=connector-producer-debezium-connector-postgres-0] Closing the Kafka producer with timeoutMillis = 30000 ms. (org.apache.kafka.clients.producer.KafkaProducer) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,729 INFO [debezium-connector-postgres|task-0] Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,729 INFO [debezium-connector-postgres|task-0] Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,729 INFO [debezium-connector-postgres|task-0] Closing reporter org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter (org.apache.kafka.common.metrics.Metrics) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,729 INFO [debezium-connector-postgres|task-0] Metrics reporters closed (org.apache.kafka.common.metrics.Metrics) [task-thread-debezium-connector-postgres-0] 2025-05-28 05:09:01,730 INFO [debezium-connector-postgres|task-0] App info kafka.producer for connector-producer-debezium-connector-postgres-0 unregistered (org.apache.kafka.common.utils.AppInfoParser) [task-thread-debezium-connector-postgres-0] --
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/7817eaba-1424-44ad-9dbf-910537e7193an%40googlegroups.com.

[George Tsopouridis]

Hey Chris,
thank you so much for your feedback - I really appreciate it.

Do you have any suggestions for how to work around this issue? It occurs during every maintenance window when the AWS RDS service applies a database update or patch that requires a restart.

As always, thanks in advance for your support.

Best regards,
George

[Chris Cranford]

Hi George -

It appears to me this error is being retried, see:

    Going to restart connector after 10 sec. after a retriable exception

    Awaiting end of restart backoff period after a retriable error

Debezium delegates retry/back-off to the Kafka Connect, so the connector should be successfully restarted. If the database remains unavailable longer than the Kafka Connect restart time period, you may need to utilize a script that checks and restarts the task using the Kafka Connect REST API as needed, but it seems Kafka Connect is doing its job here. Is that not what you observe?

-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/3474ef75-04fd-4883-a33b-2955bc1527b9n%40googlegroups.com.