Debezium stuck in an infinite loop on boot

[Brandon Harvey]

Hello,

Looking for some assistance with resolving our "infinite loop on boot" issue. Some context on our stack. We have a Kafka Connect container that hosts the Debezium connector, hooking into a Postgres 12 server. We're using pg_output.

Kafka cluster=2.6.0

dbz-kafka-connect=1.5.1

apicurio=1.3.2.Final

postgres=v12.4

When starting Debezium our kafka-connect instance continually throws the following logs:

2021-03-22 20:48:56,293 INFO   ||  WorkerSourceTask{id=booking-manager-ledger-connector-0} Committing offsets   [org.apache.kafka.connect.runtime.WorkerSourceTask]

2021-03-22 20:48:56,293 INFO   ||  WorkerSourceTask{id=booking-manager-ledger-connector-0} flushing 0 outstanding messages for offset commit   [org.apache.kafka.connect.runtime.WorkerSourceTask]

2021-03-22 20:48:56,293 WARN   ||  Couldn't commit processed log positions with the source database due to a concurrent connector shutdown or restart   [io.debezium.connector.common.BaseSourceTask]

2021-03-22 20:49:56,293 INFO   ||  WorkerSourceTask{id=booking-manager-ledger-connector-0} Committing offsets   [org.apache.kafka.connect.runtime.WorkerSourceTask]

2021-03-22 20:49:56,294 INFO   ||  WorkerSourceTask{id=booking-manager-ledger-connector-0} flushing 0 outstanding messages for offset commit   [org.apache.kafka.connect.runtime.WorkerSourceTask]

2021-03-22 20:49:56,294 WARN   ||  Couldn't commit processed log positions with the source database due to a concurrent connector shutdown or restart   [io.debezium.connector.common.BaseSourceTask]

Debezium appears to be running the following query to seemingly no end. We've left the system for more than two days with no change in it's behaviour:

SELECT DISTINCT ON (typname) typname, typinput='array_in'::regproc, typtype, sp.r, pg_type.oid FROM pg_catalog.pg_type LEFT JOIN (select ns.oid as nspoid, ns.nspname, r.r from pg_namespace as ns join ( select s.r, (current_schemas(false))[s.r] as nspname from generate_series(1, array_upper(current_schemas(false), 1)) as s(r) ) as r using ( nspname )) as sp ON sp.nspoid = typnamespace ORDER BY typname, sp.r, pg_type.oid

Additionally Debezium refuses to create a replication slot nor publications. There are no events streaming through and no mention of requiring to do a snapshot; the behaviour is the same whether we opt in or out of the snapshot.

We've done a search through the docs and gitter but haven't found any information on resolving this. A user from a while back did mention they were running into the same issue but they did not post a solution https://gitter.im/debezium/user?at=5f0f458be3505d7736004c06. Following the advice of one of the responders https://gitter.im/debezium/user?at=5f0ffcfa49fe2127949ab12e I ran the query manually which completed in ~7 seconds, most of the query listed out a ton of pg_toast_* typname, unsure if that is of any significance?

Kind Regards,

[Chris Cranford]

Hi Brandon -

This sounds like a bug with the priming of the PostgreSQL connector's TypeRegistry's SqlTypeMapper. 
Could you open a Jira issue for this and would you be willing to please run the following two queries and attach the results to the issue?

Query 1

SELECT t.oid AS oid, t.typname AS name, t.typelem AS element, t.typbasetype AS parentoid, t.typtypmod as modifiers, t.typcategory as category, e.values as enum_values
    FROM pg_catalog.pg_type t
    JOIN pg_catalog.pg_namespace n ON (t.typnamespace = n.oid)
    LEFT JOIN ( SELECT t.enumtypid as id, array_agg(t.enumlabel) as values FROM pg_catalog.pg_enum t GROUP BY id ) e ON (t.oid = e.id)
    WHERE n.nspname != 'pg_toast'

Query 2

SELECT DISTINCT ON (typname) typname, typinput='array_in'::regproc, typtype, sp.r, pg_type.oid
    FROM pg_catalog.pg_type
    LEFT
    JOIN (select ns.oid as nspoid, ns.nspname, r.r
        from pg_namespace as ns 
            join ( select s.r, (current_schemas(false))[s.r] as nspname
                from generate_series(1, array_upper(current_schemas(false), 1)) as s(r) ) as r
                    using ( nspname )
                    ) as sp
            ON sp.nspoid = typnamespace

    ORDER BY typname, sp.r, pg_type.oid;

Hopefully with the results we'll be able to identify the bottleneck.

Thanks,
Chris

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/b73901df-180e-49bb-b415-7e34738492c5n%40googlegroups.com.