VARCHAR data type

[Bernhard Pieber]

Hi everyone,

I want to use Debezium to stream changes from PostgreSQL to Oracle. I am using the Kafka Connect JDBC Sink Connector with Debezium's ExtractNewRecordState transform. I use Avro with the Apicurio Schema Registry.

Debezium converts the PostgreSQL VARCHAR data types to the String data type in the Avro schema. The JDBC Sink Connector then converts String to CLOB, which is not what I want.

As far as I can see in the Apicurio UI the VARCHAR max length seems not to be present in the Avro schema at all. Is there some way to preserve the VARCHAR max length without manually creating all tables and manually migrating them when the schema changes?

As you can probably guess I am new to Debezium, Kafka, and Kafka Connect. Any help is greatly appreciated.

Cheers,

Bernhard

[Chris Cranford]

Hi Bernhard -

What is the exact table column definition in PostgreSQL?

CC

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/fde20a35-b23e-4ff8-88c1-73089b37dcf7n%40googlegroups.com.

[Bernhard Pieber]

Hi Chris,

Here is my create table statement:

CREATE TABLE test (

    id VARCHAR(255) NOT NULL,

    PRIMARY KEY (id)

);

The ids are created on the client side.

Cheers,

Bernhard

[Chris Cranford]

Hi Bernhard -

What happens if you specify `datatype.propagate.source.type` [1], does the JDBC sink you're using perhaps parse the parameters for these? 

Lets take the following table:

    CREATE TABLE s2.b (
        pk serial,
        bb varchar2(50),
        primary key(pk)
    );

When I specify "datatype.propagate.source.type=s2.b.varchar" in the connector configuration, the emitted event's schema for the "bb" column has the following parameters:

    __debezium.source.column.type=VARCHAR
    __debezium.source.column.scale=0
    __debezium.source.column.length=50

HTH,
CC

[1]: https://debezium.io/documentation/reference/1.7/connectors/postgresql.html#postgresql-property-datatype-propagate-source-type