Kafka returning integers as "almost" Base64 ?

13 views
Skip to first unread message

Emanuele Meazzo

unread,
May 4, 2021, 6:19:27 AMMay 4
to debezium
Hello Everybody,

I have a strange issue with the Oracle connector that is blocking a POC I'm creating.
I was able to set everything up, and get the events posted in Kafka using the Logminer configuration, however, while strings are displayed ok, using the Kafka console consumer the Json message shows the integer fields in the Database in some kind of "almost" BASE64 string.

I tried to configure the connector specifying  "decimal.handling.mode": "precise" , but nothing changes.

I see something like this:

ORACLE
  • INSERT INTO TESTTABLE
  • VALUES (1,'hey hey hey hey');
  • COMMIT;

Console Output:
"payload":{"before":null,"after":{"ID":"AQ==","VALUE":"hey hey hey hey"}

"AQ==" really seems like base64, but it's definitely not the base64 equivalent of 1 (which is "MQ==")

Here are the details of my environment:
  1. Local Docker connector built with compose
  2. Oracle 19c hosted on AWS RDS
docker-compose.yaml is attached
This is the connector configuration I'm deploying:

{
  "name": "inventory-connector",
  "config": {
    "connector.class" : "io.debezium.connector.oracle.OracleConnector",
    "tasks.max" : "1",
    "database.server.name" : "XXX",
    "database.hostname" : XXXXXXXXXXXX",
    "database.port" : "1521",
    "database.user" : "LOGMINER",
    "database.password" : "XXXXXXXXX",
    "database.dbname" : "XXX",
"database.schema" : "XXX",
    "database.connection.adapter" : "LOGMINER",
    "database.history.kafka.bootstrap.servers" : "kafka:9092",
    "database.history.kafka.topic": "schema-changes.inventory",
"snapshot.mode": "schema_only"
}
}

Any idea of what's going on?

Regards

Chris Cranford

unread,
May 4, 2021, 2:00:23 PMMay 4
to debe...@googlegroups.com, Emanuele Meazzo
Hi Emanuele -

What is the DDL of the TESTTABLE, I suspect this has to do with the scale/precision of the ID field, but just want to check.

Thanks,
Chris
--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/50f2eb58-f00a-46e4-90af-442f7e428f22n%40googlegroups.com.

Emanuele Meazzo

unread,
May 4, 2021, 2:20:31 PMMay 4
to debezium
Hi Chris,

the ID it's a simple integer:

CREATE TABLE LOGMINER.TESTTABLE
(
  ID     INTEGER,
  VALUE  VARCHAR2(50 BYTE),
  SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS,
  SUPPLEMENTAL LOG DATA (ALL) COLUMNS
)
TABLESPACE LOGMINER_TBS
PCTUSED    0
PCTFREE    10
INITRANS   1
MAXTRANS   255
STORAGE    (
            INITIAL          64K
            NEXT             1M
            MINEXTENTS       1
            MAXEXTENTS       UNLIMITED
            PCTINCREASE      0
            BUFFER_POOL      DEFAULT
           )
LOGGING 
NOCOMPRESS 
ILM ADD POLICY ROW STORE COMPRESS ADVANCED ROW AFTER 1 MONTH OF NO MODIFICATION
NOCACHE;


Chris Cranford

unread,
May 4, 2021, 4:58:46 PMMay 4
to debe...@googlegroups.com, Emanuele Meazzo
Emanuele -

From what I can tell for a simple INTEGER data type, that would be emitted as a Decimal schema type with precision=38 and scale=0 with the value being a BigDecimal.  We get the values from LogMiner's REDO SQL, so just to cover all bases, could  you enable TRACE logging for the connector and perform the insert and see what the connector detects as the value for "ID"? 

Emanuele Meazzo

unread,
May 5, 2021, 1:18:54 PMMay 5
to debezium
Hey Chris,

I'm not sure I know how to do it, I used the following command as in the docs but I see no additional logging in the console output:

curl -s -X PUT -H "Content-Type:application/json" http://localhost:8083/admin/loggers/io.debezium.connector.oracle -d '{"level": "TRACE"}'

Am I doing something wrong? How can I get you the log?

Reply all
Reply to author
Forward
0 new messages