Setting up Debezium-server for Pubsub with Avro and Cloud events

[Samuel Roux]

Hi,

I'm trying to setup Debezium server to send crud events from PostgreSQL to PubSub. So far everything runs fine and I can stream cloud events from my DB to Pubsub with Json serialized message.

My issue is when I'm trying to setup avro serialization. For this kind of setup it seems mandatory to have a schema registry (Confluent or Apicurio) or at least I wasn't able to setup anything without these kind of component. 

My configuration file so far : 

debezium.sink.type=pubsub
debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector
debezium.source.offset.storage.file.filename=data/offsets.dat
debezium.source.offset.flush.interval.ms=0
debezium.source.database.hostname=localhost
debezium.source.database.port=5432
debezium.source.database.user=debezium_user
debezium.source.database.password=*****
debezium.source.database.dbname=upsides
debezium.source.database.server.name=data-db
debezium.source.table.include.list=public.influence_partnershipupsidepotential
debezium.source.plugin.name=pgoutput
debezium.source.topic.prefix=upsides

# Signaling
debezium.source.signal.data.collection=public.debezium_signal

# PubSub deletion events fix
debezium.source.tombstones.on.delete=false

# Topic rerouting
debezium.transforms=Reroute
debezium.transforms.Reroute.type=io.debezium.transforms.ByLogicalTableRouter
debezium.transforms.Reroute.topic.regex=(.*)
debezium.transforms.Reroute.topic.replacement=upsides.cloudevents.test

# CloudEventsConverter config
debezium.format.value=cloudevents
debezium.format.value.serializer.type=avro
debezium.format.value.data.serializer.type=json

# LOGS
quarkus.log.level=INFO

Is there any way to setup an Avro message converter without schema registry for Pubsub ? Or using Pubsub API as a schema registry maybe ?

[Chris Cranford]

Hi Samuel -

In theory, yes you can use Avro without a schema registry, but the end result is you will end up writing your own schema registry as part of your pipeline since you'll need to manage the schema as well as knowing how to deserialize/serialize the message.  Therefore, most people simply make use of the already existing schema registry providers as there isn't really any major gain in reinventing the wheel here.

Hope that helps.
Chris

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/8fda7066-c896-4bdc-94cb-207137a50d16n%40googlegroups.com.

[Samuel Roux]

Hi Chris,

Thank you for your answer but I think I don't get it when I setup my configuration file this way for Debezium-server :

[...]
debezium.format.value=cloudevents
debezium.format.value.serializer.type=avro
debezium.format.value.data.serializer.type=avro

[...]

I get this error at startup :

"Need URL(s) for schema registry instances for CloudEvents when using Apache Avro"

Do I miss something ?

Regards,

Samuel

[Chris Cranford]

Hi Samuel,

The base avro SerDes implementations require a schema registry; that's why I mentioned you would have to handle the SerDes coding yourself if you wanted to avoid a registry because all the implementations that I know of require the use of a registry.

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/8b8a2b0f-03af-4427-afa8-c2d351495ea8n%40googlegroups.com.