Schema evolution in Debezium

[amit...@venmo.com]

Hi all,

I was wondering how (if at all) Debezium works with Kafka Connect's schema evolution - https://docs.confluent.io/current/avro.html#schema-evolution

What happens if the schema changes upstream (i.e., MySQL add/remove column)? how would the event value be affected?

Thanks,

Amit 

[Andrew Ehrlich]

I came to ask this question too.

Suppose the schema registry is set to be fully backwards and forwards compatible. What will happen if the upstream database makes non-compatible changes to its tables? 

I would like to run a schema registry with strict compatibility rules. Could it be that the value for the upstream record value is simply a string inside the debezium avro schema, thus preventing the upstream db from breaking the pipeline, yet keeping avro in the rest of the message?

[Chris Riccomini]

Yes, this is an issue. I have written about it here:

https://riccomini.name/kafka-change-data-capture-breaks-database-encapsulation

It is up to you to decide how to manage it. By default, if you have compatibility checks enabled on your schema registry, and you mutate the DB schema in an incompatible way, all future writes from DBZ to Kafka will fail as it tries to register the new schema in the schema registry, and it is rejected by the compatibility checks. Not great.

[Gunnar Morling]

Hey,

Yeah, it's an important topic, which I'm planning to cover in some docs, too (started to collect some ideas in https://issues.jboss.org/browse/DBZ-973).

Chris' post has some great advice on this. For simple cases I recommend you use an SMT as some means of anti-corruption layer: e.g. it could turn the schema of a new mandatory column into optional (if you're working with full compatibility checking), it could re-add a deleted column with some default value etc.

--Gunnar

[Andrew Ehrlich]

Thank you. Another idea, on the schema registry side, is if compatibility could be specified at the subject level rather than globally. This would only solve the issue of having a non-compatible debezium topic co-habitating with fully compatible topics on the same cluster. On my current project, I will probably let debezium serialize to JSON, and then handle the schema changes slowly in batch ETLs.

[Gunnar Morling]

> compatibility could be specified at the subject level rather than globally

Yes, that seems like a reasonable idea. I'd suggest to open a feature request for the registry. Did you have a chance to explore whether SMTs could help in your case?

>  Could it be that the value for the upstream record value is simply a string inside the debezium avro schema

It's not supported out of the box, but also should be doable with an SMT. I'm wondering though how much you gain; granted it'd avoid issues with the registry, but the problem only gets moved to another place: the consumers that need to deal with that string, whose format may change over time.

[Randall Hauch]

On Fri, Dec 21, 2018 at 3:43 PM 'Gunnar Morling' via debezium <debe...@googlegroups.com> wrote:

> compatibility could be specified at the subject level rather than globally

Yes, that seems like a reasonable idea. I'd suggest to open a feature request for the registry.

Confluent’s Schema Registry already has a global compatibility setting and subject-level compatibility settings that override the global. See 

https://docs.confluent.io/current/schema-registry/docs/avro.html and 

https://docs.confluent.io/current/schema-registry/docs/api.html

Hope that helps!

Randall

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To post to this group, send email to debe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/3d816ad2-ba01-41b6-8495-f02af44139bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gunnar Morling]

Ah, that's nice. Hadn't found it while quickly checking the registry docs. Thanks for clarifying!