AVRO schema backward compatibility check

[Iván San José]

Hi, to he honest I'm nor really sure if this is a Debezium problem or Schema Registry problem or is me who don't understand how AVRO schema compatibility works, but I'm experiencing a really weird issue... Let me give you some context:
 - I'm using a Kafka Connect connector with Debezium to ingest data from a SQL Server database table which only has two columns: id (int) and name (string). So, initially the AVRO schema generated in my schema registry looks like:
{
  "type": "record",
  "name": "Envelope",
  "namespace": "example.lobby.dbo.categories",
  "fields": [
    {
      "name": "before",
      "type": [
        "null",
        {
          "type": "record",
          "name": "Value",
          "fields": [
            {
              "name": "id",
              "type": "int"
            },
            {
              "name": "name",
              "type": "string"
            }
          ],
          "connect.name": "example.lobby.dbo.categories.Value"
        }
      ],
      "default": null
    },
(more things here)
  - Note that I've set backward compatibility mode for this AVRO schema in schema-registry
  - Now, on database table I've deleted "name" column, added it again but as "integer", and updated CDC table to have Debezium taking the updated schema
  - My expectations after that change would be schema-registry rejecting the change as this change is clearly breaking backward compatibility, but is not refusing it, is storing new version of the schema with "name" field as "int" type.

Is this the expected behaviour? Is not that change breaking backward compatibility in stored AVRO schema?

I've also checked that any backward incompatible change made inside a AVRO union which has "null" as part of the union seems to be ignored in the compatibility check. But, if you do a change in any other field which is not inside an union type that can be null, then schema-registry refuses the change if is incompatible.

Thanks

[Gunnar Morling]

Do you see the new schema with integer being registered in the registry? I.e. is there a new schema version? If so, the connector pushes the new schema version to the registry and I'd indeed it to be rejected based on the compatibility configuration.

--Gunnar

[Iván San José]

Yes, I'm seeing the schema being registered in my schema-registry as a
new version of the subject, so is not being rejected by
schema-registry. So, then is something which has nothing to do with
Debezium? I mean, who should take care about schema compatibility?
Schema Registry right? Because may be I'm asking the question to wrong
group (sorry if this is the case).

El vie., 31 ene. 2020 a las 10:25, 'Gunnar Morling' via debezium
(<debe...@googlegroups.com>) escribió:

> --
> You received this message because you are subscribed to the Google Groups "debezium" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/a49e570a-55ca-4fd9-975f-a0d21d0e30c2%40googlegroups.com.

[Gunnar Morling]

Yes, registry should take care of it. Maybe double-check your compatibility mode? Tbh. I'd be surprised if it wouldn't catch that.

If you feel adventurous, you also could give the Apicurio registry a try as an alternative and see how it handles the case (see https://github.com/Apicurio/apicurio-registry).

It's a brand-new open source API and schema registry which can also be used with Kafka Connect. Documentation is still a bit sparse atm. (the team is working on that), but any feedback will be surely welcomed by them.

Best,

--Gunnar

> To unsubscribe from this group and stop receiving emails from it, send an email to debe...@googlegroups.com.

[Iván San José]

Yep, I've checked that compatibility mode is set to BACKWARD for that
subject, and, if the breaking change is inside a "record" field which
can be nullable like "before" and "after" values are coming using
Debezium, schema-registry is saying that schema is compatible when is
containing a breaking change...

Will try to setup Apicurio registry and check if its behaviour is the
same but I'm afraid we are tied to Confluent Schema Registry in the
company.

Thanks

El vie., 31 ene. 2020 a las 11:54, 'Gunnar Morling' via debezium

> To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/9125e5cb-fde5-48e4-a6b7-8674ecdfcf51%40googlegroups.com.