Configuring MirrorMaker with Avro schema registry

[gvdm]

Hi all! 

I'm installing a MirrorMaker which will allow me to replicate one cluster data to one another cluster. Both on the source and on the target clusters I'm using the Confluent Avro schema registry and the data is binarized with Avro.

I'm using the latest released version of Confluent 3.3.0 (kafka 0.11). Moreover, the source broker is on a Windows machine while the target broker is on a Linux machine.

Now I'm writing the consumer and producer property files, but I don't know how to specify the target and source Avro schema registry endpoint. Do you have any complete example for MirrorMaker configuration with Avro schema registry?

This are my config files at the moment

consumer.properties

group.id=test-mirrormaker-group
bootstrap.servers=host01:9092
exclude.internal.topics=true
client.id=mirror_maker_consumer0
auto.commit.enabled=false


# Avro schema registry properties
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://host01:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://host01:8081


internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false

producer.properties

bootstrap.servers=host02:9093
compression.type=none
acks=1
client.id=mirror_maker_producer0


# Avro schema registry properties
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://host02:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://host02:8081


internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false

I run the MirrorMaker on the host01 Windows machine with this command

C:\kafka>.\bin\windows\kafka-mirror-maker.bat --consumer.config .\etc\kafka\consumer.properties --producer.config .\etc\kafka\producer.properties --whitelist=MY_TOPIC
[2017-09-26 10:09:58,555] WARN The configuration 'internal.key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2017-09-26 10:09:58,555] WARN The configuration 'value.converter.schema.registry.url' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2017-09-26 10:09:58,571] WARN The configuration 'internal.key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2017-09-26 10:09:58,586] WARN The configuration 'internal.value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2017-09-26 10:09:58,602] WARN The configuration 'internal.value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2017-09-26 10:09:58,633] WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2017-09-26 10:09:58,649] WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2017-09-26 10:09:58,649] WARN The configuration 'key.converter.schema.registry.url' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2017-09-26 10:09:58,727] WARN The configuration 'internal.key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2017-09-26 10:09:58,727] WARN The configuration 'value.converter.schema.registry.url' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2017-09-26 10:09:58,727] WARN The configuration 'internal.key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2017-09-26 10:09:58,742] WARN The configuration 'auto.commit.enabled' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2017-09-26 10:09:58,774] WARN The configuration 'internal.value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2017-09-26 10:09:58,789] WARN The configuration 'internal.value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2017-09-26 10:09:58,805] WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2017-09-26 10:09:58,805] WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2017-09-26 10:09:58,821] WARN The configuration 'key.converter.schema.registry.url' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)

Using the topic UI utility (https://github.com/Landoop/kafka-topics-ui) I can see that on the target broker the data is sent, but it is not shown properly and I think this is caused by the misconfiguration of the schema registry.

[gvdm]

Additions:
The two Kafka clusters are independent, thus they have different schema registries; in the future we will have several source kafka clusters, mirrored on in the same target Kafka cluster. It would be easy if we could use the same schema registry for the different Kafka clustered being mirrored but this is unfortunately not possible because the Kafka installations are independent. 

[gvdm]

It seems that the MirrorMaker producer serializes both key and value data with the ByteArraySerializer, so it ignores the Avro Schema registry case...

https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/MirrorMaker.scala#L237

I don't see no solution at this point, except if the Kafka MirrorMaker will read also the key/value serialization class parameters for producer and consumer, allowing to configue the Avro schema serializer.

What do you think? It this analysis right? Is there any solution/workaround I can perform except of using the same schema registry for all the Kafka clusters (which is not an option for network problems)?