Connect and SSL'd Schema Registry, standalone works, distributed does not

[Marios Andreopoulos]

Hello,

I am on CP3.1.1. I've setup the Schema Registry with SSL. As funny as it may sound, Connect Standalone works fine with this setup. Connect Distributed fails to post the connectors' schemas to the registry and thus doesn't work.

When I disable SSL on the Registry and adjust Connect (key.converter.schema.registry.url and value.converter.schema.registry.url keys) both versions work correctly.

On Schema Registry's logs I can see that it returns HTTP 200 to the standalone version and HTTP 500 to the distributed.

I probably missed something but can't find it. My configurations are below. The brokers also use SSL and require SSL client auth.

Thanks for  looking at this.

Distributed Configuration:

config.storage.topic=connect-configs 

group.id=connect-cluster                                                                                                             

internal.key.converter=org.apache.kafka.connect.json.JsonConverter

internal.key.converter.schemas.enable=false

internal.value.converter=org.apache.kafka.connect.json.JsonConverter

internal.value.converter.schemas.enable=false

key.converter=io.confluent.connect.avro.AvroConverter 

offset.storage.topic=connect-offsets 

security.protocol=SSL

status.storage.topic=connect-status

value.converter=io.confluent.connect.avro.AvroConverter

rest.port=8083

ssl.key.password=reducted

ssl.keystore.location=reducted

ssl.keystore.password=reducted

ssl.truststore.location=reducted

ssl.truststore.password=reducted 

ssl.protocol=TLS

ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

ssl.keystore.type=JKS

ssl.truststore.type=JKS  

producer.security.protocol=SSL

producer.ssl.key.password=reducted

producer.ssl.keystore.location=reducted

producer.ssl.keystore.password=reducted

producer.ssl.truststore.location=reducted

producer.ssl.truststore.password=reducted 

producer.ssl.protocol=TLS

producer.ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

producer.ssl.keystore.type=JKS

producer.ssl.truststore.type=JKS

consumer.security.protocol=SSL

consumer.ssl.key.password=reducted 

consumer.ssl.keystore.location=reducted

consumer.ssl.keystore.password=reducted 

consumer.ssl.truststore.location=reducted

consumer.ssl.truststore.password=reducted 

consumer.ssl.protocol=TLS

consumer.ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

consumer.ssl.keystore.type=JKS

consumer.ssl.truststore.type=JKS

zookeeper.connect=zk1:2181,zk2:2181,zk3:2181/

bootstrap.servers=SSL://broker1:9093,SSL://broker2:9093,SSL://broker3:9093

key.converter.schema.registry.url=https://registry:8081

value.converter.schema.registry.url=https://registry:8081

Standalone Configuration:

bootstrap.servers=broker1:9093

security.protocol=SSL

key.converter=io.confluent.connect.avro.AvroConverter

key.converter.schema.registry.url=https://registry:8081

value.converter=io.confluent.connect.avro.AvroConverter

value.converter.schema.registry.url=https://registry:8081

internal.key.converter=org.apache.kafka.connect.json.JsonConverter

internal.value.converter=org.apache.kafka.connect.json.JsonConverter

internal.key.converter.schemas.enable=false

internal.value.converter.schemas.enable=false

offset.storage.file.filename=coyote_connect.offset

zookeeper=zk:2181/

rest.port=38081

port=38081

kafka.logs.dir=logs/

producer.security.protocol=SSL

consumer.security.protocol=SSL

ssl.keystore.location=reducted

ssl.keystore.password=changeit

ssl.key.password=reducted

producer.ssl.keystore.location=reducted

producer.ssl.keystore.password=reducted

producer.ssl.key.password=reducted

consumer.ssl.keystore.location=reducted

consumer.ssl.keystore.password=reducted

consumer.ssl.key.password=reducted

 

[Ewen Cheslack-Postava]

This is currently a limitation in schema registry, we have a bug filed for it here: https://github.com/confluentinc/schema-registry/issues/386 The internal requests between instances have not been converted to support SSL yet which is why they would currently fail if you enable SSL in a multi-node cluster.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/1cf84162-b5da-40bf-ad5e-d26aab57cfaa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

[Marios Andreopoulos]

Thank you! I added a http listener to the registries and everything works now, whilst connect still uses the https listener.

I searched but missed this github issue.

Thanks again,

Marios.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/1cf84162-b5da-40bf-ad5e-d26aab57cfaa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen