Kafka Connect: Are there multiple SSL truststores?

[Ben Simmons]

Hello!

I have a question about how SSL truststores are configured in Kafka Connect.

How is it that I can configure my Kafka Connect worker's truststore with only one CA cert (the one to verify broker SSL connections), but can still securely connect with my database via a JDBC source connector and with S3 via an S3 sink connector? Surely connecting to S3 via SSL requires another CA cert?

Does Kafka Connect have a concept of separate truststores for SSL with brokers vs. SSL with everything else? Do Kafka clients in general have this concept of multiple truststores?

My concern is that I'm using Heroku Kafka, and all their EC2 broker instances use the same SSL certificate, so I can't use hostname verification from my Kafka Connect workers. I've confirmed with Heroku that to prevent any possibility of interception & decryption of my traffic to the brokers, I have to ensure there is only one CA certificate used to verify the broker connections: Heroku's kafka broker CA. My understanding is that the way to do this in Java is to put only that CA cert in my truststore. The problem is that, being Kafka Connect, I want to connect with services outside of Kafka, so I need multiple CAs to verify the certificates of the different endpoints.

For reference, I use the same truststore file for all of CONNECT_SSL_TRUSTSTORE_LOCATION, CONNECT_PRODUCER_SSL_TRUSTSTORE_LOCATION, and CONNECT_CONSUMER_SSL_TRUSTSTORE_LOCATION.

I'm guessing this problem has already been solved in Kafka code, but if anyone could point me to the relevant source code I would greatly appreciate it.

Thanks,

Ben Simmons

[Bibin Sebastian]

Is this a restriction from heruku's side rather than a generic kafka issue? 

From a general Java standpoint, if you need to trust connections from multiple SSL endpoints, I believe you need to add all the relevant CA certificates in your Java trust store to trust the certificates from those SSL endpoints. So going by this, if you can add your S3 endpoint's certificate (or CA certificate) into Java truststore (used by kafka connect), I think that should resolve the issue.

-bibin