Kafka REST Proxy in SSL mode issue

[Nishit Sinha]

Hi,

Right now I am digging around and evaluation the Kafka REST Proxy. It appears working fine in normal http mode. But, I got stuck while trying to configure it with SSL i.e. https.

Let me tell explain about my configuration.

Server 1. Has zookeeper running on port 2181 (using vanilla Kafka)

Server 2. Has Apache's Kafka running on SSL (using vanilla Kafka)

Server 3. Using Confluent's Kafka REST Proxy.

I am able to do all the operations through Kafka REST Proxy in http mode i.e. 

http://localhost:8082/topics

But, when I am trying to access the same with 

https://localhost:8083/topics

It throws me the following exception

[2017-10-18 11:50:56,590] DEBUG onClose HttpConnection@19687300[FILLING,DecryptedEndPoint@48487d03{10.120.192.59/10.120.192.59:49653<->8083,CLOSED,ISHUT,OSHUT,-,-,312/30000,HttpConnection}->NetworkTrafficSelectChannelEndPoint@43d14326{10.120.192.59/10.120.192.59:49653<->8083,CLOSED,ISHUT,OSHUT,-,-,7/30000,SslConnection}{io=0,kio=-1,kro=-1}][p=HttpParser{s=START,0 of 0},g=HttpGenerator{s=START},c=HttpChannelOverHttp@222324cd{r=0,c=false,a=IDLE,uri=}] (org.eclipse.jetty.io.AbstractConnection:210)
[2017-10-18 11:50:56,590] DEBUG onClose SslConnection@4d7f19ae{NEED_WRAP,eio=190/-1,di=-1} -> HttpConnection@19687300[FILLING,DecryptedEndPoint@48487d03{10.120.192.59/10.120.192.59:49653<->8083,CLOSED,ISHUT,OSHUT,-,-,313/30000,HttpConnection}->NetworkTrafficSelectChannelEndPoint@43d14326{10.120.192.59/10.120.192.59:49653<->8083,CLOSED,ISHUT,OSHUT,-,-,8/30000,SslConnection}{io=0,kio=-1,kro=-1}][p=HttpParser{s=START,0 of 0},g=HttpGenerator{s=START},c=HttpChannelOverHttp@222324cd{r=0,c=false,a=IDLE,uri=}] (org.eclipse.jetty.io.AbstractConnection:210)
[2017-10-18 11:50:56,591] DEBUG onClose NetworkTrafficSelectChannelEndPoint@43d14326{10.120.192.59/10.120.192.59:49653<->8083,CLOSED,ISHUT,OSHUT,-,-,8/30000,SslConnection}{io=0,kio=-1,kro=-1} (org.eclipse.jetty.io.AbstractEndPoint:108)
[2017-10-18 11:50:56,591] DEBUG SslConnection@4d7f19ae{NEED_WRAP,eio=190/-1,di=-1} -> HttpConnection@19687300[FILLING,DecryptedEndPoint@48487d03{10.120.192.59/10.120.192.59:49653<->8083,CLOSED,ISHUT,OSHUT,-,-,314/30000,HttpConnection}->NetworkTrafficSelectChannelEndPoint@43d14326{10.120.192.59/10.120.192.59:49653<->8083,CLOSED,ISHUT,OSHUT,-,-,9/30000,SslConnection}{io=0,kio=-1,kro=-1}][p=HttpParser{s=START,0 of 0},g=HttpGenerator{s=START},c=HttpChannelOverHttp@222324cd{r=0,c=false,a=IDLE,uri=}] fill exit (org.eclipse.jetty.io.ssl.SslConnection:690)
[2017-10-18 11:50:56,592] DEBUG  (org.eclipse.jetty.server.HttpConnection:275)
javax.net.ssl.SSLHandshakeException: null cert chain
        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1478)
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:520)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:239)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: null cert chain
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:292)
        at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1862)
        at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:233)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:620)
        ... 5 more

The basic configuration I have used for SSL is as follows:

ssl.keystore.location=/opt/confluent-3.3.0/keystore1/kafka.server.keystore.jks

ssl.keystore.password=kafkabroker
ssl.key.password=*******
ssl.truststore.location=/opt/confluent-3.3.0/keystore1/kafka.server.truststore.jks
ssl.truststore.password=*******
ssl.client.auth=true
ssl.keystore.type=JKS
ssl.truststore.type=JKS
security.inter.broker.protocol=SSL
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
#Configuration options for SSL Encryption between REST Proxy and Apache Kafka Brokers
#Protocol used to communicate with brokers. Valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL
client.security.protocol=SSL
#Password of the private key in the key store file. (Optional)
client.ssl.key.password=**********
client.ssl.keystore.location=/opt/confluent-3.3.0/keystore/kafka.client.keystore.jks
client.ssl.keystore.password=**********
client.ssl.truststore.location=/opt/confluent-3.3.0/keystore/kafka.client.truststore.jks
client.ssltruststore.password=***********
client.ssl.enabled.protocols=TLSv1.2
client.ssl.keystore.type=JKS
client.ssl.protocol=TLS
#client.ssl.provider=
client.ssl.truststore.type=JKS

Also, listener configuration and others are defined properly. Is there anything which I have missed or something I did wrong.

Kindly, help me to configure it in the desired manner.

[mag...@confluent.io]

The SSL handshake is all handled by Jetty server under the hood. From the exception, it appears that your keystore probably doesn't have the complete certificate chain. You would either need to have a CA cert or you need a self signed cert.

[Nishit Sinha]

Now its working fine. The resolution is, I have changed the value of the ss.client.auth property to false.

The issue was present till the value of it was true.

[mag...@confluent.io]

Thats good to know. Since you had the ssl.client.auth property to true I thought you wanted to do a mutual auth.

[David I Rock]

@Nishit could you explain me how what parameters are you using when you are doing CURL to https://localhost:8083/topics pls?

[David I Rock]

@Nishit could you explain me what parameters are you using when you are doing CURL to https://localhost:8083/topics pls?