I can't ingest data from Amazon MSK with SSL, please help

[Marco Villalobos]

I'm having trouble ingesting data on Amazon MSK with SSL.

First, I did a test locally with a Kafka client that could retrieve data from Kafka over SSL.

I created a client.properties file as such:

security.protocol=SSL
ssl.truststore.location=$JDK_HOME/lib/security/cacerts
ssl.truststore.password=changeit

Then I tested the connectivity and received  data:

kafka-console-consumer --bootstrap-server <bootstrap servers with port> --consumer.config client.propertie --topic my-kafka-topic --from-beginning

However, when I pass along these properties to my 

"ioConfig": {

   "consumerProperties": {

       "bootstrap.servers": "bootstrap servers with port",

        "security.protocol": "SSL"
        "ssl.truststore.location": "$JDK_HOME/lib/security/cacerts"

        "ssl.truststore.password": "changeit"
   }

}

in my druid ingestion spec, well, it fails to ingest data.

Please note that $JDK_HOME is an actual path to my JDK

Also, I use the root signed JDK cacerts file because it contains the AWS certificate. 

Also, I tested my configuration with Kafka running in Docker with no SSL and it all worked fine. However, my deployment environment requires SSL.

Any help would be deeply appreciated.

[Peter Marshall]

Are there any hints in the supervisor or task logs out of interest?

[Marco Villalobos]

It finally started working after I used the following consumer property:  isolation.level = read_uncommitted.

I'm not sure why that fixed this.

Thank you.

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/77ba1ee5-f73c-4265-a090-2295166d1eb4n%40googlegroups.com.