Connect: Bootstrap broker disconnected and high memory usage

[Thijs Cadier]

Hi All,

I've been working on getting a complete Kafka/Streams/Connect setup running on staging this week. Everything is working smoothly now. The Kafka cluster is processing messages, the java process that's using the streams library is processing incoming messages. All connections run over SSL.

The connect instance which will flush data to MongoDB Is having some issues though. When I boot it everything connects and seems to be working. After a few minutes the logs are filled with messages like this for every broker:

[2016-07-07 15:34:03,783] WARN Bootstrap broker kafka1.staging.x.x:9093 disconnected (org.apache.kafka.clients.NetworkClient:568)

These warnings keep going endlessly. Another thing I noticed is that a heap size of at least 4 gigs is needed, otherwise connect is logging out of memory errors. I feel like I must be missing something obvious here. Any thoughts on this?

Thanks!

Thijs

[Thijs Cadier]

I've found and fixed the issue: I'd missed the part in the documentation about having separate SSL config for consumer and producer.

[Kåre Blakstad]

Where exactly did you find this in the documentation. I'm having the exact same problem myself.

[Thijs Cadier]

Not sure where I found the docs, but it comes down to adding the config three times:

# SSL configuration for Kafka client

security.protocol=SSL

ssl.keystore.location=***

ssl.keystore.password=***

ssl.truststore.location=***

ssl.truststore.password=***

# SSL configuration for Kafka producer

producer.security.protocol=SSL

producer.ssl.keystore.location=***

producer.ssl.keystore.password=***

producer.ssl.truststore.location=***

producer.ssl.truststore.password=***

# SSL configuration for Kafka consumer

consumer.security.protocol=SSL

consumer.ssl.keystore.location=***

consumer.ssl.keystore.password=***

consumer.ssl.truststore.location=***

consumer.ssl.truststore.password=***