KSQL CrashLoopBackOff When Using New CA Certificates for Different Trust Chains

[Divya Prabhu]

Hi Team

Issue: KSQL CrashLoopBackOff When Using New CA Certificates for Different Trust Chains

Description

We encountered an issue where replacing KSQL's SSL certificates with a new set signed by a different Certificate Authority (CA) causes KSQL to enter a CrashLoopBackOff state. The issue arises because Kafka brokers and KSQL use one trust chain, while KSQL’s REST API uses a different trust chain.

When using a new CA for either trust chain, KSQL fails to start. However, replacing certificates while keeping the same CA for both trust chains works without issues.

Environment Details

• KSQL Version:7.7.1
• Kafka Version: 7.7.1
• Deployment Method: Helm
• SSL Configuration:
  • Kafka Broker to KSQL: Uses Trust Chain A
  • KSQL to KSQL REST API: Uses Trust Chain B
  • Previous working certificates: Both chains signed by CA1

Reproduction Steps

1. Deploy KSQL with existing SSL certificates (signed by CA1 for both trust chains) – works fine.
2. Generate new certificates:
   • Kafka broker-to-KSQL SSL communication → Trust Chain A (CA1)
   • KSQL to KSQL REST API SSL communication → Trust Chain B (CA2)
3. Update the keystore and truststore with the new certificates.
4. Restart KSQL – it enters CrashLoopBackOff.

Troubleshooting Done

• Verified that both new CAs are correctly added to the truststore.
• Ensured that the correct truststore/keystore  locations are mounted and referenced:
  • ssl.keystore.location=/etc/kafka/shared/restKeystore
  • ksql.streams.ssl.keystore.location=/etc/ksql/ssl/kafka/keyStore
  • ssl.truststore.location=/etc/kafka/shared/restTruststore
  • ksql.streams.ssl.truststore.location=/etc/ksql/ssl/kafka/trustStore

Expected Behavior

KSQL should start successfully with any valid CA-signed certificate, provided the keystore and truststore are correctly updated for both Kafka and KSQL REST communications.

Actual Behavior

KSQL crashes when using a new CA-signed certificate for either trust chain, but works fine when using a different certificate signed by the original CA.

Request for Help

Has anyone faced this issue before? Are there additional steps needed to reload certificates when using separate trust chains for Kafka and KSQL REST? Any insights or workarounds would be greatly appreciated!

Thanks & Regards

Divya