Deploying Debezium On EKS

[Rohit Panwar]

Hi team, i have doubts regarding deploying debezium on EKS(Kubernetes). I am completely new to this so pardon me if some of the doubts are very basic. I already have AWS MSK running for kafka which is not part of my eks cluster.

Below is my dockerfile which is being built and pushed to ECR. I have a kuberntes config which is also attached below, as soon as my pod starts it throws a log4j error stating the file is not found and goes in a crash loop state. Can you guide me on how to acheive this? Can we do this without having to install strimzi?

FROM quay.io/strimzi/kafka:latest-kafka-3.7.0

USER root

# Add Debezium MySQL connector

ADD https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/2.6.1.Final/debezium-connector-mysql-2.6.1.Final-plugin.tar.gz /tmp/

# Unpack plugin

RUN mkdir -p /opt/kafka/plugins/debezium-mysql && \

tar -xvf /tmp/debezium-connector-mysql-*.tar.gz -C /opt/kafka/plugins/debezium-mysql && \

rm -rf /tmp/*.tar.gz

# Copy smt-jars directory (if it exists) and then move any jar files

COPY ./smt-jars /tmp/smt-jars/

# Create directory and copy any JAR files if they exist

RUN mkdir -p /opt/kafka/plugins/custom-smt/ && \

find /tmp/smt-jars -name "*.jar" -exec cp {} /opt/kafka/plugins/custom-smt/ \; 2>/dev/null || true && \

rm -rf /tmp/smt-jars

USER 1001

======= Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
name: lsq-dp-db-connector-debezium
namespace: lsq-dp-db-connector
spec:
replicas: 1
selector:
matchLabels:
app: lsq-dp-db-connector-debezium
template:
metadata:
labels:
app: lsq-dp-db-connector-debezium
spec:
nodeSelector:
type: lsq-dp-db-connector-spot-nodepool
containers:
- name: kafka-connect
image: <ECR REPO>

command: ["/opt/kafka/kafka_connect_run.sh"]
ports:
- containerPort: 8083
name: rest-api
env:
# Kafka Cluster Connection
- name: KAFKA_CONNECT_BOOTSTRAP_SERVERS
value: "abc:9092"(dummy value)
# Kafka Connect Cluster Identity
- name: KAFKA_CONNECT_GROUP_ID
value: "lsq-dp-connect-cluster" # Unique ID for this Connect cluster

# Internal Storage Topics (Kafka Connect stores its state in Kafka)
- name: KAFKA_CONNECT_CONFIG_STORAGE_TOPIC
value: "lsq-dp-connect-configs" # Stores connector configurations
- name: KAFKA_CONNECT_OFFSET_STORAGE_TOPIC
value: "lsq-dp-connect-offsets" # Stores source offsets (critical for Debezium)
- name: KAFKA_CONNECT_STATUS_STORAGE_TOPIC
value: "lsq-dp-connect-status" # Stores connector health/status

# Replication Factor
- name: KAFKA_CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR
value: "2"
- name: KAFKA_CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR
value: "2"
- name: KAFKA_CONNECT_STATUS_STORAGE_REPLICATION_FACTOR
value: "2"

# Default Converters (how data is serialized)
- name: KAFKA_CONNECT_KEY_CONVERTER
value: "org.apache.kafka.connect.json.JsonConverter"
- name: KAFKA_CONNECT_VALUE_CONVERTER
value: "org.apache.kafka.connect.json.JsonConverter"
- name: KAFKA_CONNECT_KEY_CONVERTER_SCHEMAS_ENABLE
value: "false" # Debezium includes schema in the message
- name: KAFKA_CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE
value: "false"

# Additional Debezium Configuration
- name: KAFKA_CONNECT_INCLUDE_SCHEMA_CHANGES
value: "false"
- name: KAFKA_CONNECT_DATABASE_HISTORY_SKIP_UNPARSEABLE_DDL
value: "true"
- name: KAFKA_CONNECT_DATABASE_HISTORY_STORE_ONLY_CAPTURED_TABLES_DDL
value: "true"
- name: KAFKA_CONNECT_PLUGIN_PATH
value: "/opt/kafka/plugins"

# Logging Configuration
- name: KAFKA_CONNECT_LOG4J_ROOT_LOGLEVEL
value: "INFO"
- name: KAFKA_CONNECT_LOG4J_LOGGERS
value: "org.apache.kafka.connect.runtime.rest=WARN"

# REST API Configuration
- name: KAFKA_CONNECT_REST_ADVERTISED_HOST_NAME
value: "lsq-dp-db-connector-debezium"

# Performance Tuning (optional but recommended)
- name: KAFKA_CONNECT_PRODUCER_COMPRESSION_TYPE
value: "snappy" # Compress data sent to Kafka
- name: KAFKA_CONNECT_CONSUMER_MAX_POLL_RECORDS
value: "500"

# AWS MSK Specific (if using IAM authentication)
# - name: CONNECT_SECURITY_PROTOCOL
# value: "SASL_SSL"
# - name: CONNECT_SASL_MECHANISM
# value: "AWS_MSK_IAM"
# - name: CONNECT_SASL_JAAS_CONFIG
# value: "software.amazon.msk.auth.iam.IAMLoginModule required;"
# - name: CONNECT_SASL_CLIENT_CALLBACK_HANDLER_CLASS
# value: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
readinessProbe:
httpGet:
path: /
port: 8083
initialDelaySeconds: 30
periodSeconds: 10
livenessProbe:
httpGet:
path: /
port: 8083
initialDelaySeconds: 60
periodSeconds: 30
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: 2
memory: 2Gi