Hello,
Just to mention: I am new to Citus DB.
On an AWS EKS cluster, I have a small 3 node cluster with 1 master and 2 workers. I agree that the nodes are small, like, t3 large with 8GB RAM.
But when I push data from Kafka to a Java microservice, I batch them into groups of 2K-5K records and send them for a PreparedStatement Batch Insert for a on-conflict-do-update statement.
Through performance testing, I have checked that the Java microservice is not lagging and the only time it is waiting is when the batch is invoked. It is very light on the CPU because all it is really doing is I/O stuff of pulling from kafka and pushing to Citus.
I get really low TPS of 5-6 TPS. If I restart everything, it start with approx 32 TPS, works for about 75K records or about 40 minutes or so, and then the CPU and memory usage for that Java Pod drops significantly and then the TPS settles forever after that at 5-6 TPS.
After reading this, I am sure I am doing something wrong, but dont know what.
Kindly help me start with some configuration or checks or tools that I can use to start understanding and debugging why I am getting such low numbers.
The shared memory setting seems alright though.
Thanks,
Sandip