Very less TPS for inserts

[Sandeep khanzode]

Hello,

Just to mention: I am new to Citus DB.

On an AWS EKS cluster, I have a small 3 node cluster with 1 master and 2 workers. I agree that the nodes are small, like, t3 large with 8GB RAM.

But when I push data from Kafka to a Java microservice, I batch them into groups of 2K-5K records and send them for a PreparedStatement Batch Insert for a on-conflict-do-update statement.

Through performance testing, I have checked that the Java microservice is not lagging and the only time it is waiting is when the batch is invoked. It is very light on the CPU because all it is really doing is I/O stuff of pulling from kafka and pushing to Citus.

I get really low TPS of 5-6 TPS. If I restart everything, it start with approx 32 TPS, works for about 75K records or about 40 minutes or so, and then the CPU and memory usage for that Java Pod drops significantly and then the TPS settles forever after that at 5-6 TPS.

http://docs.citusdata.com/en/v10.0/performance/performance_tuning.html?highlight=scaling#scaling-out-data-ingestion

After reading this, I am sure I am doing something wrong, but dont know what.

Kindly help me start with some configuration or checks or tools that I can use to start understanding and debugging why I am getting such low numbers. 

The shared memory setting seems alright though.

Thanks,

Sandip

[Burak Yucesoy]

Hi Sandeep,

Other Citus engineers/users can give more tuning tips but there are two quick wins that you can use;

• Using COPY instead of INSERTs: If possible, try using COPY instead of multi-row INSERTs. COPY is already order of magnitude faster than multi-row INSERTs in PostgreSQL and its impact is even more apparent with Citus.
• Changing instance types: t series in AWS are burstable instances. They have some amount of bursting quota (where you can use full system resources) but after your quota runs out, their performance drops drastically. Then you need to wait for your quota to replenish. Their use-case is workloads that need high resource consumption only short period of time and needing minimal resources rest of the time (Think about an idle database that generates a one daily report and needs high resource consumption during that). For t3.large, you get use only 30% of the CPU after you run out your quota.

Thanks,

Burak

---

From: citus...@googlegroups.com <citus...@googlegroups.com> on behalf of Sandeep khanzode <san...@shiftright.ai>
Sent: Tuesday, June 1, 2021 08:34
To: citus-users <citus...@googlegroups.com>
Subject: [EXTERNAL] Very less TPS for inserts

 

--
You received this message because you are subscribed to the Google Groups "citus-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to citus-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/citus-users/da5cedca-24b5-4539-bb46-b36c2e39ebbfn%40googlegroups.com.