Problems with batching in KairosDB and Cassandra

[Rosita Hormann Lobos]

Hello all,

I am having concurrent error and warning logs in both my KairosDB service and in my Cassandra database.

The error I have on the KairosDB Query instance is attached in the file "kairosdb-query.log" (is quite long)

Here's the warnings I am having in the KairosDB Ingest instance:

16:36:37.371 [cluster1-nio-worker-44] WARN [RequestHandler.java:251] - Query '[182 statements, 910 bound values] BEGIN UNLOGGED BATCH INSERT INTO data_points (key, column1, value) VALUES (?, ?, ?) USING TTL ? AND TIMESTAMP ?; INSERT INTO data_points (key, column1, value) VALUES (?, ?, ?) USING TTL ? AND TIMESTAMP ?; INSERT INTO data_points (key, column1, value) VALUES (?, ?, ?) USING TTL ? AND TIMESTAMP ?; INSERT INTO data_points (key, column1, value) VALUES (?, ?, ?) USING TTL ? AND TIMESTAMP ?; INSERT INTO data_points (key, column1, value) VALUES (?, ?, ?) USING TTL ? AND TIMESTAMP ?; INSERT INTO data_points (key... [truncated output]' generated server side warning(s): Unlogged batch covering 38 partitions detected against table [kairosdb.data_points]. You should use a logged batch for atomicity, or asynchronous writes for performance.

And here is the warning that I am having in Cassandra:

WARN [Native-Transport-Requests-4] 2025-08-12 16:36:59,517 BatchStatement.java:362 - Batch for [kairosdb.data_points] is of size 28.951KiB, exceeding specified threshold of 5.000KiB by 23.951KiB.

WARN [Native-Transport-Requests-2] 2025-08-12 16:36:59,720 BatchStatement.java:362 - Batch for [kairosdb.data_points] is of size 24.951KiB, exceeding specified threshold of 5.000KiB by 19.951KiB.

WARN [Native-Transport-Requests-4] 2025-08-12 16:36:59,721 BatchStatement.java:362 - Batch for [kairosdb.data_points] is of size 14.820KiB, exceeding specified threshold of 5.000KiB by 9.820KiB.

WARN [Native-Transport-Requests-2] 2025-08-12 16:36:59,876 BatchStatement.java:362 - Batch for [kairosdb.data_points] is of size 12.154KiB, exceeding specified threshold of 5.000KiB by 7.154KiB.

WARN [Native-Transport-Requests-4] 2025-08-12 16:36:59,892 BatchStatement.java:362 - Batch for [kairosdb.data_points] is of size 22.762KiB, exceeding specified threshold of 5.000KiB by 17.762KiB.

WARN [Native-Transport-Requests-1] 2025-08-12 16:37:00,267 BatchStatement.java:362 - Batch for [kairosdb.row_key_time_index, kairosdb.row_keys] is of size 29.042KiB, exceeding specified threshold of 5.000KiB by 24.042KiB.

Specially in Cassandra, these are quite frequent.

I investigated a bit, and it seems to be related to the batching configuration in both Cassandra and KairosDB. But I am not sure how to fix it, I am not familiar at all with this concept.

Here are some of the configurations I think are related to batching:

in kairosdb.conf:

row_key_cache_size: 50000
string_cache_size: 50000

                batch_size: 10000

min_batch_size: 100

min_batch_wait: 500

memory_queue_size: 100000

                page_size: 52428800

in cassandra.yaml:

batchlog_replay_throttle_in_kb: 1024

batch_size_warn_threshold_in_kb: 125
batch_size_fail_threshold_in_kb: 150
unlogged_batch_across_partitions_warn_threshold: 100
compaction_large_partition_warning_threshold_mb: 100

How can I improve my deployment to avoid this warning and errors from occurring? Any help is highly appreciated.