Subject: Segment allocation throughput ceiling with high-rate single-datasource batch ingestion (36.0.0) — tuning advice & load-test experiences wanted

[Shashikanth G K]

  Hi all,
 
  We're running a high-rate batch ingestion workload and have hit what appears to be a per-datasource segment allocation bottleneck. I'd appreciate a sanity check on our diagnosis and advice on next steps — and I'd especially like to hear from anyone who has load-tested batch segment allocation tuning at similar rates.
 
  Setup
 
  - Druid 36.0.0, on EC2 (based on the AWS "scalable analytics" solution)
  - Metadata store: PostgreSQL (RDS), healthy — sub-ms write latency
  - Overlord: 24 GB heap, httpRemote runner
  - ~100 MiddleManager nodes × druid.worker.capacity=2 → ~200 ingestion slots (plus a separate worker category for compaction, via equalDistributionWithCategorySpec)
  - Workload: ~200 index_parallel tasks/minute, each ingesting one small S3 file, all appending to a single datasource with HOUR segment granularity, dynamic partitioning, maxNumConcurrentSubTasks=1. An upstream consumer submits tasks from S3-event notifications and polls status.
 
  Relevant Overlord config
 
  druid.indexer.runner.type=httpRemote
  druid.indexer.runner.pendingTasksRunnerNumThreads=20
  druid.indexer.tasklock.batchSegmentAllocation=true
  druid.indexer.tasklock.batchAllocationWaitTime=500
  druid.server.http.numThreads=150
  druid.indexer.queue.maxSize=2500
 
  What we've observed
 
  1. Originally on the ZK remote runner, the single rtr-pending-tasks-runner-0 thread was pinned at 100% CPU in ZkWorker.getRunningTasks Jackson deserialization — assignment ceiling ~150–180 tasks/min, below our 200/min arrival, so pending grew without bound. Switching to httpRemote fixed assignment throughput.
  2. With assignment fixed, healthy task wall-time is ~12–25 s, but under load it degrades to ~66 s+. Phase breakdown of a sample task (99 s total): ~11 s peon startup, 56 s waiting on segment allocate, ~3 s build/push, 32 s waiting on publish. So ~90% of task time is queued on Overlord metadata operations, not on MM work.
  3. The hot hour-interval accumulates ~1,800 partitions (dynamic shards) — one per task — so allocation transactions scan an increasingly large pendingSegments/ segments set as the hour progresses.
  4. From reading SegmentAllocationQueue in the 36 source: batches for a single datasource execute strictly sequentially regardless of batchAllocationNumThreads, so a single hot datasource can't be parallelized — the per-datasource batch rate × batch size is the hard ceiling.
 
  What we tried that went badly
 
  We enabled druid.manager.segments.useIncrementalCache=ifSynced (Overlord + Coordinator) together with batchAllocationWaitTime=3000 and batchAllocationReduceMetadataIO=true. Shortly after the restart we saw a  wave of task failures with Cannot allocate segment for timestamp[...] → ISE: Failed to add a row. We've since attributed at least part of that to a restart-induced backlog plus our client's aggressive task-kill policy (kills anything unfinished 5 min after creation, including queue time — a race where killed tasks lose their locks mid-run and allocation returns null). We reverted the cache. Question: is the incremental segment metadata cache considered usable at this scale in 36.0.0, or should it wait for 37 where it's default?
   
  Questions for the group
 
  1. Has anyone load-tested batchSegmentAllocation (batchAllocationWaitTime, batch size behavior) at ≥200 allocations/min on a single datasource? What allocation throughput ceiling did you observe, and what settings worked?
  2. Are there known strategies for scaling allocation for one hot datasource — beyond reducing task count — given per-datasource batches are serial? (Splitting into multiple datasources is unattractive for query reasons.)
  3. Is ~1,800 dynamic partitions in an open hour interval itself a known allocation-cost amplifier, and does anything besides compaction (which can't touch the open hour) mitigate it?
  4. Any experience with useIncrementalCache on 36.0.0 in production, positive or negative?
 
  Any pointers to prior threads, benchmarks, or design docs are much appreciated. Happy to share more config or metrics.
 
  Thanks!