Question on heap and direct memory and fine tuning historical properties

[Adithya Shetty]

Hi all,
We are fine tuning few druid properties and had few doubts when going through docs, could someone please clarify1. Query processing buffers/threads

• druid.processing.buffer.sizeBytes
• druid.processing.numMergeBuffers
• druid.processing.numThreads

I want to understand how are these used when processing the druid queries, couldn't get much clarity from docs. Could anyone please explain these?I see there are formulas in the docs, but wanted to understand the logic behind them. For example, when we increase numThreads, more query processing and row scanning happens in parallel, but at some point groupBy/aggregation phases require merge buffers. If merge buffers are insufficient, does query processing block waiting for a merge buffer? What are the key factors to decide appropriate values for these?
https://druid.apache.org/docs/latest/configuration/#historical2. Heap vs direct memory allocation
Here,
https://druid.apache.org/docs/30.0.1/configuration/#segmentwriteoutmediumfactory
it mentions that for MiddleManagers, heap and direct memory should be the same size. Why is that requirement there?
Also, does this recommendation apply to Historicals as well? Thanks

[John Kowtko]

1. Yes, GroupBy query processing can be blocked on merge buffers.  Every query that is initialized for processing requires a Jetty thread ... and in addition any query that is a GroupBy also requires a Merge Buffer.   These are required before the query ca can begin processing ... until then it sits in the queue.    The proportion of your merge buffers to Jetty threads should correspond to the proportion of Group By to overall queries.   So if 100% of your queries are Group By, then your merge buffer count ideally should match the http.numThreads Jetty thread count.

processing.numThreads simply specifies how many Jetty threads the Historical should spawn to process queries.  these are "shared" threads and do not block queries ... but the more processing threads you have the more pieces of query processing can be done in parallel.

2. I can't speak to MM memory allocation ... if you are talking about Peons though, they do operated a bit like Historicals ... in which case there are considerations for setting both heap and direct memory sizing.

===

I hope the above helps a little bit ... probably brings up more questions ... let us know what you are still not clear on.

Thanks.  John