" request split points lookup for chunk "

[tetlika]

hi

I've noticed that after appearing such messages in logs like:

Mon Nov 26 18:09:20 [conn60091] request split points lookup for chunk
booble.field_values { : ObjectId('4ff0fbed1e14f63169407fbc'), :
ObjectId('50639426cebb3e06630003ef'), :
ObjectId('50639b1ca2b58461cf008ac1') } -->> { :
ObjectId('4ffb8fca8843541ffda12945'), :
ObjectId('4ffd65445eee181fe4000120'), :
ObjectId('4ffd63c206fa1c55d1000131') }

the load on corresponding shard flies up to the sky :) the queue
becomes around 100-200, the load average on the node is around 100

is that expected behavior ?

[tetlika]

anyone? :)

[Andre de Frere]

Hi,

The "request split points for chunk" messages come about from a splitVector operation taking place.  The splitVector command is issued relatively early in the maximum size of a chunk, at a somewhat random point around 20%, and then frequently afterward.  It is somewhat random because this prevents all of the threads and processes testing splitting chunks at exactly the same time.  More often than not, these tests return nothing, meaning that the chunk does not need to be split.  If the splitVector command does return some values, then it means the chunk can be split and we can then start a split process.

SplitVector commands are designed to yield often, so that they do not compete on I/O.  This also means long running splitVector commands do not necessarily indicate slow performance.

Are you getting high I/O on specific shards, or is more general across all shards?

What could also have happened after these log messages appear is that a split took place and then a subsequent moveChunk.  As moveChunks actually migrate the data between shards you would see higher load on the shards that are involved in the move.

Regards,

André

[tetlika]

Thanks much for your reply Andre!

Looks like this is not root cause . I'll be digging deeper to find out
what is the root cause of load spikes.