Kafka Indexing Service - Ingestion was throttled

[Ankush Kapur]

Hi,

I am seeing the following error once in a while in the middleManager logs,

2017-01-26T00:57:56,336 WARN [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Ingestion was throttled for [14,948] millis because persists were pending.

Usually it appears in bursts.

Any ideas on why and when does this happen, and there are config changes that need to be looked at.

Running Druid 0.9.2

Let me know what config settings are relevant so I can post them to get a better idea of the setup.

~ Ankush

[Gian Merlino]

Hey Ankush,

This means that because it took more time to write to disk than expected, ingestion was paused and wasn't reading the stream for 15 seconds. If this is only intermittent, it probably means you had a load spike that the system took 15 seconds to recover from. If you want to avoid this then you could shard out ingestion more, or use faster disks.

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/7073110a-3777-4e2a-a40d-8dbcc27ddde7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ankush Kapur]

Gian,

Thanks for the quick reply.

If I have to go the sharding route, would it mean lowering the segmentGranularity? Currently, I have it set to FIFTEEN_MINUTE.

I also lowered maxRowsInMemory in the meanwhile. Does maxRowsPerSegment also affect the ingestion/writing to disk throughput?

~ Ankush

On Wednesday, January 25, 2017 at 8:00:32 PM UTC-7, Gian Merlino wrote:

Hey Ankush,

This means that because it took more time to write to disk than expected, ingestion was paused and wasn't reading the stream for 15 seconds. If this is only intermittent, it probably means you had a load spike that the system took 15 seconds to recover from. If you want to avoid this then you could shard out ingestion more, or use faster disks.

Gian

On Wed, Jan 25, 2017 at 5:02 PM, Ankush Kapur <ankush...@fidelissecurity.com> wrote:

Hi,

I am seeing the following error once in a while in the middleManager logs,

2017-01-26T00:57:56,336 WARN [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Ingestion was throttled for [14,948] millis because persists were pending.

Usually it appears in bursts.

Any ideas on why and when does this happen, and there are config changes that need to be looked at.

Running Druid 0.9.2

Let me know what config settings are relevant so I can post them to get a better idea of the setup.

~ Ankush

--
You received this message because you are subscribed to the Google Groups "Druid User" group.

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.

[Gian Merlino]

maxRowsInMemory and segmentGranularity affect how often data is written to disk but don't have a big effect on how much is written overall. One other thing you can do is set "buildV9Directly" : true in your tuningConfig if you haven't already. That usually helps indexing performance and can reduce this persist throttling. But if you do that, and still keep seeing this throttling, then more tasks or better disks should help.

Gian

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.

To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/94a717f4-41a2-4217-82bb-c0b4601569c2%40googlegroups.com.

[Ankush Kapur]

Hi Gian,

This is late, but wanted to let you know that things did improve making changes you recommended and also other tweaks like taskDuration, number of workers and task counts. A combination of all these made good improvements.

I have a few more questions, but I will post another thread for them.

~ Ankush

Gian