Indexing tasks are in pending state even though worker's used capacity is below available capacity

[rohit kochar]

I have a druid setup in which indexing service is running in remote mode i.e 

'druid.indexer.runner.type=remote'

And also i have set the max capacity of worker=15 i.e

druid.worker.capacity=15 

still i observe that only 4 indexing task are in running state at any given time and overlord's web interface show that used capacity of worker is 4 and worker capacity is 15.

What could be the reason for all other task to be added to pending queue instead of running queue??

And also what happens to data that i push to indexing service while my task was in waiting state?Does that data get lost?

I am using tranquility for ingesting data.

Thanks

Rohit

[Gian Merlino]

Hi Rohit, the three usual reasons for pending tasks are (a) no capacity on workers; (b) not enough workers to satisfy redundancy requirements; (c) some other task has a lock that the pending task needs. You said that the overlord console shows that you have capacity, so it's probably not (a). Could it be (b)- do you have redundancy in tranquility set higher than the number of available workers?

Data pushed while tasks are pending will be buffered on the client side for the windowPeriod, then dropped. If you have a copy of the data in batch storage like S3 or HDFS, you could re-ingest it using batch indexing.

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/c0e18302-47de-4c8c-8d4f-7ad3f63bebc2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rohit Kochar]

Gian,

Redundancy has been set at 1 and i have exactly 1 worker running.

And had been case (c) those pending task should have been shown under the heading “Waiting Tasks-Tasks waiting on locks” on web interface instead of under “Pending Tasks-Tasks waiting to be assigned to a worker”.

Is there any other reason for pending tasks??

Thanks

Rohit

You received this message because you are subscribed to a topic in the Google Groups "Druid User" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-user/b15kUO-obpg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-user+...@googlegroups.com.

To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/CAPP9sfa18bZDMG_G%3DZVawYsq2jW7DDWZHwmC_cXZbr1PYZmuZw%40mail.gmail.com.

[Gian Merlino]

Not really that I can think of. Can you attach a screenshot of your overlord console, and your overlord logs? There might be something interesting in one of those.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/0CEF61FE-2AC6-4B77-BEBE-AF41FD3DC4CE%40gmail.com.

[Sumit Khanna]

Any updates on this one ? well, my taska arent in pending, but running mode, and also ... they are outputing logs regularly real time, just that the task has been running for 2 hours now, for ingesting hardly any 1000 records ??

[David Lim]

Hey Sumit,

Can you post your task spec and task logs?

[Sonny Manurung]

Hi Gian, I'm facing similar issue about worker utilisation. I set druid.worker.capacity=10, but I only see 2 running workers. Previously, I had 1 task in pending state, though it's not there in the screen shot. Why is task got into pending while it's not even reaching the capacity?

Overlord log: https://pastebin.com/Xnq9PH9a.