Druid Segment scaning take more than 5 minutes

[Prabakaran Hadoop]

Hi

Any one please tell me , In druid if segment scanning taken more than 5 minutes it will be time out. How solve this issue.

Segment having around 5million rows .

Any one help me to avoid it.

Thanks

Prabakaran Krishnan

[Sashidhar Thallam]

Hi Prabhakaran,

Are you talking about the query timeout ? The default value of druid.server.http.defaultQueryTimeout is 300000 ms which is 5 minutes beyond which unfinished queries will be cancelled. Look at http://druid.io/docs/latest/configuration/index.html#broker. You can change this property using query context or in the broker runtime.properties to a value (in milliseconds) which is reasonable for your use case. 

If 5 minutes is not reasonable you should look at tuning your cluster/cluster configuration. Moreover the segment file size should be in the recommended range 300mb-700mb for better performance.

Thanks,

Sashi

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/6c44b6dd-2dea-49a6-bc5b-a8323e88b3fa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Prabakaran Hadoop]

Hi Shashidhar

Thanks for reply. I got the error message like "gateway time out".

Thanks

Prabakaran krishnan

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/CAGytpa%3DQ4b2Z%3Dxtf2i1F0sDBqSb0u0j89uP%2BRA0ojCTiQwqPZA%40mail.gmail.com.

[Prabakaran Hadoop]

Thanks for your reply.

I have setup in broker runtime.properties below value:

druid.server.http.defaultQueryTimeout=900000

But this gateway time out issue continous.

Thanks

Prabakaran Krishnan

[Sashidhar Thallam]

Hi Prabakaran,

Looking at the broker logs might give some clues. 

Thanks,
Sashi

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/CAEoLoksy-ZduXTW5P1s_yVmOLJ%2B6SZ%2B7my1o1KSzPiWCQHo2Aw%40mail.gmail.com.

[Laxmikant Pandhare]

Did this issue got fixed? or timeout increased to more than 5 minutes by any chance.

[William Volkman]

Both the broker and historical may need to have that value set, I was getting it from historical after I made changes to the broker runtime.properties file.  I upped mine to 30 minutes because I have a larger dataset.

Hope this helps.

[Laxmikant Pandhare]

Oh okay! I updated only broker runtime properties for druidQueryTimeout and it didn’t work. I might need to update historical too. Let me give it a try. Thank you for your comment.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/d9f8c537-73d9-49ab-97e8-2f705f43db16n%40googlegroups.com.

[Laxmikant Pandhare]

It didn't work for me. I update runtime.properties files of broker and historicals.

[Alex Alex]

try to add "timeout" parameter with milliseconds in query context. It works for me. Message about timeout of 10 minutes:

2023-10-25T12:58:45.139Z        10.140.130.131          {"sqlQuery/time":600799,"sqlQuery/planningTimeMs":797,"sqlQuery/bytes":0,"success":false "context":{"timeout":600000,"sqlQueryId":"177da0bc-1f38-414f-b503-05c2dc89e90c","nativeQueryIds":"[622f3d9f-9ce2-4bed-a110-a38fb58b9b7c]"}