Help needed regarding the druid exception

[Roo Riop]

Hi All,

We are trying to cache around 80 millions rows of data in druid with the below hardware:

No. of nodes - 4

CPU : 46

RAM : 96 GB

OS Disk : 100 Gb

Data Disk : 1 * 500 GB

Storage : 2 *Data Disk 1 TB

- 20 parallee queries - few are manually executed and few are scheduled

- Dashboards with 6 charts.

Now, the application is failing to load the dashboards from druid with the below exception:

Historicals:

2024-06-08T10:17:38,141 WARN [qtp1990720701-72[groupBy_[ds_ws_mvno_d_trm_rat_miss_t_ra]_f1c07071-9885-4b96-b9c3-0f3d1c896ce2]] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [f1c07071-9885-4b96-b9c3-fdgfdgfdgfdg] (org.apache.druid.query.QueryTimeoutException: Query Timed Out!)

Router:

TRACE StatusLogger Call to LogManager.getLogger(com.sun.jersey.server.impl.application.CloseableServiceFactory)

2024-06-07T17:47:39,388 ERROR [CoordinatorRuleManager-Exec--0] org.apache.druid.server.router.CoordinatorRuleManager - Exception while polling for rules

org.apache.druid.java.util.common.IOE: No known server

 at org.apache.druid.discovery.DruidLeaderClient.getCurrentKnownLeader(DruidLeaderClient.java:267) ~[druid-server-24.0.1.jar:24.0.1]

 at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:122) ~[druid-server-24.0.1.jar:24.0.1]

 at org.apache.druid.server.router.CoordinatorRuleManager.poll(CoordinatorRuleManager.java:138) ~[druid-services-24.0.1.jar:24.0.1]

 at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) ~[druid-core-24.0.1.jar:24.0.1]

Broker:

2024-06-08T10:17:38,132 WARN [sql[b563386c-0247-4a90-80f8-95dff3577d58]] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [a4660137-48bf-4dfc-8374-05425aa4a0a0] (org.apache.druid.query.QueryTimeoutException: Query [a4660137-48bf-4dfc-8374-05425aa4a0a0] timed out!)

2024-06-08T10:17:38,133 WARN [sql[64e55900-20d0-43f5-815c-6b94b019a3ac]] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [1e035b2c-d951-4802-a618-320a03289976] (org.apache.druid.query.QueryTimeoutException: Query [1e035b2c-d951-4802-a618-320a03289976] timed out!)

2024-06-08T10:18:58,884 ERROR [qtp1389978471-113] org.apache.druid.sql.http.SqlResource - Unable to send SQL response [ee876e5a-b24b-440a-9744-b0c391aa941e]

org.eclipse.jetty.io.EofException: null

 at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:280) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]

 at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]

 at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]

 at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]

 at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:831) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]

Caused by: java.io.IOException: Broken pipe

 at sun.nio.ch.FileDispatcherImpl.writev0(Native Method) ~[?:1.8.0_275]

 at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) ~[?:1.8.0_275]

 at sun.nio.ch.IOUtil.write(IOUtil.java:148) ~[?:1.8.0_275]

 at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:503) ~[?:1.8.0_275]

 at java.nio.channels.SocketChannel.write(SocketChannel.java:502) ~[?:1.8.0_275]

 at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:274) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]

 ... 84 more

2024-06-08T10:18:59,404 ERROR [qtp1389978471-122] org.apache.druid.sql.http.SqlResource - Unable to send SQL response [7175fc93-f81e-4137-b103-419b3986d9a7]

org.eclipse.jetty.io.EofException: null

 at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:280) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]

 at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]

 at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]

 at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]

 at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:831) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]

can anyone pls help me with this issue?? Does the above said hardware suffice the requirement? and also how to fix this issue

[DMT]

Ok