Druid tasks are in Running, waiting and pending stats not finishing successfully.

[Tariq Hasan]

Hi Druid team,

I am using Druid 0.22 version . Few days ago, Druid tasks were not finishing successfully, most of the tasks are in "pending" and in "waiting " states.  There are jobs also in "Running" status but not finishing. The setup is not huge means I have 1 service on each node. Druid has been running smoothly from last few years and few days ago we see the tasks are in running, pending and waiting states.  The  new thing we have introduced  few months ago is "lookup"  in our environment which is cachedNamespace lookup on s3. We have been loading data successfully but things are not working for last few days.

When I checked the middle manager tasks  I found there is no host and port were assigned to the jobs, I have pasted below the jobs status of all types of jobs. To assign the host info to the task, I have hard coded the druid_host ip address in middle manager configuration and I found new jobs have host and port information. I could finish new tasks but still some new tasks are not reflecting host and port name. Also there are 1500 plus old tasks  which are in pending and waiting stats and I think these jobs causing other jobs to fail. I want to know how to fix it fast. I have disabled all data source and tried to kill all old jobs using  curl /post command but it is taking very long time to kill one job. I want to finish all old tasks quickly and restore the normal operation. I will really appreciate any help. 

Following are the information what I am seeing in Ingestion tasks  and click on each job to see inside of running, waiting and Pending jobs

Running jobs  

Ststus Tab

  "statusCode": "RUNNING",

  "status": "RUNNING",

  "runnerStatusCode": "RUNNING",

  "duration": -1,

  "location": {

    "host": null,

    "port": -1,

    "tlsPort": -1

  },

Reports  Tab -Request failed with status code 404

Pending jobs 

Status tab:

  "statusCode": "RUNNING",

  "status": "RUNNING",

  "runnerStatusCode": "PENDING",

  "duration": -1,

  "location": {

    "host": null,

    "port": -1,

    "tlsPort": -1

  },

  "dataSource": "12345678,

  "errorMsg": null

}

Reports  Request failed with status code 404

Waiting jobs:

Status tab:

"statusCode": "RUNNING",

  "status": "RUNNING",

  "runnerStatusCode": "WAITING",

  "duration": -1,

  "location": {

    "host": null,

    "port": -1,

    "tlsPort": -1

  },

  "dataSource": "12345678",

  "errorMsg": null

}

Reports and Logs : Request failed with status code 404

[Tariq Hasan]

one update: I can see the new task are running with host information  but when I clicked on the report tab, it is showing this message --> Request failed with status code 500. It could be because I have disabled all datasource ?