When a task completes and we view the execution details via the query panel in the Druid Console, we see a blank box that says: “Request failed with status code 404" instead of the breakdown we're used to seeing.
We observe this error with:
- druid.indexer.runner.type: k8s
- druid.indexer.runner.type: k8sAndWorker
druid.indexer.runner.k8sAndWorker.runnerStrategy.type=k8s
but not for:
- druid.indexer.runner.type: k8sAndWorker
druid.indexer.runner.k8sAndWorker.runnerStrategy.type=worker
The theory about a missing task report.json file not getting created stems from two observations:
- We observe the error No task reports file exists to push from AbstractTask.cleanUp. We do however, observe the status file in GCS once the task completes.
- The network tab shows that the response from /druid/v2/sql/statements/{queryId} is used to power that view. After tracing through the code, and adding a log statement to confirm, we saw GoogleTaskLogs.streamTaskReports is downstream of it.
Have others run into the same issue?