Query log isn't very helpful in finding queries that crash server

Skip to first unread message


Jan 18, 2022, 10:41:25 PMJan 18
to Prometheus Users
It seems to me that because query log records end time, it doesn't log queries until they are finished. So, if server runs into OOM and crashes due to some expensive queries, the offending queries will never be logged in query log.

  "params": {
    "end": "2022-01-19T03:37:34.316Z",
    "query": "... ...",
    "start": "2022-01-19T03:37:34.316Z",
    "step": 0
  "ruleGroup": {
    "file": "... ...",
    "name": "... ..."
  "stats": {
    "timings": {
      "evalTotalTime": 0.0027992,
      "execQueueTime": 6.581e-06,
      "execTotalTime": 0.002811287,
      "innerEvalTime": 0.001367707,
      "queryPreparationTime": 0.001420335,
      "resultSortTime": 0
  "ts": "2022-01-19T03:37:34.401Z"


Jan 18, 2022, 10:49:49 PMJan 18
to Prometheus Users
To address this, logging query start and end events separately would help.

Julien Pivotto

Jan 19, 2022, 1:36:41 AMJan 19
to pin...@hioscar.com, Prometheus Users

The queries that crash the server are logged in the normal Prometheus log during the next startup, even if the query log is disabled.

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c10865ab-b339-4c29-98fb-e0c3aaecb94an%40googlegroups.com.
Reply all
Reply to author
0 new messages