Query log isn't very helpful in finding queries that crash server

10 views
Skip to first unread message

pin...@hioscar.com

unread,
Jan 18, 2022, 10:41:25 PMJan 18
to Prometheus Users
It seems to me that because query log records end time, it doesn't log queries until they are finished. So, if server runs into OOM and crashes due to some expensive queries, the offending queries will never be logged in query log.

{
  "params": {
    "end": "2022-01-19T03:37:34.316Z",
    "query": "... ...",
    "start": "2022-01-19T03:37:34.316Z",
    "step": 0
  },
  "ruleGroup": {
    "file": "... ...",
    "name": "... ..."
  },
  "stats": {
    "timings": {
      "evalTotalTime": 0.0027992,
      "execQueueTime": 6.581e-06,
      "execTotalTime": 0.002811287,
      "innerEvalTime": 0.001367707,
      "queryPreparationTime": 0.001420335,
      "resultSortTime": 0
    }
  },
  "ts": "2022-01-19T03:37:34.401Z"
}

pin...@hioscar.com

unread,
Jan 18, 2022, 10:49:49 PMJan 18
to Prometheus Users
To address this, logging query start and end events separately would help.

Julien Pivotto

unread,
Jan 19, 2022, 1:36:41 AMJan 19
to pin...@hioscar.com, Prometheus Users
Hello,

The queries that crash the server are logged in the normal Prometheus log during the next startup, even if the query log is disabled.


--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c10865ab-b339-4c29-98fb-e0c3aaecb94an%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages