Query log isn't very helpful in finding queries that crash server

[pin...@hioscar.com]

It seems to me that because query log records end time, it doesn't log queries until they are finished. So, if server runs into OOM and crashes due to some expensive queries, the offending queries will never be logged in query log.

{
  "params": {
    "end": "2022-01-19T03:37:34.316Z",
    "query": "... ...",
    "start": "2022-01-19T03:37:34.316Z",
    "step": 0
  },
  "ruleGroup": {
    "file": "... ...",
    "name": "... ..."
  },
  "stats": {
    "timings": {
      "evalTotalTime": 0.0027992,
      "execQueueTime": 6.581e-06,
      "execTotalTime": 0.002811287,
      "innerEvalTime": 0.001367707,
      "queryPreparationTime": 0.001420335,
      "resultSortTime": 0
    }
  },
  "ts": "2022-01-19T03:37:34.401Z"
}

[pin...@hioscar.com]

To address this, logging query start and end events separately would help.

[Julien Pivotto]

Hello,

The queries that crash the server are logged in the normal Prometheus log during the next startup, even if the query log is disabled.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c10865ab-b339-4c29-98fb-e0c3aaecb94an%40googlegroups.com.