failed to start prometheus on centos9

[Gharbi abdelaziz]

× prometheus.service - Prometheus
     Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Fri 2023-05-26 07:09:35 UTC; 21min ago
   Duration: 45ms
    Process: 43908 ExecStart=/usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries (code=exited, status=2)
   Main PID: 43908 (code=exited, status=2)
        CPU: 32ms

May 26 07:09:35 grafana prometheus[43908]: ts=2023-05-26T07:09:35.475Z caller=query_logger.go:113 level=error component=activeQueryTracker msg="Failed to create directory for logging active queries"
May 26 07:09:35 grafana prometheus[43908]: ts=2023-05-26T07:09:35.475Z caller=query_logger.go:91 level=error component=activeQueryTracker msg="Error opening query log file" file=/var/lib/prometheus/queries.active err="open /var/lib/prometheus/queries.active: no such file or directory"
May 26 07:09:35 grafana prometheus[43908]: panic: Unable to create mmap-ed active query log
May 26 07:09:35 grafana prometheus[43908]: goroutine 1 [running]:
May 26 07:09:35 grafana prometheus[43908]: github.com/prometheus/prometheus/promql.NewActiveQueryTracker({0x7ffe15a37e80, 0x14}, 0x14, {0x38169e0, 0xc000805a90})
May 26 07:09:35 grafana prometheus[43908]:         /app/promql/query_logger.go:121 +0x3cd
May 26 07:09:35 grafana prometheus[43908]: main.main()
May 26 07:09:35 grafana prometheus[43908]:         /app/cmd/prometheus/main.go:597 +0x6713
May 26 07:09:35 grafana systemd[1]: prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 26 07:09:35 grafana systemd[1]: prometheus.service: Failed with result 'exit-code'.

[Daniel Swarbrick]

This is a simple filesystem permissions issue, as you can see by these two log messages:

May 26 07:09:35 grafana prometheus[43908]: ts=2023-05-26T07:09:35.475Z caller=query_logger.go:113 level=error component=activeQueryTracker msg="Failed to create directory for logging active queries"
May 26 07:09:35 grafana prometheus[43908]: ts=2023-05-26T07:09:35.475Z caller=query_logger.go:91 level=error component=activeQueryTracker msg="Error opening query log file" file=/var/lib/prometheus/queries.active err="open /var/lib/prometheus/queries.active: no such file or directory"

Make sure that either /var/lib/prometheus already exists and is writable by the process, or that /var/lib is writable by the process (so that it can create the prometheus directory itself).

[Brian Candler]

And bear in mind that on RHEL/CentOS systems, SELinux may also be a factor (especially if permissions otherwise all appear to be OK)