Prometheus and Grafana not receiving metrics

[Vincent Brown]

Hello. I'm new to PGO and working through the tutorials in the documentation. I have a K3S cluster managed by Rancher in my homelab.

Following the Getting Started documentation, I forked the Postgres Operator examples repository and cloned it to my host machine.

You can see my code at vincentertainment / postgres-operator-examples

See the kustomize/hq-db-pg01 folder.

I've managed to deploy an HA cluster with scheduled backups, patroni, as well as the monitoring stack.I added the spec.monitoring.pgmonitor.exporter.image entry per the 'Adding the Exporter Sidecar' documentation. 

I deployed the monitoring stack from the kustomize/monitoring folder.

The Monitoring documentation says "PGO will detect the change and add the Exporter sidecar to all Postgres Pods that exist in your cluster. PGO will also do the work to allow the Exporter to connect to the database and gather metrics that can be accessed using the PGO Monitoring stack."

I can successfully log into the Prometheus and Grafana dashboards but I'm not seeing any metrics populated. Everything is the same postgres-operator namespace.

Should I be seeing a deployment with pgmonitor in the name? I'm not seeing any deployments or stateful set with pgmonitor, nor am I seeing any mention of pgmonitor in my cluster events.

Any suggestions on how to troubleshoot further?

Thank you,

Vince

[Tony Landreth]

Hi Vince!

If you've applied the exporter after you've applied the monitoring stack, you may need to delete and reapply the monitoring stack.

You won't see a pgmonitor deployment -- just deployments for monitoring components (grafana, prometheus, alertmanager).

I hope this helps,

Tony

[Vincent Brown]

Thanks Tony.

I'm getting at least some data now. Does PGExporterScrapeError indicate a problem sending data? Poking around, I see some dashboards that don't load graphs. 

Also, while the exporter doesn't have its own deployment, I did find exporter containers in the pods of the cluster's stateful set.

[Tony Landreth]

Hi Vince,

The PGExporterScrapeError indicates a problem retrieving data. Our monitoring architecture doc provides an overview of the alert system.

The exporter is installed as a sidecar.

We need to update the PostgreSQL Overview dashboard -- fortunately, that isn't one of the more informationally rich displays in our Grafana deployment. The other dashboards should show activity. Note that you can toggle views through the dropdown menus in the upper left part of a dashboard. If things still don't look right, please let us know which dashboards are giving you trouble.

Thanks,

Tony