Simple Prometheus + Grafana setup not showing metrics

[membrano phone]

I have grafana and prometheus running on localhost and trying to run a sample app to get some graphs within grafana. This is my first dip into prometheus.

I am running a simple fastapi server

from fastapi import FastAPI
from prometheus_client import make_asgi_app, Gauge

# Create app
app = FastAPI(debug=False)

metrics_app = make_asgi_app()
app.mount("/metrics", metrics_app)

g = Gauge('my_counter', 'Description of gauge')


@app.get('/inc')
def process_request():
   g.inc()
   return {}

I am running this with

pipenv run uvicorn fastapi_example:app --port 9090

Navigating to http://localhost:9090/metrics/ shows my_counter 0.0 if I navigate to the /inc I can observe the number increasing.

For the setup within Grafana I only add a Prometheus data source and set the url to http://127.0.0.1:9090/metrics

However when I navigate to add a panel that exposes the counter my_counter it just says 'No metrics found' I can however see that the fastapi server is seeing requests and responding with 200 OK.

I also have the first example for start_http_server running but that is also not giving me any metrics from within Grafana.

It feels like I am missing something glaringly obvious. Can anyone see anything wrong with my test setup?

[Brian Candler]

http://localhost:9090/metrics is the wrong place to look. This is the endpoint for metrics generated by prometheus itself (e.g. stats about the operation of its timeseries database).

To access the data stored within prometheus, you can use its HTTP API as documented here: https://prometheus.io/docs/prometheus/latest/querying/api/ - or you can go to its web interface at http://localhost:9090/ (which in turn talks to the HTTP API).

This API is what Grafana also talks to. You'd need to point Grafana at http://127.0.0.1:9090/ only, and it will add the api path automatically. Do not append "/metrics".

Apart from this: if you are able to see the metrics of interest in the Prometheus web interface, but have a problem with Grafana, the best place to ask for help is the Grafana community: https://community.grafana.com/.  This is because Grafana is a completely separate project to Prometheus.