Metrics shown are not reflected correctly

[Honsen Clock]

Issue Description:

I have a Python Flask app and I wanna add a new metric "by_path_counter_doc_quality" for an API, which I added by:

```Python

# app.py

by_path_counter = metrics.counter(
    'by_path_counter_doc_quality', 'Request count by request paths',
    labels={'path': lambda: request.path, 'method': lambda: request.method}
)

...

@app.route("/document_quality/transformation", methods=["POST"])
@by_path_counter
def run_comp_metrics():
    # code

```

But when I start prometheus server and monitor, I cannot find this metrics on the metrics list. The flask app itself is working and using metric "flask_http_request_total" can find the requests.

- Moreover, I notice that the metrics list on the "metrics explorer" tab shows out-of-dated metrics. See the attached image, "by_path_counter" and "in_progress" were the metrics I added in previous scrape_config job.

My prometheus.yml:

```yaml

# my global config
global:
  scrape_interval: 15s 
  evaluation_interval: 15s 

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

scrape_configs:
  - job_name: "doc-quality"
    static_configs:
      - targets: ["localhost:8881"]

```

Some context about the issue:

In my prometheus.yaml, I created a job "flask_app", with metrics called "by_path_counter" and "in_progress". The metrics works well. So I wanna applied the same way to the current new job "doc-quality" and new metrics "by_path_counter_doc_quality".

My questions:

- What's the root cause for above issue? Why my new metric is now shown?

- Are there any ways that I can reset/refresh the metrics? Maybe clean the time series DB? If so, how to to that.

[Brian Candler]

I suggest you take Prometheus entirely out of the picture, and scrape your metrics endpoint directly using curl:

curl localhost:8881/metrics

See if your expected custom metric is there. If it's not, then that's why Prometheus isn't showing it, and you need to debug your Python code.

[Novigrad Clock]

I found out the root cause after that. We have duplicated API with sama name so the one I added metrics is not seen. Legacy code drive me crazy. About those out-of-date metrics, I delete the db through the admin-api and those metrics are cleared. Thanks for looking into that.