MongoDB scrape target fails with 401 error on Prometheus web ui

[Arun Lakshmanan]

Hello Team,

We are trying to integrate Azure-managed Prometheus to our MongoDB cluster for monitoring but we are running into some issues.

We got the below scrape config from the  prometheus integration page in Atlas MongoDB

 - job_name: "sandbox-mongo-metrics"

        scrape_interval: 10s
        metrics_path: /metrics
        scheme: https
        basic_auth:
          username: "prometheus"
          password: "xxx"
        http_sd_configs:
          - url: https://cloud.mongodb.com/prometheus/v1.0/groups/xxx/discovery?targetScheme=PRIVATE
            refresh_interval: 60s
            basic_auth:
              username: "prometheus"
              password: "xxx"

And we use the below ConfigMap to add the scrape config to prometheus.

kind: ConfigMap
apiVersion: v1
data:
  prometheus-config: |-
    scrape_configs:
      - job_name: "sandbox-mongo-metrics"
        scrape_interval: 10s
        metrics_path: /metrics
        scheme: https

        basic_auth:
          username: "prometheus"
          password: "xxx"

        http_sd_configs:
          - url: https://cloud.mongodb.com/prometheus/v1.0/groups/xxx/discovery?targetScheme=PRIVATE
            refresh_interval: 60s
            basic_auth:
              username: "prometheus"
              password: "xxx"
metadata:
  name: ama-metrics-prometheus-config
  namespace: kube-system

The target discovery works without issues when we check with a curl command locally and in the kubernetes pod running prometheus.

But the target on the prometheus web ui fails with the below error

server returned HTTP status 401 Unauthorized

Any assistance is appreciated!

Thanks,
Arun

[Brian Candler]

My detailed reply was deleted by Google Groups for some unknown reason, and I don't have a copy because I made it in the web interface, so I will make a briefer one now.

Target discovery and target scraping are two different things. Target discovery is making a single call to an API endpoint to get a list of targets; scraping is making an HTTP request to each of the targets in that list, to collect metrics.

If you post a screenshot of what you're seeing, it would make things clearer, but it sounds to me like target *discovery* is successful (otherwise there would be no targets shown), but target *scraping* is failing.

In that case, you need to replicate the target *scraping* with curl and find out why it's failing.