HTTP status 503 service unavailable: json_exporter with basic_auth

[Siva Prakash Karna]

Hi,

I'm getting below error on prometheus console log while scraping metrics from json_exporter. Target endpoint is available and able to get response with curl using basic authentication.

ts=2022-04-18T20:58:07.156Z caller=scrape.go:1292 level=debug component="scrape manager" scrape_pool=json target="http://localhost:7979/probe?target=http%3A%2F%2Flocalhost%3A9015%2Fservices%2Fv2%2Fmpoints%2FE_TIP5%2Fstatistics" msg="Scrape failed" err="server returned HTTP status 503 Service Unavailable"

Here is the prometheus.yaml

scrape_configs:

## gather metrics of prometheus itself

- job_name: prometheus

static_configs:

- targets:

- localhost:9090

## gather the metrics of json_exporter application itself

- job_name: json_exporter

static_configs:

- targets:

- localhost:7979 ## Location of the json exporter's real <hostname>:<port>

## gather the metrics from third party json sources, via the json exporter

- job_name: json

scrape_interval: 15s

scrape_timeout: 10s

metrics_path: /probe

basic_auth:

username: XXXXXX

password: XXXXX

static_configs:

- targets: ['http://localhost:9015/services/v2/mpoints/E_TIP5/statistics']

relabel_configs:

- source_labels: [__address__]

target_label: __param_target

- source_labels: [__param_target]

target_label: instance

- target_label: __address__

replacement: localhost:7979 ## Location of the json exporter's real <hostname>:<port>    

[Brian Candler]

The configuration of prometheus isn't really of interest, because it's json_exporter that's returning the error.

Scrape the exporter by hand:

curl -vg 'http://localhost:7979/probe?target=http:%2f%2flocalhost%3a9015%2fservices%2fv2%2fmpoints%2fE_TIP5%2fstatistics'

I suspect you'll see the 503 error there too, but you may get a more detailed error message that may help understand what's going on.  Also try scraping the JSON target directly:

curl -vh 'http://localhost:9015/services/v2/mpoints/E_TIP5/statistics'

If the latter doesn't work, or doesn't return JSON, then obviously the former won't work either.

Likely problems are:

- the .../statistics endpoint isn't working or isn't returning JSON

- the configuration of json_exporter is bad

[Brian Candler]

Also you should realise that if you set "basic_auth" in prometheus.yml, this only sets basic auth for the HTTP request from prometheus to json_exporter, not from json_exporter to target.

Does the target endpoint http://localhost:9015/services/v2/mpoints/E_TIP5/statistics require authentication? If so, you'll need a HTTPClientConfig section in your json_exporter configuration.

Admittedly I couldn't find this documented in the json_exporter config examples, but you can find what you need here:

https://github.com/prometheus-community/json_exporter/blob/v0.4.0/config/config.go#L40-L46

https://github.com/prometheus/common/blob/main/config/http_config.go