Prometheus Federation: cannot unmarshal number into Go struct field

[Edwin Vasquez]

The issue is happening on Prometheus version: 2.49.1

I'm getting the following error on Prometheus federation:

ts=2024-02-03T13:30:14.209Z caller=file.go:343 level=error component="discovery manager scrape" discovery=file config=nrp-federation msg="Error reading file" path=/etc/prometheus/federate_sd.json err="json: cannot unmarshal number into Go struct field .labels of type model.LabelValue"

It is not happening on my old instance (with version 2.29.2).  One thing I have noticed that if the generated federation JSON file has json objects that are not properly arrange, the error occurs.

Any thoughts?

[Brian Candler]

Can you show the content of your /etc/prometheus/federate_sd.json file?  The error suggests to me that you are putting a number where you need a string, for example

    {"labels": {"foo": 123}}

where it should be

   {"labels": {"foo": "123"}}

[Edwin Vasquez]

Hey Brian,

You're awesome!  You gave me a hint on what to check It's now working perfectly!  Now I need to figure out on how prometheus read the updates in federate_sd.json without restarting the prometheus service.  It seems like `curl -X POST http://localhost:9090/-/reload` is not working.  Note that "--web.enable-lifecycle" is included in the docker compose YAML file.  Any thoughts?

[Brian Candler]

If you are using file_sd_configs to read /etc/prometheus/federate_sd.json (which is what I guess you're doing), then changes will be automatically picked up. There's no need to hit the reload endpoint for this (*). 

Of course, the new file has to be valid JSON (or YAML); if it's not, then it will be ignored and prometheus will continue using the old contents. You can check validity using:

/path/to/promtool check config /path/to/prometheus.yml

/path/to/promtool check service-discovery /path/to/prometheus.yml job_name

(*) You only need to hit /-/reload if you've changed prometheus.yml or any rules files. Note that if you'd omitted --web.enable-lifecycle then you'd get a 403 Forbidden response with text "Lifecycle API is not enabled."