unexpected end of json input

[akshay sharma]

Hi, 

I have an application running, which is accepting the post request to add targets in the path /etc/prometheus/config/target.json. 

Currently, I'm posting 2-3 targets using curl to the application which in turn adds to the Prometheus path, post is happening in a time difference of 2-3 minutes. 

In Prometheus UI, I could see it is adding only one target or can see only one target in service discovery, but I have added 3 targets. 

could see some errors in Prometheus:

ts=2023-12-12T10:27:15.936Z caller=file.go:343 level=error component="discovery manager scrape" discovery=file config=ne-federation msg="Error reading file" path=/etc/prometheus/config/targets.json err="unexpected end of JSON input"
ts=2023-12-12T10:35:50.707Z caller=file.go:343 level=error component="discovery manager scrape" discovery=file config=ne-federation msg="Error reading file" path=/etc/prometheus/config/argets.json err="unexpected end of JSON input"

I checked the JSON file using JSON validator, it's perfectly valid. 

can you please help with what could be the issue?

[Brian Candler]

Perhaps you didn't write the file atomically. If you start appending to the file, and Prometheus notices you've changed it, it may read it before you've finished writing.

You should:

1. Write out a new file, e.g. target.json.new
2. fsync it (os.File.Sync)

3. rename target.json.new to target.json

[akshay sharma]

If I rename the new file to the existing file, then it would lose the old data in the existing file right?

so what we can do! 

[Brian Candler]

You misunderstand me. Create the new file "target.json.new" from scratch. If you want to merge data from the old target.json then read the old data in, modify it, and write out to the new file.

You cannot simply append to the end of a JSON file. What you will have is multiple JSON objects in one file, rather than one JSON object. AFAIK, Prometheus will not let you do that. However, whatever JSON validator you're using *might* allow multiple JSON objects.

If you show the contents of the file you've created, then it may become clear what the problem is.

For example, the following is not a valid JSON file:

{"hello":"world"}

{"foo":"bar"}

You would have to merge the objects, like this:

{"hello":"world", "foo":"bar"}