Questions on Prometheus Agent Mode

[tejaswini vadlamudi]

Hi,

Can we configure multiple remote-write receivers in agent mode?  

• If the answer is "no", could you explain the background reason, please?

Is federation supported in agent mode?

Thanks,

Teja 

[Brian Candler]

On Wednesday, 7 June 2023 at 23:41:31 UTC+1 tejaswini vadlamudi wrote:

Can we configure multiple remote-write receivers in agent mode?  

Good question. The configuration file allows a list of remote_write receivers:

# Settings related to the remote write feature. remote_write: [ - <remote_write> ... ]

... but since agent mode uses a "customized TSDB WAL" that deletes data as soon as it has been successfully delivered, I don't know if it can handle the case where one receiver is backlogging but another is not.

Is federation supported in agent mode?

 

I would expect not, since federation involves making queries to the prometheus database - and agent mode "disables querying, alerting, and local storage" completely.

[tejaswini vadlamudi]

Thanks, Brain for the answer!

I think Prometheus Agent supports only one remote-write end-point by implementation (not due to config), please correct me if I'm wrong.

+1 to your answer on federation.

[Brian Candler]

Perhaps you could test it? It certainly *accepts* multiple URLs in agent mode:

$ cat test-agent.yml
remote_write:
  - url: http://127.0.0.1:1234/
  - url: http://127.0.0.1:1235/
$ /opt/prometheus/prometheus --config.file=test-agent.yml --storage.agent.path=/var/tmp/agent --enable-feature=agent --web.listen-address=127.0.0.1:9091

...

ts=2023-06-09T15:22:27.935Z caller=main.go:957 level=info msg="Server is ready to receive web requests."

$ curl -fsS localhost:9091/metrics | grep remote_storage_samples_pending
# HELP prometheus_remote_storage_samples_pending The number of samples pending in the queues shards to be sent to the remote storage.
# TYPE prometheus_remote_storage_samples_pending gauge
prometheus_remote_storage_samples_pending{remote_name="5e94e3",url="http://127.0.0.1:1234/"} 0
prometheus_remote_storage_samples_pending{remote_name="d296a2",url="http://127.0.0.1:1235/"} 0

Whether it actually delivers to both endpoint or not, I have not tested.  However, I looked in the source code under tsdb/agent/db.go (the WAL-only database used in agent mode) and I see nothing special that suggests it wouldn't work.

If you find that it doesn't work with multiple remote_write endpoints, then it's a bug one way or the other. At very least, it ought to be changed to reject a config like the one above.