Prometheus to Prometheus Remote Write Without Persistent TCP Connection and Time Interval

[Umut Gerçek]

I'm trying to a implement remote write setup via only pushing at strict 10 minute interval requirement. Both push based and time intervals are must for me :(

Is remote write is possible without streaming? In Prometheus remote write spec, streaming is mentioned but I'm ok with losing data and scalability problems due to high humber of new tcp connection creation.

I increased `batch_send_deadlines` but still get requests in every 15 second, there's an http config called `keepAlivesEnabled`  in [prometheus/common](https://github.com/prometheus/common/blob/main/config/http_config.go#L50) . AFAIK from code and documentation there's no way to disable this option.

I tested via `tcpdump`:
- `tcpdump -i eth0 'port 9090'`: Main server (receives data) gets request in every 15 second
- `tcpdump -i eth0 'port 9090 and tcp[tcpflags] & (tcp-syn) != 0'`: Only one tcp connection is created and persisted

[Bryan Boreham]

I believe remote-write in Prometheus will send samples if the outgoing buffer is full or the deadline is reached.

So increasing both max_samples_per_send and batch_send_deadline in queue_config should work.

You should be able to increase the number of parallel sends via min_shards , so it doesn't all go in one buffer.

Then there's metadata -  by default it will send once per minute in metadata_config.send_interval 

I don't see anything defaulting to 15 seconds, so check your config to see if that is set anywhere.

Bryan