Fundamental Question about Remote Read

1,208 views
Skip to first unread message

Dave Cadwallader

unread,
Mar 13, 2018, 1:00:12 AM3/13/18
to Prometheus Users
Hi,

I think I'm missing something basic about how remote read works.  I set up remote read and remote write to InfluxDB.  I successfully verified that remote write is working by setting up Grafana to query InfluxDB, and I'm seeing Prometheus metrics showing up there. 

So far so good.

Next, I wanted to test out remote read.  My understanding is that when this is set up, Prometheus no longer uses a local TSDB, nor a local cache, and every single query gets fulfilled by the remote read endpoint.  To test this out, I stopped the InfluxDB server.  To my surprise, the PromQL queries in Prometheus continued to work just fine.  I expected the queries to fail since the remote InfluxDB server is unavailable.

Am I misunderstanding how this works?

Config:

remote_write:

remote_read:

Thanks!
Dave

Brian Brazil

unread,
Mar 13, 2018, 3:51:25 AM3/13/18
to Dave Cadwallader, Prometheus Users
On 13 March 2018 at 05:00, Dave Cadwallader <dcadwa...@gmail.com> wrote:
Hi,

I think I'm missing something basic about how remote read works.  I set up remote read and remote write to InfluxDB.  I successfully verified that remote write is working by setting up Grafana to query InfluxDB, and I'm seeing Prometheus metrics showing up there. 

So far so good.

Next, I wanted to test out remote read.  My understanding is that when this is set up, Prometheus no longer uses a local TSDB, nor a local cache, and every single query gets fulfilled by the remote read endpoint.  To test this out, I stopped the InfluxDB server.  To my surprise, the PromQL queries in Prometheus continued to work just fine.  I expected the queries to fail since the remote InfluxDB server is unavailable.

Am I misunderstanding how this works?

Remote read is an additional data source, local storage is still the primary storage. For queries over recent data which will all be local, remote read may not even be used.

Brian
 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3286fa97-c40a-49c3-9c8a-4b4116568922%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

mro...@platformscience.com

unread,
Apr 20, 2018, 3:54:49 PM4/20/18
to Prometheus Users
Hello,

I am also having a similar issue with timescaledb as the remote storage.  Writing is working (I can see all the data in postgresql) and I can see the read queries in both the adapter and postgresql but prometheus doesn't seem to be using them.  If I execute those queries directly in postgresql I get the data back.

I first noticed this because I'm running prometheus in docker/rancher and I deployed a change that destroyed the prometheus container (and data directory). When it came back online all historical data was gone.

I also tested removing a single 2-hr block from local storage and, after restarting prometheus, that data was missing even though it exists in postgresql.

I'm using Prometheus 2.2.1 with remote read/write enabled.

Thanks,
Michael


On Tuesday, March 13, 2018 at 12:51:25 AM UTC-7, Brian Brazil wrote:
On 13 March 2018 at 05:00, Dave Cadwallader <dcadwa...@gmail.com> wrote:
Hi,

I think I'm missing something basic about how remote read works.  I set up remote read and remote write to InfluxDB.  I successfully verified that remote write is working by setting up Grafana to query InfluxDB, and I'm seeing Prometheus metrics showing up there. 

So far so good.

Next, I wanted to test out remote read.  My understanding is that when this is set up, Prometheus no longer uses a local TSDB, nor a local cache, and every single query gets fulfilled by the remote read endpoint.  To test this out, I stopped the InfluxDB server.  To my surprise, the PromQL queries in Prometheus continued to work just fine.  I expected the queries to fail since the remote InfluxDB server is unavailable.

Am I misunderstanding how this works?

Remote read is an additional data source, local storage is still the primary storage. For queries over recent data which will all be local, remote read may not even be used.

Brian
 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.



--

coolfi...@gmail.com

unread,
May 30, 2019, 12:13:11 AM5/30/19
to Prometheus Users
I have met the same issue, prometheus not read from InfluxDB. Is there anyway to make prometheus read from InfluxDB instead of local storage?

在 2018年3月13日星期二 UTC+8下午1:00:12，Dave Cadwallader写道：

Stuart Clark

unread,
May 30, 2019, 5:04:38 AM5/30/19
to coolfi...@gmail.com, Prometheus Users
On 30/05/2019 05:13, coolfi...@gmail.com wrote:
I have met the same issue, prometheus not read from InfluxDB. Is there anyway to make prometheus read from InfluxDB instead of local storage?


If you have remote read configured Prometheus will use that as well as local storage.

If you are wanting to use InfluxDB primarily, just set local retention to say 4 hours or something.



在 2018年3月13日星期二 UTC+8下午1:00:12，Dave Cadwallader写道：
Hi,

I think I'm missing something basic about how remote read works.  I set up remote read and remote write to InfluxDB.  I successfully verified that remote write is working by setting up Grafana to query InfluxDB, and I'm seeing Prometheus metrics showing up there. 

So far so good.

Next, I wanted to test out remote read.  My understanding is that when this is set up, Prometheus no longer uses a local TSDB, nor a local cache, and every single query gets fulfilled by the remote read endpoint.  To test this out, I stopped the InfluxDB server.  To my surprise, the PromQL queries in Prometheus continued to work just fine.  I expected the queries to fail since the remote InfluxDB server is unavailable.

Am I misunderstanding how this works?

Config:

remote_write:

remote_read:

Thanks!
Dave
--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


-- 
Stuart Clark

Aliaksandr Valialkin

unread,
May 30, 2019, 5:51:16 AM5/30/19
to Stuart Clark, coolfi...@gmail.com, Prometheus Users

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5262b07e-e219-be91-41b5-e30a3d4c8d6f%40Jahingo.com.

For more options, visit https://groups.google.com/d/optout.


--
Best Regards,

Aliaksandr
Reply all
Reply to author
Forward
0 new messages