Is file SD still good enough as a replacement for unsupported Discover mechanism?

838 views
Skip to first unread message

Krasi Georgiev

unread,
Jan 29, 2018, 7:27:39 PM1/29/18
to Prometheus Developers
I am interested to hear if  everyone thinks that flie SD is good enough for an answer to unsupported Discovery mechanisms?

The only disadvantage that I see mentioned so far in an issue is that it requires access to the same file system as Prometheus which is not always possible.


a PR with an idea from @Gouthamve for a simple daemon that people can just plug in their SD implementation and it spits out a file-sd.

The official post how to use file SD

Support database query for list of nodes 
https://github.com/prometheus/prometheus/issues/677

Consider reducing number of built-in discovery methods in 2.0 

Feature Request: dynamic target discovery via URL 

Docker engine swarm api service discovery
https://github.com/prometheus/prometheus/issues/1766

Krasi Georgiev

unread,
Jan 29, 2018, 7:40:05 PM1/29/18
to Prometheus Developers

Callum Styan

unread,
Jan 30, 2018, 12:40:07 AM1/30/18
to Krasi Georgiev, Prometheus Developers
I do think it's "good enough" with the addition of the sd adapter for 3658.

Via the adapter people just need to implement the Discoverer interface, as discussed, and then run it. In this scenario I don't see how file_sd having to be on the same filesystem as Prometheus is an issue.
The adapter process can run on the same system as Prometheus and write the file for file_sd, and the actual SD can be running anywhere. It could be a database or it could be via HTTP api as I implemented in the example.

On Mon, Jan 29, 2018 at 4:40 PM, Krasi Georgiev <kr...@vip-consult.solutions> wrote:

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/fd1c5d55-fa45-4db2-b974-ebe65ef89053%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Ben Kochie

unread,
Jan 30, 2018, 4:57:36 AM1/30/18
to Callum Styan, Krasi Georgiev, Prometheus Developers
I've proposed in the past that we add the option for http URLs in the file_sd_configs.  This would allow it to be a somewhat generic way to pull down updates.

There is also some discussion in the CNCF about making a generic discovery API that can be used cross-project.  I don't know what the state of this is.

On Tue, Jan 30, 2018 at 6:40 AM, Callum Styan <callu...@gmail.com> wrote:
I do think it's "good enough" with the addition of the sd adapter for 3658.

Via the adapter people just need to implement the Discoverer interface, as discussed, and then run it. In this scenario I don't see how file_sd having to be on the same filesystem as Prometheus is an issue.
The adapter process can run on the same system as Prometheus and write the file for file_sd, and the actual SD can be running anywhere. It could be a database or it could be via HTTP api as I implemented in the example.
On Mon, Jan 29, 2018 at 4:40 PM, Krasi Georgiev <kr...@vip-consult.solutions> wrote:

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

To post to this group, send email to prometheus-developers@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAN2d5OT%2BXWPKzFqBVQdvpizzJTOQyHBcaAfFxJTUMp-P82SbrA%40mail.gmail.com.

Krasi Georgiev

unread,
Jan 30, 2018, 5:08:41 AM1/30/18
to Prometheus Developers

do you have a link for the CNCF project?



On Tuesday, 30 January 2018 09:57:36 UTC, Ben Kochie wrote:
I've proposed in the past that we add the option for http URLs in the file_sd_configs.  This would allow it to be a somewhat generic way to pull down updates.

There is also some discussion in the CNCF about making a generic discovery API that can be used cross-project.  I don't know what the state of this is.
On Tue, Jan 30, 2018 at 6:40 AM, Callum Styan <callu...@gmail.com> wrote:
I do think it's "good enough" with the addition of the sd adapter for 3658.

Via the adapter people just need to implement the Discoverer interface, as discussed, and then run it. In this scenario I don't see how file_sd having to be on the same filesystem as Prometheus is an issue.
The adapter process can run on the same system as Prometheus and write the file for file_sd, and the actual SD can be running anywhere. It could be a database or it could be via HTTP api as I implemented in the example.
On Mon, Jan 29, 2018 at 4:40 PM, Krasi Georgiev <kr...@vip-consult.solutions> wrote:

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

Ben Kochie

unread,
Jan 30, 2018, 5:58:34 AM1/30/18
to Krasi Georgiev, Richard Hartmann, Prometheus Developers
+RichiH

No, I don't.

On Tue, Jan 30, 2018 at 11:08 AM, Krasi Georgiev <kr...@vip-consult.solutions> wrote:

do you have a link for the CNCF project?


On Tuesday, 30 January 2018 09:57:36 UTC, Ben Kochie wrote:
I've proposed in the past that we add the option for http URLs in the file_sd_configs.  This would allow it to be a somewhat generic way to pull down updates.

There is also some discussion in the CNCF about making a generic discovery API that can be used cross-project.  I don't know what the state of this is.
On Tue, Jan 30, 2018 at 6:40 AM, Callum Styan <callu...@gmail.com> wrote:
I do think it's "good enough" with the addition of the sd adapter for 3658.

Via the adapter people just need to implement the Discoverer interface, as discussed, and then run it. In this scenario I don't see how file_sd having to be on the same filesystem as Prometheus is an issue.
The adapter process can run on the same system as Prometheus and write the file for file_sd, and the actual SD can be running anywhere. It could be a database or it could be via HTTP api as I implemented in the example.
On Mon, Jan 29, 2018 at 4:40 PM, Krasi Georgiev <kr...@vip-consult.solutions> wrote:

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/1ba017cd-2e14-4588-a882-7265dda5ab10%40googlegroups.com.

Krasi Georgiev

unread,
Jan 30, 2018, 6:25:27 AM1/30/18
to Ben Kochie, Richard Hartmann, Prometheus Developers
It seems that initially fabian had an idea to implement SD via grpc which got dropped in favour of the current design.

On Jan 30 2018, at 10:58 am, Ben Kochie <sup...@gmail.com> wrote:
+RichiH

No, I don't.
On Tue, Jan 30, 2018 at 11:08 AM, Krasi Georgiev <kr...@vip-consult.solutions> wrote:

do you have a link for the CNCF project?


On Tuesday, 30 January 2018 09:57:36 UTC, Ben Kochie wrote:
I've proposed in the past that we add the option for http URLs in the file_sd_configs.  This would allow it to be a somewhat generic way to pull down updates.

There is also some discussion in the CNCF about making a generic discovery API that can be used cross-project.  I don't know what the state of this is.
On Tue, Jan 30, 2018 at 6:40 AM, Callum Styan <callu...@gmail.com> wrote:
I do think it's "good enough" with the addition of the sd adapter for 3658.

Via the adapter people just need to implement the Discoverer interface, as discussed, and then run it. In this scenario I don't see how file_sd having to be on the same filesystem as Prometheus is an issue.
The adapter process can run on the same system as Prometheus and write the file for file_sd, and the actual SD can be running anywhere. It could be a database or it could be via HTTP api as I implemented in the example.
On Mon, Jan 29, 2018 at 4:40 PM, Krasi Georgiev <kr...@vip-consult.solutions> wrote:

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

Callum Styan

unread,
Jan 30, 2018, 12:51:18 PM1/30/18
to Krasi Georgiev, Ben Kochie, Richard Hartmann, Prometheus Developers
grpc would be less generic than file_sd or even http though, that PR doesn't seem related though?

On Tue, Jan 30, 2018 at 3:25 AM, Krasi Georgiev <kr...@vip-consult.solutions> wrote:
It seems that initially fabian had an idea to implement SD via grpc which got dropped in favour of the current design.

On Jan 30 2018, at 10:58 am, Ben Kochie <sup...@gmail.com> wrote:
+RichiH

No, I don't.
On Tue, Jan 30, 2018 at 11:08 AM, Krasi Georgiev <kr...@vip-consult.solutions> wrote:

do you have a link for the CNCF project?


On Tuesday, 30 January 2018 09:57:36 UTC, Ben Kochie wrote:
I've proposed in the past that we add the option for http URLs in the file_sd_configs.  This would allow it to be a somewhat generic way to pull down updates.

There is also some discussion in the CNCF about making a generic discovery API that can be used cross-project.  I don't know what the state of this is.
On Tue, Jan 30, 2018 at 6:40 AM, Callum Styan <callu...@gmail.com> wrote:
I do think it's "good enough" with the addition of the sd adapter for 3658.

Via the adapter people just need to implement the Discoverer interface, as discussed, and then run it. In this scenario I don't see how file_sd having to be on the same filesystem as Prometheus is an issue.
The adapter process can run on the same system as Prometheus and write the file for file_sd, and the actual SD can be running anywhere. It could be a database or it could be via HTTP api as I implemented in the example.
On Mon, Jan 29, 2018 at 4:40 PM, Krasi Georgiev <kr...@vip-consult.solutions> wrote:

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/local-3c9c3e2c-28c2%40lenovo-p50.

ca...@strayorange.com

unread,
Feb 8, 2018, 6:21:49 PM2/8/18
to Prometheus Developers
On Tuesday, January 30, 2018 at 9:27:39 AM UTC+9, Krasi Georgiev wrote:
> I am interested to hear if  everyone thinks that flie SD is good enough for an answer to unsupported Discovery mechanisms?
>
> The only disadvantage that I see mentioned so far in an issue is that it requires access to the same file system as Prometheus which is not always possible.

This is my biggest concern. In our case this is simply not practical (put it otherwise: it's not impossible but very impractical and violates the separation of concerns).

I agree that it would probably be best if all SD implementations lived externally to the Prometheus tree and all were refactored to use a stable interface, exposed by Prometheus, that does not require access to the filesystem of the Prometheus server.

krasi...@gmail.com

unread,
Feb 12, 2018, 8:04:11 AM2/12/18
to Prometheus Developers
after gathering some more context it seems that the only other option would be a HTTP based Discoverer where Prometheus calls the remote/local HTTP server to get the targets at a set interval.

It was also mentioned that http based method would have the disadvantage of a constantly running http server to provide the targets, but I don't see a problem in that.

To me it seems that the HTTP pull method fits better in the micro services domain that Prometheus is targeting.

Brian Brazil

unread,
Feb 12, 2018, 8:23:33 AM2/12/18
to Krasi Georgiev, Prometheus Developers
On 12 February 2018 at 13:04, <krasi...@gmail.com> wrote:
after gathering some more context it seems that the only other option would be a HTTP based Discoverer where Prometheus calls the remote/local HTTP server to get the targets at a set interval.

It was also mentioned that http based method would have the disadvantage of a constantly running http server to provide the targets, but I don't see a problem in that.

That's a notable disadvantage, it doesn't persist across restarts for example.

To me it seems that the HTTP pull method fits better in the micro services domain that Prometheus is targeting.

Prometheus has more than one target.

Brian
 


On Tuesday, 30 January 2018 02:27:39 UTC+2, Krasi Georgiev wrote:
I am interested to hear if  everyone thinks that flie SD is good enough for an answer to unsupported Discovery mechanisms?

The only disadvantage that I see mentioned so far in an issue is that it requires access to the same file system as Prometheus which is not always possible.


a PR with an idea from @Gouthamve for a simple daemon that people can just plug in their SD implementation and it spits out a file-sd.

The official post how to use file SD

Support database query for list of nodes 
https://github.com/prometheus/prometheus/issues/677

Consider reducing number of built-in discovery methods in 2.0 

Feature Request: dynamic target discovery via URL 

Docker engine swarm api service discovery
https://github.com/prometheus/prometheus/issues/1766

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

krasi...@gmail.com

unread,
Feb 12, 2018, 8:32:18 AM2/12/18
to Prometheus Developers

It was also mentioned that http based method would have the disadvantage of a constantly running http server to provide the targets, but I don't see a problem in that.

That's a notable disadvantage, it doesn't persist across restarts for example.

what do we need to persist?
if Prometheus restarts it will query to get all targets and if the http server restarts it will be designed in a way to return all targets at the first call.
 

To me it seems that the HTTP pull method fits better in the micro services domain that Prometheus is targeting.

Prometheus has more than one target.

could you mention any usage that would work better with the current file based SD vs the http pull ?

Brian Brazil

unread,
Feb 12, 2018, 8:34:53 AM2/12/18
to Krasi Georgiev, Prometheus Developers
Something like Ansible or Chef writing out the targets to a file, which is fairly common.

Brian
 

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

krasi...@gmail.com

unread,
Feb 12, 2018, 9:05:58 AM2/12/18
to Prometheus Developers


On Monday, 12 February 2018 15:34:53 UTC+2, Brian Brazil wrote:
On 12 February 2018 at 13:32, <krasi...@gmail.com> wrote:

It was also mentioned that http based method would have the disadvantage of a constantly running http server to provide the targets, but I don't see a problem in that.

That's a notable disadvantage, it doesn't persist across restarts for example.

what do we need to persist?  
if Prometheus restarts it will query to get all targets and if the http server restarts it will be designed in a way to return all targets at the first call.
 

To me it seems that the HTTP pull method fits better in the micro services domain that Prometheus is targeting.

Prometheus has more than one target.

could you mention any usage that would work better with the current file based SD vs the http pull ?

Something like Ansible or Chef writing out the targets to a file, which is fairly common.

yep that is a good use case. 
Isn't this the opposite of the  pull based model that prometheus is following? 
In other words ansible/chef need to know about all prometheus instances and write the json targets to each one.

The same use case using  http would look like
1. Ansible/chef write the json targets file in the http discoverer or can even call the api directly if it doesn't have access the file system.
2. All prometheus instances will query this same discoverer which will return the expected targets taking them from the written json file.

Brian Brazil

unread,
Feb 12, 2018, 9:27:54 AM2/12/18
to Krasi Georgiev, Prometheus Developers
On 12 February 2018 at 14:05, <krasi...@gmail.com> wrote:


On Monday, 12 February 2018 15:34:53 UTC+2, Brian Brazil wrote:
On 12 February 2018 at 13:32, <krasi...@gmail.com> wrote:

It was also mentioned that http based method would have the disadvantage of a constantly running http server to provide the targets, but I don't see a problem in that.

That's a notable disadvantage, it doesn't persist across restarts for example.

what do we need to persist?  
if Prometheus restarts it will query to get all targets and if the http server restarts it will be designed in a way to return all targets at the first call.
 

To me it seems that the HTTP pull method fits better in the micro services domain that Prometheus is targeting.

Prometheus has more than one target.

could you mention any usage that would work better with the current file based SD vs the http pull ?

Something like Ansible or Chef writing out the targets to a file, which is fairly common.

yep that is a good use case. 
Isn't this the opposite of the  pull based model that prometheus is following? 
In other words ansible/chef need to know about all prometheus instances and write the json targets to each one.

I don't follow you here, your service databases always need to know about what instances are running. This is not an additional burden, as it's already doing this for the configuration file etc.

The same use case using  http would look like
1. Ansible/chef write the json targets file in the http discoverer or can even call the api directly if it doesn't have access the file system.
2. All prometheus instances will query this same discoverer which will return the expected targets taking them from the written json file.

In such an environment I'd expect that to boil down to running a HTTP server on every machine Prometheus is running on. Systems where is is impossible to share filesystems between processes are rare, and I don't think we should base our decisions on extreme cases.

--

Nat Welch

unread,
Feb 12, 2018, 2:14:39 PM2/12/18
to Brian Brazil, Krasi Georgiev, Prometheus Developers
I feel like many docker as a service providers make it a pain to share filesystems (GKE & ECS as two largish examples). I would love to have http based service discovery. Right now Prometheus is the only thing not running inside of docker because of this requirement to have multiple processes accessing the same filesystem. 

/Nat

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLqRqnrX2kZvfC_15h0CLU07w3R_qwo2aCXBNCn1jLYBTQ%40mail.gmail.com.

krasi...@gmail.com

unread,
Feb 12, 2018, 2:19:40 PM2/12/18
to Prometheus Developers
Thanks for the feedback Nat.

Since we want to support only one custom SD It would be nice to hear some Ansible/Chef users if HTTP can be made to work well in those cases as well.

Am also very interested in Ben's comment about CNCF making a generic discovery API , but couldn't find any info at all.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

Brian Brazil

unread,
Feb 12, 2018, 2:37:42 PM2/12/18
to Krasi Georgiev, Prometheus Developers
On 12 February 2018 at 19:19, <krasi...@gmail.com> wrote:
Thanks for the feedback Nat.

Since we want to support only one custom SD It would be nice to hear some Ansible/Chef users if HTTP can be made to work well in those cases as well.

The basis of Unix configuration is textfiles, why should configuration on a Unix system require a HTTP server? It'd be an extra moving part for no particular reason.

Files on disk are a more generic approach than HTTP, as they don't require a server to be running at all times.

Brian
 

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Callum Styan

unread,
Feb 12, 2018, 6:04:15 PM2/12/18
to Brian Brazil, Krasi Georgiev, Prometheus Developers
I also still think file sd is good enough. We also haven't even shipped the adapter yet. 

I think we're should just ship that and see what the adoption is like. We don't know if the filesystem thing is even an issue beyond the two or three people who are mentioning it right now. 
To post to this group, send email to prometheus-developers@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLpCME6kVrDhcvLJtgS9Nik1EBhNot%3DZgiXjCR_04Uc_3w%40mail.gmail.com.

Matt Palmer

unread,
Feb 13, 2018, 9:08:22 PM2/13/18
to Prometheus Developers
On Mon, Feb 12, 2018 at 11:19:40AM -0800, krasi...@gmail.com wrote:
> Since we want to support only one custom SD It would be nice to hear some
> Ansible/Chef users if HTTP can be made to work well in those cases as well.

I, for one, am strongly opposed to making a generic HTTP request "the"
Prometheus SD mechanism. If your chosen deployment mechanism can't handle
sharing a configuration file between two units of deployment, you are
screwed for *so* many more things than just Prometheus. I can't think of
*any* other service whose sole means of configuration is "make a request to
a HTTP server".

- Matt

carloalber...@gmail.com

unread,
Feb 13, 2018, 9:27:07 PM2/13/18
to Prometheus Developers
On Wednesday, February 14, 2018 at 11:08:22 AM UTC+9, Matt Palmer wrote:

> On Mon, Feb 12, 2018 at 11:19:40AM -0800, ***@gmail.com wrote:
> If your chosen deployment mechanism can't handle
> sharing a configuration file between two units of deployment, you are
> screwed for *so* many more things than just Prometheus. I can't think of
> *any* other service whose sole means of configuration is "make a request to
> a HTTP server".

How about any IaaS/CaaS/PaaS platform? Note that existence of *most* other SD mechanisms in Prometheus contradicts your point... To prove me wrong, feel free to implement service discovery for e.g. all apps on a kubernetes deployment using just the file SD and *without* calling any form of API. To preempt pointless answers, assume realistic conditions in which you add/remove apps and versions, change configurations, do blue/green deploys, scale out/in your apps and so on.

If you then want to even see things from my perspective, do this for apps you don't control (i.e. whose lifecycle events I described above are not done by you but by others).

Sure, I'm absolutely aware that not every prometheus user has to deal with this kind of complexity. But I'm not sure demeaning the requirements of others just because they don't align with yours (as is being done in this thread and in the linked github issues) is a policy that is going to solve any problem [1].

[1] https://github.com/prometheus/prometheus/commit/81db4716c1a95db540a1dd874f8e66e57a1f9dc1

Matt Palmer

unread,
Feb 14, 2018, 1:21:28 AM2/14/18
to Prometheus Developers
On Mon, Feb 12, 2018 at 07:14:26PM +0000, Nat Welch wrote:
> I feel like many docker as a service providers make it a pain to share
> filesystems (GKE & ECS as two largish examples).

What? GKE is Kubernetes, and Kubernetes has "filesystem sharing" built
right in, it's a core part of the whole "pod" concept. Run the
thing-that-writes-the-SD-file in the same pod as Prometheus itself ->
problem solved. If your containerisation system of choice doesn't support
something similar to Kubernetes pods, you really need to get a better one --
it's a very useful and powerful concept.

- Matt
(not a Kubernetes user, BTW)

Carlo Alberto Ferraris

unread,
Feb 14, 2018, 1:42:39 AM2/14/18
to Matt Palmer, Prometheus Developers
Matt,
I see that the part in which I suggested not to demean other’s requirements completely went over your head… I would like to ask you to avoid poisoning the discussion, if possible.

Anyway what would the "thing-that-writes-the-SD-file" be if not a daemon that polls the kubernetes API (remember you said this would be completely *crazy* in your previous mail) and then writes to the filesystem? Also what if your prometheus is not running on kubernetes? Or what if you’re running on a IaaS, where the idea of accessing the filesystem of another VM is not exactly an option (sure you have NFS, but that’s again a network protocol *hint* *hint*).

Just to clarify: obviously it can be done *somehow*. But the question that this discussion is trying to answer is: is forcing users who have dynamic systems to write files really the best we can do, especially since often the "thing-that-writes-the-SD-file” has to call remote APIs *anyway* just to then spit out a file? Nobody is arguing (I think) that the file SD is not useful, just that *maybe* it’s not a very good candidate as “the most generic solution”.

- Carlo
(not a Kubernetes user, either)
(nor a Prometheus user, just trying to help our common users)

Brian Brazil

unread,
Feb 14, 2018, 3:46:52 AM2/14/18
to Carlo Alberto Ferraris, Matt Palmer, Prometheus Developers
Setups where you are incapable of sharing filesystems are rather niche, and if someone chose to deploy in such a limited environment then they also implicitly took on the burden of dealing with the many many pieces of software that require shared filesystem access (e.g. MySQL backups). I don't think we should make things harder for users in simple standard scenarios for the sake of making things slightly easier for users who already have complex setups.

Brian
 

- Carlo
(not a Kubernetes user, either)
(nor a Prometheus user, just trying to help our common users)
--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/4FC7D32A-BE3B-4802-80FF-4B0EFB8765EA%40strayorange.com.

For more options, visit https://groups.google.com/d/optout.



--

krasi...@gmail.com

unread,
Feb 14, 2018, 4:37:41 AM2/14/18
to Prometheus Developers
really like the heated discussion :)

there is no doubt that file based SD is a blocker for some users, but what I am really hoping to find out is if http would be a blocker for other users as well.
Brian Brazil mentioned ansible and chef so It would be good to hear someone on that front.

It would be in best interest for everyone not try and insist on personal opinions, but guided by real use cases.

Krasi Georgiev

Carlo Alberto Ferraris

unread,
Feb 14, 2018, 5:59:14 AM2/14/18
to krasi...@gmail.com, Prometheus Developers
Good point about real use cases, and I’m sorry if I got focused on suggesting solutions before stating the problem. 

My users' use case is doing automated service discovery of their apps running on Cloud Foundry. Cloud Foundry (like Kubernetes) exposes all the information required for this task via its HTTP APIs. I would rather prefer to provide a hands-off experience to our users, avoiding if possible the need to run a daemon on their Prometheus server (that is on different infrastructure).

This wouldn’t be all that’s required BTW, because currently Prometheus is unable to reach individual Cloud Foundry instances because it can’t set the appropriate headers (https://github.com/prometheus/prometheus/issues/1724). But that’s a much easier problem to solve, even without cooperation from Prometheus, while maintaining proper separation of concerns.

Carlo

-- 
You received this message because you are subscribed to a topic in the Google Groups "Prometheus Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-developers/JcE5nSbCe4k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-devel...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

David Karlsen

unread,
Feb 14, 2018, 1:42:12 PM2/14/18
to krasi...@gmail.com, Prometheus Developers
We do this (although with saltstack). We query our infra about different facts, and based on that we render config files for sd_file. It runs from cron so that it converges. Works very well. We also have some consul integration - which works very smooth too.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/091d2b27-1ee1-4eee-ab6e-335549b9a373%40googlegroups.com.

Nicholas Capo

unread,
Feb 14, 2018, 1:55:36 PM2/14/18
to David Karlsen, Prometheus Developers, krasi...@gmail.com
I'm doing an almost identical thing (also with SaltStack).

Most of our targets come from consul, but all node exporters are "discovered" through salt and then written to a file on each Prometheus server.

Nicholas

On Wed, Feb 14, 2018, 12:42 David Karlsen <davidk...@gmail.com> wrote:
We do this (although with saltstack). We query our infra about different facts, and based on that we render config files for sd_file. It runs from cron so that it converges. Works very well. We also have some consul integration - which works very smooth too.
Den ons. 14. feb. 2018, 10:37 skrev <krasi...@gmail.com>:
really like the heated discussion :)

there is no doubt that file based SD is a blocker for some users, but what I am really hoping to find out is if http would be a blocker for other users as well.
Brian Brazil mentioned ansible and chef so It would be good to hear someone on that front.

It would be in best interest for everyone not try and insist on personal opinions, but guided by real use cases.

Krasi Georgiev

On Wednesday, 14 February 2018 08:21:28 UTC+2, Matt Palmer wrote:
On Mon, Feb 12, 2018 at 07:14:26PM +0000, Nat Welch wrote:
> I feel like many docker as a service providers make it a pain to share
> filesystems (GKE & ECS as two largish examples).

What?  GKE is Kubernetes, and Kubernetes has "filesystem sharing" built
right in, it's a core part of the whole "pod" concept.  Run the
thing-that-writes-the-SD-file in the same pod as Prometheus itself ->
problem solved.  If your containerisation system of choice doesn't support
something similar to Kubernetes pods, you really need to get a better one --
it's a very useful and powerful concept.

- Matt
(not a Kubernetes user, BTW)

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAGO7Ob37CPmHtxNbuiKyHj_5rh%3D_f%2BR8gFUgBhg1nSU65J5zMQ%40mail.gmail.com.

krasi...@gmail.com

unread,
Feb 14, 2018, 5:10:20 PM2/14/18
to Prometheus Developers
What if Prometheus could call an HTTP endpoint that returns a json with all scrape targets. 
Would this simplify or complicate your setup?


Nicholas

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

David Karlsen

unread,
Feb 14, 2018, 5:28:33 PM2/14/18
to krasi...@gmail.com, Prometheus Developers
It would not do anything for us - with files it is very versatile - and besides salt-stack adds a lot of other versatile functions which it is good at - so just stuffing the files a place where prometheus looks for them is a very loosely coupled interface for us.

Nicholas

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/34c0e9b3-1cfb-4082-b025-a5a9d9b0f2ff%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
--
David J. M. Karlsen - http://www.linkedin.com/in/davidkarlsen

Carlo Alberto Ferraris

unread,
Feb 14, 2018, 5:44:35 PM2/14/18
to krasi...@gmail.com, Prometheus Developers
That would be great (at least for our use case)

Carlo

You received this message because you are subscribed to a topic in the Google Groups "Prometheus Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-developers/JcE5nSbCe4k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-devel...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

Nicholas Capo

unread,
Feb 14, 2018, 6:40:37 PM2/14/18
to Carlo Alberto Ferraris, Prometheus Developers, krasi...@gmail.com
It makes sense to me, though as a solution to my current need, it would complicate things a bit.

Nicholas

Nicholas

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Prometheus Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-developers/JcE5nSbCe4k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-devel...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

nam...@gmail.com

unread,
Feb 21, 2018, 11:08:31 PM2/21/18
to Prometheus Developers
在 2018年2月15日星期四 UTC+8上午6:10:20，krasi...@gmail.com写道：
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
>
> To post to this group, send email to prometheus...@googlegroups.com.
>
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/091d2b27-1ee1-4eee-ab6e-335549b9a373%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
>
> To post to this group, send email to prometheus...@googlegroups.com.
>
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAGO7Ob37CPmHtxNbuiKyHj_5rh%3D_f%2BR8gFUgBhg1nSU65J5zMQ%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.

It would be great to support the HTTP.

brad.ry...@gmail.com

unread,
Mar 31, 2018, 3:54:26 PM3/31/18
to Prometheus Developers
I came across https://github.com/prometheus/prometheus/issues/3602 because I was also looking for dynamic target discovery via URL. We have a system that automatically provisions and de-provisions servers (Krasi is familiar with it) and we want to provide prometheus with a way to dynamically discover these servers and gather metrics. Since prometheus already pulls metrics via URL, target discovery via URL seemed like a good approach (admittedly I do not understand the technical implications for prometheus). I did suggest the cron approach to my community but received pushback.

I just wanted to voice my support for http on behalf of the community I represent. I am happy to provide additional details and sample use cases, and ultimately look forward to whatever solution the prometheus community decides to implement. Thanks everyone for the consideration :)

Krasimir Georgiev

unread,
Apr 1, 2018, 3:09:09 AM4/1/18
to brad.ry...@gmail.com, Prometheus Developers
Yes I much prefer http based SD as well, but based on the feedback we decided to develop a sidecar that will allow anyone to implement their preferred SD method and spit out a json file that will be consumed by the prometheus file SD.


Krasi Georgiev
On Mar 31 2018, at 10:54 pm, brad.ry...@gmail.com wrote:
I came across https://github.com/prometheus/prometheus/issues/3602 because I was also looking for dynamic target discovery via URL. We have a system that automatically provisions and de-provisions servers (Krasi is familiar with it) and we want to provide prometheus with a way to dynamically discover these servers and gather metrics. Since prometheus already pulls metrics via URL, target discovery via URL seemed like a good approach (admittedly I do not understand the technical implications for prometheus). I did suggest the cron approach to my community but received pushback.

I just wanted to voice my support for http on behalf of the community I represent. I am happy to provide additional details and sample use cases, and ultimately look forward to whatever solution the prometheus community decides to implement. Thanks everyone for the consideration :)

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages