Clarifying jobs/targets/instances when monitoring multiple systems from a single Prometheus server

[John Dexter]

While I understand the technical definitions I would appreciate some real-world advice in the best way to structure my configuration file for scalability and maintenance.

Our system includes many instrumented modules running on an application server on distinct ports, e.g on APPSERVER001 modules [A, B, C, D, E, ..., Z] expose endpoints APPSERVER001:[1001,1002,1003,1004, 1005, ..., 1026]/metrics 

However, we deploy multiple systems so we actually have APPSERVER001, 002, ... each with the same modules exposing the same endpoints. I might want a dashboard (Grafana) for each system, but I might also want a dashboard displaying data across systems (e.g. show status of module B on all systems) 

Initially we only want to monitor one system as we get this all up and running, before adding other systems over time.

 

Can anyone suggest how they would approach this in terms of job/scrape config? file-based discovery seems a great option but I'm struggling how to avoid a lot of copy-paste and duplicate information. 

Many thanks for your advice.

[Jiacai Liu]

Prometheus have support for k8s/marathon/zk based discovery, since
you ask for monitoring method I assume you haven't used any of
them. I think a (python)script that generate the file-based config
is enough.

[Stuart Clark]

How are your applications deployed and managed? Kubernetes, Docker,
Ansible, Puppet, etc.?

--
Stuart Clark

[John Dexter]

Stuart, this is totally bespoke C++ Windows system,  none of those :) The file-discovery seems able to pull out my targets but I can't find a good example how to structure my files. I suppose each Module A, B, C, ... should be the jobs, but I'd far rather not have to list each of my targets/servers once for every job when it's all the same.

Thanks.

[Stuart Clark]

Yes each module probably wants to be a separate job.

Multiple jobs can use the same YAML/JSON files with file_sd, so you could create a single file (with just the hostnames/IPs) and then use relabling to add the correct port number for each job.

-- 
Stuart Clark

[John Dexter]

I've been looking at the examples here https://prometheus.io/docs/guides/file-sd/ and I cannot see how I would apply relabeling as described - their final example is similar to my case and they duplicate the entire section for the same job on two targets. You seem to be suggesting that could be avoided but I cannot see how. 

[Stuart Clark]

On 03/03/2021 17:18, John Dexter wrote:

--
Stuart Clark

Stuart, this is totally bespoke C++ Windows system,  none of those :) The file-discovery seems able to pull out my targets but I can't find a good example how to structure my files. I suppose each Module A, B, C, ... should be the jobs, but I'd far rather not have to list each of my targets/servers once for every job when it's all the same.

Yes each module probably wants to be a separate job.

Multiple jobs can use the same YAML/JSON files with file_sd, so you could create a single file (with just the hostnames/IPs) and then use relabling to add the correct port number for each job.

-- 
Stuart Clark

I've been looking at the examples here https://prometheus.io/docs/guides/file-sd/ and I cannot see how I would apply relabeling as described - their final example is similar to my case and they duplicate the entire section for the same job on two targets. You seem to be suggesting that could be avoided but I cannot see how. 

In your YAML/JSON files just have the bare IP/hostname (no port) and then within each job you can use relabing to add the correct port number.

-- 
Stuart Clark