How do you add scrape targets to a Prometheus server that was installed with Kubernetes-helm?

[Zhang Zhao]

Background: 

I have installed Prometheus on AKS using the helm chart stable/prometheus-operator.

The Problem:

I had couple of VMs (Linux and Windows), where node exporter and wmi exporter had installed that need to connect to the Prometheus running on Kubernetes. I cannot figure out how to add scrape targets to the Prometheus server. The prometheus.io site describes how I can mount a prometheus.yml file (which contains a list of scrape targets) to a Prometheus Docker container -- I have done this locally and it works. However, I don't know how to specify scrape targets for a Prometheus setup installed via Kubernetes-Helm. I am also not clear on how to expose metrics in a Kubernetes Pod.

Zhang

[赵一晗]

There are two ways to solve this problem：

1. servicemoniors；

    You can create a corresponding crd servicemonior 

2. additionalScrapeConfigs
     you could  add additionalScrapeConfigs to your prometheus

If the additional collection point is outside the cluster ，I think the second one  additionalScrapeConfigs is more suitable，vice versa

在 2020年6月12日星期五 UTC+8上午5:18:33，Zhang Zhao写道：

[Zhang Zhao]

Thanks for you advice. Yes, the VMs are outside the cluster.

For adding additional scrape configs, what file specifically shall I update? Are you using stable/prometheus-operator Helm chart to deploy Prometheus as well?

-- 
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/78ac1367-4cf4-428f-9471-22281779bdc7o%40googlegroups.com.

[Zhang Zhao]

Hi,

Could you please gimme an example how did you configure the additionalScrapeConfigs? My testing case is quite simple. I wanted to feed 2 VMs (one Windows wmi exporter, one Linux node exporter) metrics to Prometheus on Azure Kubernetes, which was deployed by stable/prometheus-operator helm chart.

I see in the values.yaml file, there is a section for additionalScrapeConfigs below.. Shall I add config in the values.yaml directly? Or I have to create a prometheus-additional.yaml like below and make a secret out of this configuration? The doc is not clear. If I do this way, how does Prometheus pick this additional scrape_config up after secret is created??

prometheus-additional.yaml

- job_name: “node_exporter”

  static_configs:

  - targets: ['xx.xx.xx.xx:9100']

Default values.yaml

## The scrape configuraiton example below will find master nodes, provided they have the name .*mst.*, relabel the

    ## port to 2379 and allow etcd scraping provided it is running on all Kubernetes master nodes

    ##

    additionalScrapeConfigs: []

    # - job_name: kube-etcd

    #   kubernetes_sd_configs:

    #     - role: node

    #   scheme: https

    #   tls_config:

    #     ca_file:   /etc/prometheus/secrets/etcd-client-cert/etcd-ca

    #     cert_file: /etc/prometheus/secrets/etcd-client-cert/etcd-client

    #     key_file:  /etc/prometheus/secrets/etcd-client-cert/etcd-client-key

    #   relabel_configs:

    #   - action: labelmap

    #     regex: __meta_kubernetes_node_label_(.+)

    #   - source_labels: [__address__]

    #     action: replace

    #     targetLabel: __address__

    #     regex: ([^:;]+):(\d+)

    #     replacement: ${1}:2379

    #   - source_labels: [__meta_kubernetes_node_name]

    #     action: keep

    #     regex: .*mst.*

    #   - source_labels: [__meta_kubernetes_node_name]

    #     action: replace

    #     targetLabel: node

    #     regex: (.*)

    #     replacement: ${1}

    #   metric_relabel_configs:

    #   - regex: (kubernetes_io_hostname|failure_domain_beta_kubernetes_io_region|beta_kubernetes_io_os|beta_kubernetes_io_arch|beta_kubernetes_io_instance_type|failure_domain_beta_kubernetes_io_zone)

    #     action: labeldrop

Zhang

On Jun 11, 2020, at 9:01 PM, 赵一晗 <zhaoyi...@gmail.com> wrote: