Trouble scraping Custom Python Metric in OpenShift 3.11 + Prometheus

[Jeff Tippey]

Hi,

    I have a custom python container I wrote that exports metrics on port 8001 at "/metrics". The setup used to work in Openshift 3.6 with Prometheus.   I'm using the Prometheus configuration from github at prometheus github link.  

   In my deployment yamls for my container,   For the list (in deployment yaml) I have annotation

 annotations:
    prometheus.io/path: /metrics
    prometheus.io/port: "8001"
    prometheus.io/scrape: "true"

and the service definition I have 

 annotations:
      prometheus.io/scrape: "true"
      prometheus.io/scheme: http
      prometheus.io/port: "8001"

To repeat, this worked in 3.6 configuration.  However, when I open up Prometheus this is not a target.  Under Targets, none of my service endpoints are showing up.  I do see them in the service discovery but they are being dropped.  When, I look in the prometheus.yaml, it says that it should scrape anything that is io.scrape = "true".   Has anyone experienced this issue or know why it would be dropping my service endpoint scrapes that are defined as io.scrape="true".

If not, does anyone know where the reason they are dropped is logged or how to access?   I have been looking around and am at a loss right now.

Your help is greatly appreciated,

Thank you,

Jeff 

[Stuart Clark]

If you look through the relabling rules for the job you should be able to see what is going on. Look at the details on the SD page and work out which rule is dropping it. It should then suggest what you need to do to make it work.

[Jeff Tippey]

Thanks Stuart,

    I have looked around and I can't find it.   I'd be grateful if someone could take a look at one of these dropped pods labels and my scrape config and let me know if you see where it is dropped.

Not working Config

     __address__="xxxxxxxx" 
__meta_kubernetes_endpoint_address_target_kind="Pod" 
__meta_kubernetes_endpoint_address_target_name="rcax-ios0-br-1-56z6v"
__meta_kubernetes_endpoint_port_name="queuemonitor"
__meta_kubernetes_endpoint_port_protocol="TCP"
__meta_kubernetes_endpoint_ready="true"
__meta_kubernetes_endpoints_name="rcax-ios-0" 
__meta_kubernetes_namespace="envb" 
__meta_kubernetes_pod_annotation_openshift_io_deployment_config_latest_version="1" 
__meta_kubernetes_pod_annotation_openshift_io_deployment_config_name="rcax-ios0-br" 
__meta_kubernetes_pod_annotation_openshift_io_deployment_name="rcax-ios0-br-1" 
__meta_kubernetes_pod_annotation_openshift_io_scc="restricted" 
__meta_kubernetes_pod_container_name="queuemonitor" 
__meta_kubernetes_pod_container_port_name="" 
__meta_kubernetes_pod_container_port_number="8001" 
__meta_kubernetes_pod_container_port_protocol="TCP" 
__meta_kubernetes_pod_controller_kind="ReplicationController"
__meta_kubernetes_pod_controller_name="rcax-ios0-br-1" 
__meta_kubernetes_pod_host_ip="xxxxxxxx" 
__meta_kubernetes_pod_ip="xxxxxxxx" 
__meta_kubernetes_pod_label_component="rcax-ios0-br"
__meta_kubernetes_pod_label_deployment="rcax-ios0-br-1"
__meta_kubernetes_pod_label_deploymentconfig="rcax-ios0-br" 
__meta_kubernetes_pod_label_scac="rcax" 
__meta_kubernetes_pod_label_subsystem="rcax-ios0"
__meta_kubernetes_pod_name="rcax-ios0-br-1-56z6v"
__meta_kubernetes_pod_node_name="stb-node002"
__meta_kubernetes_pod_ready="true"
__meta_kubernetes_pod_uid="356fe668-5a73-11eb-bf8a-0050568584bc" 
__meta_kubernetes_service_annotation_prometheus_io_port="8001"
__meta_kubernetes_service_annotation_prometheus_io_scheme="http" 
__meta_kubernetes_service_annotation_prometheus_io_scrape="true"
__meta_kubernetes_service_label_name="rcax-ios0"
__meta_kubernetes_service_name="rcax-ios-0" 
__metrics_path__="/metrics" 
__scheme__="http" 
job="kubernetes-service-endpoints" 

Scrape Config

- job_name: 'kubernetes-service-endpoints'
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          # TODO: this should be per target
          insecure_skip_verify: true
        kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        # only scrape infrastructure components
        - source_labels: [__meta_kubernetes_namespace]
          action: keep
          regex: 'default|logging|metrics|kube-.+|openshift|openshift-.+|envb|envc'
        # drop infrastructure components managed by other scrape targets
        - source_labels: [__meta_kubernetes_service_name]
          action: drop
          regex: 'prometheus-node-exporter'
        # only those that have requested scraping
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
          action: replace
          target_label: __scheme__
          regex: (https?)
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          action: keep
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
          action: replace
          target_label: __address__
          regex: (.+)(?::\d+);(\d+)
          replacement: $1:$2
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          action: replace
          target_label: kubernetes_name

[Stuart Clark]

On 04/02/2021 16:42, Jeff Tippey wrote:

Thanks Stuart,

    I have looked around and I can't find it.   I'd be grateful if someone could take a look at one of these dropped pods labels and my scrape config and let me know if you see where it is dropped.

It looks like this bit might be dropping the target:

- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          action: keep
          target_label: __metrics_path__
          regex: (.+)

It is looking for an annotation called prometheus_io_path which is copied to the __metrics_path__ label (which sets the path to scrape).

As this annotation doesn't seem to exist in your example this doesn't match, and because the action is keep the target is dropped.

So try adding that annotation with the correct metric path value.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0598e514-fdd5-449c-aa77-5241957c3446n%40googlegroups.com.

-- 
Stuart Clark

[Jeff Tippey]

Stuart, 

    Thanks a bunch.  That was a huge help.