prometheus - kubernetes - eureka discovery

[fernando dominguez]

hi all,
i'm trying to configure  prometheus with eureka running in kubernetes cluster
my configuration is
1 prometheus server outside kubernetes cluster
1 eureka app running in kubernetes cluster
1-n apps running in kubernetes cluster

apps are instrumentalized with micrometer library
eureka is registering apps
in external prometheus i configured a eureka sd_eureka pointing to eureke ingress route configured in kubernetes cluster and i can get all apps properties
but my problem is that  i can't get metrics for my apps because
the url metrics is the internal pod configured in kubernetes  and it cant be reachable out of cluster.

someone can help me with it or point me in the right direction

thanks in advanced

[Matthias Rampke]

Whether this is possible at all depends on your Kubernetes network setup. What are you using there?

Can you ping and curl a pod by hand, using its IP, from the Prometheus box? If so, you can use relabeling to set __address__ based on __meta_eureka_app_instance_ip_addr and __meta_eureka_app_instance_port.

If your pods are not individually reachable from outside the cluster (this is usually the case if you are using an overlay network), you will have to make it somehow possible for Prometheus. The easiest way may be to run it in Kubernetes itself as well.

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAC_bq%3DX%2BJQyx4cYeFRB-ZftrjzEW3s18pv3E8jZkhODDauumHg%40mail.gmail.com.

[fernando dominguez]

thanks for reply

no, ping is not reachable i guess i have 2 options

1. deploy prometheus inside kubernetes cluster and use this conf in prometheus.yml

# Discover targets from Eureka and scrape metrics from them (Whitebox monitoring)
- job_name: eureka-discovery
  scrape_interval: 15s
  scrape_timeout: 15s
  eureka_sd_configs:
  - server: http://<eureka_ip:port>
    refresh_interval: 30s
  relabel_configs:
  - source_labels: [__meta_eureka_app_name]
    target_label: application
  - source_labels: [__meta_eureka_app_instance_metadata_metrics_path]
    target_label: __metrics_path__
  - source_labels: [__address__]
    target_label: instance

2. prometheus federation? not sure if it will work. im considering this option because we have 2 kubernetes cluster where we can  centralize and unify all information , dashboards ...

i guess it will be more easy from and administrative point of view

thnks in advance

 

[Matthias Rampke]

Prometheus works best when it is deployed in each cluster. For global visibility there are a few options besides federation (which is indeed limiting for this use case).

If you don't need queries or graphs across clusters, Grafana with multiple data sources works well, especially if you give them consistent names and use data source variables.

Or consider Thanos as a "front" to multiple servers. It can give you a single query endpoint that goes out to all clusters under the hood.

/MR

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAC_bq%3DU8N2ZQW-ca%2BWj3RmRhFJ2F9ZtTZ_cEeGHRqW8b4PY2bA%40mail.gmail.com.