Unable to scrape metrics from pods

[thomas.k...@gmail.com]

I am able to scrape Prometheus metrics from a Kubernetes service using this Prometheus job configuration:

- job_name: 'prometheus-potapi'
  static_configs:
  - targets: ['potapi-service.potapi:1234']

It uses Kubernetes DNS and it gives me the metrics from any of my three pods I use for my service.

I would like to see the result for each pod.

I am able to see the data I want using this configuration:

- job_name: 'prometheus-potapi-pod'
  static_configs:
  - targets: ['10.1.0.126:1234']

I have searched and experimented using the service discovery mechanism available in Prometheus. Unfortunately, I don't understand how it should be setup. The service discovery reference, https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml, isn't really helpful if you don't know how it works.

I am looking for an example where the job using the IP number is replaced with some service discovery mechanism. Specifying the IP was enough for me to see that the data I'm looking for is exposed.

The pods I want to scrape metrics from all live in the same namespace, potapi.

The metrics are always exposed through the same port, 1234.

Finally, the are all named like this:

potapi-deployment-754d96f855-lkh4x
potapi-deployment-754d96f855-pslgg
potapi-deployment-754d96f855-z2zj2

When I do

kubectl describe pod potapi-deployment-754d96f855-pslgg -n potapi

I get this description:

Name:           potapi-deployment-754d96f855-pslgg
Namespace:      potapi
Node:           docker-for-desktop/192.168.65.3
Start Time:     Tue, 07 Aug 2018 14:18:55 +0200
Labels:         app=potapi
  pod-template-hash=3108529411
Annotations:    <none>
Status:         Running
IP:             10.1.0.127
Controlled By:  ReplicaSet/potapi-deployment-754d96f855
Containers:
  potapi:
    Container ID:   docker://72a0bafbda9b82ddfc580d79488a8e3c480d76a6d17c43d7f7d7ab18458c56ee
    Image:          potapi-service
    Image ID:       docker://sha256:d64e94c2dda43c40f641008c122e6664845d73cab109768efa0c3619cb0836bb
    Ports:          4567/TCP, 4568/TCP, 1234/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Tue, 07 Aug 2018 14:18:57 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-4fttn (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          True
  PodScheduled   True
Volumes:
  default-token-4fttn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-4fttn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
  node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

How would you rewrite the job definition given these prerequisites?

Thomas

[Cody Boggs]

You'll want to take a peek at the Kubernetes Service Discovery config options. I think the one you're looking for specifically is Endpoints. Let me know if that doesn't lead you where you want to go.

Cody Boggs | Yak Shaver (Ops)

FreshTracks.io

Intelligent Alerting for Kubernetes and Prometheus

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/083eeced-36d0-431f-b29b-adf244096960%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thomas Sundberg]

On Tue, 14 Aug 2018 at 20:26, Cody Boggs <co...@freshtracks.io> wrote:
>
> You'll want to take a peek at the Kubernetes Service Discovery config options. I think the one you're looking for specifically is Endpoints. Let me know if that doesn't lead you where you want to go.
>

Thank you!

I ended up here:
https://stackoverflow.com/questions/51731112/unable-to-scrape-metrics-from-pods

Cheers,
Thomas

>> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
>> To post to this group, send email to promethe...@googlegroups.com.

>> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/083eeced-36d0-431f-b29b-adf244096960%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>

--
Thomas Sundberg
M. Sc. in Computer Science

Mobile: +46 70 767 33 15
Blog: http://thomassundberg.wordpress.com/
Twitter: @thomassundberg

Better software through faster feedback