Prometheus isn't scraping custom metrics from the pushgateway

[fabio...@gmail.com]

Hi guys, 

I have a deployment of prometheus and pushgateway services, but I cannot list from the prometheus side ("curl localhost:9090/api/v1/label/__name__/values") metrics direct pushed in the pushgateway.  However, these pushed metrics can be shown in locahost:9091/metrics (pushgateway metrics endpoint) and other metrics from pushgateway can be listed using the prometheus API (e.g. pushgateway_build_info).

I'm using these yaml files to deploy the services: 

# prometheus.yaml
global:
  scrape_interval: 15s
scrape_configs:
- job_name: 'kubernetes-pods'
  honor_labels: true
  kubernetes_sd_configs:
  - role: pod
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: pod
  - source_labels: [__meta_kubernetes_job_name]
    action: replace
    target_label: job
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2

# pushgateway.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: pushgateway
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pushgateway
  template:
    metadata:
      labels:
        app: pushgateway
      annotations:
        # based on your Prometheus config above, this tells prometheus
        # to scrape this pod for metrics on port 8080 at "/metrics"
        prometheus.io/scrape: "true"
        prometheus.io/port: "9091"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - image: prom/pushgateway
        name: pushgateway
        ports:
        - name: http
          containerPort: 9091

Can anyone help me to fix this? Getting pushed metrics in the pushgateway by calling the prometheus API. 

Thank you. 

[Björn Rabenstein]

Hi ladies,

On 13 March 2018 at 20:33, <fabio...@gmail.com> wrote:
> Getting pushed metrics in the pushgateway by calling the prometheus API.

Strictly speaking, that's not possible. The Prometheus API has nothing
to do with retrieving metrics from any targets, including the
Pushgateway. Since you do see pushgateway_build_info, you have proven
that your Prometheus server is scraping the /metrics endpoint of the
Pushgateway. Therefore, all the metrics you can see by looking at the
/metrics endpoint of the Pushgateway will be ingested by Prometheus.
Most likely, you are making a mistake during querying. I recommend to
play around with the Web UI of the Prometheus server itself to explore
the metrics.

--
Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales
with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
HRB 110657B

[Sankar]

If prometheus running on port 9090, then 

1. go to prometheus home page -> localhost:9090 -> Stats -> targets : make sure pushgateway is up and no errors
2. go to graph, then you will see lots of items in dropdown, if pushgateway was up unders prometheus -> stats -> targets, then you should see few items in dropdown with pushgateway prefix
3. select anything which starts with pushgateway then click on execute, you see some value, that shows that prometheus scraping pushgateway
4. you can also see custom metrics if you have any

Thanks,

Shankar

shank...@gmail.com