Can't remove metrics from pushgateway

[Eduard Ahmatgareev]

Hi All,

I need your help to find a way how I can remove metrics from pushgateway
I have simple python scripts:

#!/usr/bin/env python

import os

from prometheus_client import CollectorRegistry, Gauge, push_to_gateway

registry = CollectorRegistry()

pushgateway_endpoint = os.getenv('pushgateway_endpoint', 'localhost:9091')

gauge_test_metrics = Gauge("test_metrics", "Random count", ['instance'], registry=registry)

gauge_test_metrics.labels( "TEST" ).set(10)

push_to_gateway(pushgateway_endpoint, job='ec2_stats', registry=registry)


When I run it, it will push metric to my local pushgateway:

curl -s http://localhost:9091/metrics | grep "TEST"                                                                     

test_metrics{instance="TEST",job="ec2_stats"} 10

in this step, everything looks fine, but now, I need to remove this metric:

I used command:

curl -X DELETE 'http://localhost:9091/metrics/job/ec2_stats/instance/TEST'                      

But, I checked again:

curl -s http://localhost:9091/metrics | grep "TEST"                                                                     

test_metrics{instance="TEST",job="ec2_stats"} 10

and metric still exists, nothing changed.

What am I doing wrong?

metrics which I pushed to pushgateway from command line, successfully removed, but metrics which I pushed from python script, I can't remove 

Docker compose for pushgateway:

version: '2.1'

networks:

  monitor-net:

    driver: bridge

volumes:

    prometheus_data: {}

    grafana_data: {}

services:

  pushgateway:

    image: prom/pushgateway

    container_name: pushgateway

    restart: unless-stopped

    command:

      - '--web.enable-admin-api'

      - '--log.level=debug'

    expose:

      - 9091

    ports:

      - "9091:9091"

    networks:

      - monitor-net

[Brian Candler]

What's the HTTP response from the curl -X DELETE?  Try adding -v, i.e. curl -v -X DELETE ...

On Monday, 30 August 2021 at 17:50:48 UTC+1 e.ahma...@gmail.com wrote:

metrics which I pushed to pushgateway from command line, successfully removed, but metrics which I pushed from python script, I can't remove 

Then there's something different between those two metrics.  Use tcpdump to capture the creation of the metric from curl, and the creation of the metric from python, and see what's different between them.

e.g. (Linux)  tcpdump -i lo -s0 -nn -A tcp port 9091