need create custom metrics

[Deekshith V]

How to write Custom - exporter for specific application? We need create custom metrics to full fill our application level metrics .

[Brian Candler]

See: https://prometheus.io/docs/instrumenting/clientlibs/

Choose your programming language, write your exporter using the prometheus client library for that language. There are various tutorials, e.g.

https://prometheus.io/docs/guides/go-application/

At the end of the day, an exporter is just a HTTP server that returns a response body containing prometheus metrics exposition format. The client libraries can just make things a little easier, e.g. maintaining counters for you.

Some alternative approaches you could also consider:

- use node_exporter's textfile collector. Then you just need to write metrics to a file, and node_exporter will pick them up automatically.

- use exporter_exporter, which is able to exec a script

- use one of the other generic exporters like statsd_exporter or pushgateway, and write your metrics to that (where they will persist, waiting for prometheus to scrape them)

[Deekshith V]

Thanks a lot for your valuable inputs and suggestions. 

query1:

I had created with python with  Prometheus-client library,  I want to execute this lib as Pod so need to create docker image(python,prometheus-client ).

here size of the image is going up to 130+mb and also CPU core utilization is more. Only for few 4 to 6 custom metrics its consuming approximate 1 core.

Do you have any idea to minimize the CPU utilization

query2:

When I am using this lib , I can still see some unwanted metrics are populating other than mu custom metrics. Do you have any idea like jow I can remove it from my metrics list?

Once again thanks a lot @Brian Candler

[Brian Candler]

On Thursday, 21 April 2022 at 19:40:07 UTC+1 deeshu...@gmail.com wrote:

Thanks a lot for your valuable inputs and suggestions. 

query1:

I had created with python with  Prometheus-client library,  I want to execute this lib as Pod so need to create docker image(python,prometheus-client ).

here size of the image is going up to 130+mb and also CPU core utilization is more. Only for few 4 to 6 custom metrics its consuming approximate 1 core.

Do you have any idea to minimize the CPU utilization

In principle it should use zero cores while it's idle.  Normally an exporter only does work when it's being scraped (i.e. handling an incoming http request).  If that's not the case, then it's doing whatever you told it to do.  Maybe you have an infinite loop or something in your code?

If you copied their sample code:

if __name__ == '__main__':

    # Start up the server to expose the metrics.

    start_http_server(8000)

    # Generate some requests.

    while True:

        process_request(random.random())

then clearly you'll be using a whole CPU core as this loop spins as fast as it can.  This is not meant to be how a real exporter works.  Rather, when your application does some other work (e.g. processing an incoming HTTP request) it can also increment counters or whatever.

 

query2:

When I am using this lib , I can still see some unwanted metrics are populating other than mu custom metrics. Do you have any idea like jow I can remove it from my metrics list?

Can you show the metrics?

[Deeraj V]

changes(node_boot_time_seconds{job="kubernetes-service-endpoints"} [1h])

Above query giving wrong alert on reboot, can you provide me the better query

[Brian Candler]

1. That question doesn't seem to be related to this thread.

2. You've said this gives a "wrong alert", without showing the alert, saying what's "wrong" with it, or how you want it to behave instead.

Personally I use a rule like this:

      - alert: rebootNode
        expr: node_boot_time_seconds > (node_boot_time_seconds offset 5m + 5)
        labels:
          severity: warning
        annotations:
          summary: 'Device rebooted at {{ $value | humanizeTimestamp }}'