metrics are not updated

[Eden Borochov]

Hi guys, please help

I am using micrometer + Prometheus push gateway to publish metrics from AWS lambda to Pushgateway. The problem is that metrics aren't updated as soon as my params are changed. It takes a while for changes to be reflected in scraped metrics.

Is were any cache or scraping interval that prevents from Prometheus updating the collector with new values? What configuration should be set to see the latest value that was registered to the metric. 

Thanks

Eden

[Stuart Clark]

When you say "a while" how long are you meaning? And when you say aren't
updated, do you mean running a query in Prometheus doesn't reflect any
metrics published by your application? What is the scrape interval for
the job that scrapes Pushgateway?

--
Stuart Clark

[Stuart Clark]

On 06/01/2022 14:46, Eden Borochov wrote:
> A while can take a few minutes,
> I am testing the metrics on the push gateway server after running lambda.
> I expect that metrics should be immediately updated in the push
> gateway as there is a gauge registry in lambda.
>  What do you mean by scrape interval,
>  the one that I found for Pushgateway ->
> management_metrics_export_prometheus_pushgateway_pushRate, and this
> one is defined to 1ms

Prometheus works by scraping endpoints for metrics. Ideally this is
scraping the applications themselves or exporters, but in your case this
is the Pushgateway.

So you need to look at your prometheus.yaml configuration (also visible
via the Prometheus UI) to see what it is set to.

I would never expect things to be "instant" - you should expect it to
take 1 or 2 scrape intervals for things to be visible. You'd normally
have the scape interval set to between about 15 seconds and say a minute.

--
Stuart Clark

[Sanjib Bisoyi]

how to read custom metric in node exporter  using POMQL or postgresql,

like  app_health 1 ,centrify_health 1. etc

# HELP app_health Metric read from /var/tmp/node_exporter/pscc.prom
# TYPE app_health untyped
app_health 1
# HELP centrify_health Metric read from /var/tmp/node_exporter/pscc.prom
# TYPE centrify_health untyped
centrify_health 1
# HELP cnr_health Metric read from /var/tmp/node_exporter/pscc.prom
# TYPE cnr_health untyped
cnr_health 0

dlp_demux_k 1
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.5273e-05
go_gc_duration_seconds{quantile="0.25"} 3.8132e-05
go_gc_duration_seconds{quantile="0.5"} 7.218e-05
go_gc_duration_seconds{quantile="0.75"} 0.00013091
go_gc_duration_seconds{quantile="1"} 0.001889392
go_gc_duration_seconds_sum 63.219977222
go_gc_duration_seconds_count 502793
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 9
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.14.4"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.510072e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.088473453064e+12
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 2.029967e+06
# HELP go_memstats_frees_total Total number of frees. 

# TYPE go_memstats_frees_total counter 

[Sanjib Bisoyi]

Hi All,

Please help me with the below question. waiting for it.

API call also fine 

Thanks,

Sanjib

[Brian Candler]

Firstly, please don't hijack other people's threads - it's rude.  For a new question, start a new thread.

Secondly, your question as written makes no sense. Presumably by "POMQL" you mean PromQL. I've no idea why you think postgresql is relevant, since prometheus doesn't make use of postgresql.

The way you read (i.e. query) custom metrics is the same way you query other metrics: with a PromQL query.  There is no difference.  PromQL can't tell whether a metric came from a node_exporter textfile collector, or from somewhere else.

If you want help getting started with PromQL queries, there are plenty of resources on the web.  Here are a few you can try:

* <https://prometheus.io/docs/prometheus/latest/querying/basics/>
* <https://github.com/infinityworks/prometheus-example-queries>
* <https://timber.io/blog/promql-for-humans/>
* <https://www.weave.works/blog/promql-queries-for-the-rest-of-us/>
* <https://www.slideshare.net/weaveworks/promql-deep-dive-the-prometheus-query-language>
* <https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085>
* <https://www.robustperception.io/common-query-patterns-in-promql>

If that doesn't answer your question, then you'll need to ask your question in a clearer way.  What exactly are you trying to do?  What have you tried so far?  What did you expect to happen, and what did you see happen instead?

[Sanjib Bisoyi]

Thank you, I am new to Prometheus and I will go through the link. If required any help I will create a new thread for this.

My question was we are using custom scripts in node exporter and it gives us the data like below marked as bold(app_health 1,centrify_health 1).

if app_health 1 means its green and 0 means it's red(unhealthy).

 so I need to use PromQL to query it.

# TYPE app_health untyped
app_health 1
# HELP centrify_health Metric read from /var/tmp/node_exporter/pscc.prom
# TYPE centrify_health untyped
centrify_health 1
# HELP cnr_health Metric read from /var/tmp/node_exporter/pscc.prom
# TYPE cnr_health untyped
cnr_health 0

Thanks,

Sanjib 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2b6586e8-c44a-405e-bc59-fa1714a93675n%40googlegroups.com.

[Brian Candler]

The simplest PromQL query is just the name of a metric, like this:

    app_health

That's it - a complete, valid PromQL query.  It will return the current value of the app_health metric, at a particular instant in time - by default, the current time.  In fact, this will be a vector of zero or more values, because there may be multiple timeseries with the same metric name, distinguished by their labels, e.g.

    app_health{job="node", instance="foo"} 0

    app_health{job="node", instance="bar"} 1

    app_health(job="node", instance="baz"} 1

Hence this is known as an "instant vector" query.