Graph of when a series is NULL

[John Dexter]

I know in Grafana I can choose to display NULL as zero or as missing but I actually want to graph when a value is null - to visualise downtime where a series has gaps. Can I craft a query which will spit out two values for NULL / not NULL?

Thanks.

[Brian Candler]

Prometheus does not store or return "null".

Prometheus will store a "staleness marker" when it performs a scrape and doesn't find a timeseries (i.e. metric+label set) which was present in the previous scrape.  Internally, it's stored as a special kind of floating-point NaN.  However it's not returned in queries.  Instead, the timeseries just "disappears" as if it didn't exist.

A timeseries consists of a series of values V1, V2, V3 .. at timestamps T1, T2, T3 ...  So when you perform an instant query for some arbitrary time T, then prometheus has to look back in time to find the nearest data point at or before time T.  It will look back up to --query.lookback-delta (by default 5m), but if it finds a staleness marker, it just doesn't return any value, i.e. that timeseries is excluded from the vector result.  This is all described here.

 

If you want to look gaps in a particular timeseries, you can build a query using absent(), giving a specific timeseries (metric+labels) that you're looking for, and prometheus will tell you when the time series doesn't exist (i.e. there is no data point within the previous 5 minutes *or* the series has been marked stale)

    absent{foo{instance="bar",job="baz"})

If you want to see when arbitrary timeseries disappear, without hardcoding specific label sets, you can try something like this:

    foo offset 5m unless foo   # it existed 5 minutes ago, but doesn't exist now

    present_over_time(foo[24h] offset 5m) unless foo    # existed any time in the last 24 hours until 5 minutes ago, but doesn't exist now

[John Dexter]

Thanks for the great explanation Brian.

absent() looks like the right solution but the documentation about empty vectors confuses me slightly. What I would ideally like is to graph the absent() as two distinct values - is there a way to return that as a query or do I have to address this in my graphing tool (Grafana)?

[Brian Candler]

These posts may be helpful:

https://www.robustperception.io/absent-alerting-for-scraped-metrics

https://www.robustperception.io/absent-alerting-for-jobs

https://www.robustperception.io/existential-issues-with-metrics

The expression

    absent(mymetric{foo="bar",baz="qux"})

"returns an empty vector if the vector passed to it has any elements and a 1-element vector with the value 1 if the vector passed to it has no elements"  [ref]

So you should be able to build something like this:

    (mymetric{foo="bar",baz="qux"} * 0) or absent(mymetric{foo="bar",baz="qux"})

That "or" expression [ref] should evaluate to either

{foo="bar",baz="qux"} 0

{foo="bar",baz="qux"} 1

But it really depends on the situation.  In some cases, the metric "up" is all you need.  It's 1 when a scrape was successful, and 0 when the scrape failed.  Even if the scrapes are successful, "up" is a useful metric to combine with absent(), as shown in the blog posts linked above.