Daily summary metric - bad fit?

[mik...@gmail.com]

I have a co-worker that is looking to import a daily summary count of volumes, once per day from a batch script.

Is my understanding correct, in that:

- You can use the push gateway without timestamped data, however this will result in continuous sampling of the push gateway with lots of identical samples. If the scraping of the push gateway happens every minute, thats essentially 60x24 identical (useless) samples.

- You can use the push gateway with timestamps, once per day. However, the metric will disappear if there have not been any further updates for 5 minutes?

I am a bit confused by the section of the push gateway docs:

"if it cannot get any samples for a metric in 5min, it will behave as if that metric does not exist anymore. Preventing that is actually one of the reasons to use a push gateway. The push gateway will make the metrics of your ephemeral job scrapable at any time. Attaching the time of pushing as a timestamp would defeat that purpose because 5min after the last push, your metric will look as stale to Prometheus as if it could not be scraped at all anymore. (Prometheus knows only one timestamp per sample, there is no way to distinguish a 'time of pushing' and a 'time of scraping'.)

You can still force Prometheus to attach a different timestamp by using the optional timestamp field in the exchange format. However, there are very few use cases where that would make sense. (Essentially, if you push more often than every 5min, you could attach the time of pushing as a timestamp.)"

Can someone explain what it means by "it will behave as if that metric does not exist anymore"? Surely prometheus doesn't delete all the timeseries data if you have a 5 minute outage?

If Prometheus is a bad fit that's fine too - just trying to figure it all out.

Thanks
Mike

[Ben Kochie]

Yes, this is definitely a bad fit for Prometheus.

What happens with the 5 min window thing is just visibility, it does not delete the data.

In order to see the data, you need to make a query that has a time range with samples within 5min.

Maybe back to the actual data problem.  Can you give us more info about what the bash script is doing?  What is the raw underlying data.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Kochie]

One other side note about those useless samples, with varbit encoding, it would be under 2kb of data to store.  Maybe useless, but Prometheus compression is very good at this kind of thing.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

[mik...@gmail.com]

The underlying data is a set of summary metrics output from Spark/MapReduce after performing analytics about a days worth of customer interactions hitting a website.

From my coworker:
"We would be interacting with the time series on a scale of days rather than seconds as we process our analytics data once a day in a single batch per journey (using Spark or Map Reduce) and we would want to push out our count metrics at the end of each daily batch.

At the moment we want to look at outputting a single count metric once a day for our analytics pipeline but eventually we could have 30 count metrics being pushed over once a day. We have no plans to use smart monitoring for real time events as we do not generate this type of data.

For example, our pipeline would be pushing a count of the daily volumes at the end of the transformation pipeline run. The transformation pipeline would run once every morning. This will allow us to check that the morning pipelines are behaving consistently.
"

Also could you expand on what you mean about the visibility problem of a 5 minute window?

If we were to use prometheus in the interim before we set up a better system, would using a push gateway with timestamp be the most sensible? It sounds like it wouldnt lose any data, and each once-per-day push would be captured once by prometheus?

Mike

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

[Ben Kochie]

Yes, this is definitely not something Prometheus would be useful for.

 Normally I wouldn't recommend this, but since this is highly important data, with a _very_ low update rate, a simple SQL schema would be a good idea.  Tying that into something like Grafana would require a bit of work, basically you would want to implement the "simple json"[0] interface as a web service in front of mysql.

However, Prometheus would be great for monitoring the progress of the Spark/MapReduce jobs themselves. :-)

[0]: https://github.com/grafana/simple-json-datasource

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

>
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

[al...@everymundo.com]

Hello,
Did you guys find a better solution to monitor this type of metric? I'm looking at monitoring data that's generated once per day but with many tags. I'm looking for software that can monitor these data samples with respect to all the possible combinations of the associated tags... something like AI for detecting anomalies.

Looks like Argus may be a good choice... but there's not too much documentation on it out there. https://github.com/salesforce/Argus

> > To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

>
> >
>
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.