Creating a time series from data in a log file

[Andrew Fielden]

Apologies if this is a dumb question, I'm a newcomer to Prometheus. I have a collection of log files which contain performance stats for a set of rules.

I want to parse these files, and create a time series of each individual stat, on a per rule basis, and make these available to the Prometheus server.

I'm just looking for a helpful code example of how to do this. Ideally in Golang.

[David Leadbeater]

Rather than writing your own Go code, you might be interested in 

https://github.com/google/mtail which is a tool designed for that exact purpose.

David

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/a5ae6e93-0061-43d3-ad1b-1fd704d7a706n%40googlegroups.com.

[Andrew Fielden]

Thanks for the suggestion, mtail looks useful. However the log files are only produced when the system being monitored is sent a SIGINT signal. I wonder if mtail can handle that?

[Stuart Clark]

On 2021-03-09 09:28, Andrew Fielden wrote:
> Thanks for the suggestion, mtail looks useful. However the log files
> are only produced when the system being monitored is sent a SIGINT
> signal. I wonder if mtail can handle that?
>

Is it that the logs are only produced when you send SIGINT for a
"current snapshot" of the status of the application, or is it that logs
are emitted for a period of time?

If the second, Prometheus doesn't have the ability to ingest chunks of
data over periods of time (just the "current" data). There is some
experimental work ongoing for one-time backfilling of data, but that
equally probably wouldn't be suitable for regular ingestion.

--
Stuart Clark

[Andrew Fielden]

The former i.e. logs produced as a snapshot, into a timestamped file, when the process receives SIGINT.