extract the data from entire log file

[dc3o]

I'd like to have the stats not only by tailing the logs but also to get the history metrics? mtail have a oneshot option - but that does not look like a proper way to reconstruct the stats. grok parser does have this option but I'd like to use a single tool here. Any advice?   

[Jamie Wilkinson]

I'm not sure what you mean by history metrics, but I'll assume you mean you want a timeseries: a lot of values for each metric each associated with a timestamp.

mtail doesn't do that, that's what other tools are for.  See https://google.github.io/mtail/Interoperability.html

Prometheus is a common example for collecting the metrics out of mtail at regular intervals and then being able to display them on a graph or inspect them and do other statistical analysis on them.

On Mon, 24 May 2021 at 18:01, dc3o <deln...@gmail.com> wrote:

I'd like to have the stats not only by tailing the logs but also to get the history metrics? mtail have a oneshot option - but that does not look like a proper way to reconstruct the stats. grok parser does have this option but I'd like to use a single tool here. Any advice?   

--
You received this message because you are subscribed to the Google Groups "mtail-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mtail-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mtail-users/ba1236e3-372d-4a38-b3be-d5c03e53f8e1n%40googlegroups.com.

[Nemanja Delic]

I'm looking if mtail is able to export metrics from the entire log file. I would use this for backfilling data to prometheus using promtool than.

[Jamie Wilkinson]

This sounds like a cool idea, but mtail doesn't support it at the moment.  

The two main issues, and several sub issues I can think of are:

* How does one expect to run mtail in this mode

  * is it a one-off command run by the operator on specific logs, or should mtail always reprocess logs that match on startup

  * will Prometheus know to discard duplicate data points if you reprocess a file that was partially read already, or does mtail need to manage that state

  * how does mtail know if a log file has been read already, and at which byte offset has it started at

* How should mtail sample the data points in order to construct a timeseries, because right now oneshot doesn't report any counters until the end -- one assumes that mtail must export every 10s or whatever to make this work.

Would you mind filing a feature request at http://github.com/google/mtail/issues with a summary of how you would like to use mtail to backfill data?  If you can speculate on command lines and expected behaviour that would be really helpful as well.