Subtracting two Timestamp in single logs line

[Samya Maiti]

Hi Team,

I have started working on a project that involves feeding data from REST logs as below to a monitoring stsrem like prometheus.

<Unique Rest RequestID>         <Client Name>         <API Name>         <Actual Request url>         <request serving start time: Mon Nov 25 23:51:04 PST 2019>       <request serving end time: Mon Nov 25 23:51:14 PST 2019>         <status of request :success or error>

I was suggested to use mtail for the use-case.

The questions am trying to answer are:-

1. The number of total requests served irrespective of any condition.
2. The number of total requests served filtered on <Client Name> or  <API Name> or <status of request>
   
3. Time is taken to serve req on avg/quantile irrespective of any condition.
   
4. Time taken to serve req on avg/quantile filtered on <Client Name> or  <API Name> or <status of request> 

Is there a way I can achieve the same in mtail? To be specific how do I do Subtracting of two timestamps <request serving start time> and <request serving end time> in mtail.

Regards,

Samya

[Jamie Wilkinson]

You'll need mtail to count the events, and something else (like http://prometheus.io) to do the aggregations.

As both timestamps are in the same log line, you can get your subtraction done pretty easily/

Once you have constructed the match pattern, you can answer the first two questions by incrementing a counter broken down by client, api, and status.  You can record the latency of the requests by client, api, and status in a histogram.

Then you';ll need to have Prometheus or something like it scrape these metrics and do the aggregations.

If I can assume some things about your log lines, then a program like:

`

counter requests by client, api, status

histogram latency by client, api, status buckets 1, 2, 4, 8

/.* (<?Pclient>\w+) (<?Papi>\w+) (<?Purl>.+) (<?start>\w{3} \w{3} +\d?\d \d{2}:\d{2}:\d{2} \w+ \d+) (<?Pend>\w{3} \w{3} +\d?\d \d{2}:\d{2}:\d{2} \w+ \d+) (<?Pstatus>success|error)/ {

    requests[$client][$api][$status]++

    latency[$client][$api][$status] = $end - $start

}

`

You can read more about the histograms here: https://github.com/google/mtail/blob/master/docs/Programming-Guide.md#histograms

I expect you'll have to tune the regular expression to match your actual log line syntax.

Good luck!

--
You received this message because you are subscribed to the Google Groups "mtail-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mtail-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mtail-users/d1d74c12-fc9f-424a-83b8-4ad1fcb2b85c%40googlegroups.com.