histogram buckets not updating

[Chris Card]

Hi all,

I've successfully used the new histogram functionality in mtail for a couple of cases, but I'm trying to implement a histogram for another case and for some reason the histogram buckets are not updating.

As far as I can see I've got the syntax correct, and _sum and _count metrics associated with the histogram *are* updating, but the buckets themselves are stuck at zero. Even if the values being added were all zero,

I'd expect to see something in the first bucket, and in any case both _count and _sum are > 0.

Any ideas about how to debug this?

UPDATE: 

while typing this question, I did get non-zero in the buckets:

XXXX_processing_time_seconds_bucket{prog="XXXX.mtail",le="2"} 1

XXXX_processing_time_seconds_bucket{prog="XXXX.mtail",le="5"} 1

XXXX_processing_time_seconds_bucket{prog="XXXX.mtail",le="10"} 1

XXXX_processing_time_seconds_bucket{prog="XXXX.mtail",le="20"} 1

XXXX_processing_time_seconds_bucket{prog="XXXX.mtail",le="40"} 1

XXXX_processing_time_seconds_bucket{prog="XXXX.mtail",le="80"} 1

XXXX_processing_time_seconds_bucket{prog="XXXX.mtail",le="160"} 1

XXXX_processing_time_seconds_bucket{prog="XXXX.mtail",le="+Inf"} 1

XXXX_processing_time_seconds_sum{prog="XXXX.mtail"} 7

XXXX_processing_time_seconds_count{prog="XXXX.mtail"} 6

But if the _count is 6 and the _sum is 7, why don't I see more in the buckets?

This is effectively what the mtail program looks like:

hidden gauge start_processing_time by pid

histogram XXXX_processing_time_seconds buckets 1,2,5,10,20,40,80,160             

/^(?P<date>\w+\s+\d+\s+\d+:\d+:\d+)  / {

    strptime($date, "Jan _2 15:04:05")

    /YYYY\[(?P<pid>\d+)\]: / {

   

        /Starting/ {

            start_processing_time[$pid] = timestamp()

        }

   

        /Finished/ {

            start_processing_time[$pid] > 0 {

                timestamp() > start_processing_time[$pid] {

                    XXXX_processing_time_seconds = timestamp() - start_processing_time[$pid]

                }

                del start_processing_time[$pid]

            }

        }

    }

}

Chris

[Michael Hobbs]

Is it possible it's due to this bug:
https://github.com/google/mtail/issues/226

It's fixed but not yet released (I think). Can you try adding a bucket of 0 at the start of your range and see if that gets you the results you expect?

Thanks,

Mike

--
You received this message because you are subscribed to the Google Groups "mtail-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mtail-users...@googlegroups.com.
To post to this group, send email to mtail...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mtail-users/061b08fc-1b89-4c9d-b8e7-90ce3f65e44c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Card]

Thanks Mike, I'll try that

[Chris Card]

Adding a 0 bucket seems to do the trick, thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to mtail...@googlegroups.com.

[Jamie Wilkinson]

v3.0.0-rc32 will have the fix in it.

To unsubscribe from this group and stop receiving emails from it, send an email to mtail-users...@googlegroups.com.

To post to this group, send email to mtail...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mtail-users/72981d09-91db-47f9-bea2-1229cb50c005%40googlegroups.com.