histogram buckets not updating

14 views
Skip to first unread message

Chris Card

unread,
Jul 1, 2019, 7:45:24 AM7/1/19
to mtail-users
Hi all,

I've successfully used the new histogram functionality in mtail for a couple of cases, but I'm trying to implement a histogram for another case and for some reason the histogram buckets are not updating.
As far as I can see I've got the syntax correct, and _sum and _count metrics associated with the histogram *are* updating, but the buckets themselves are stuck at zero. Even if the values being added were all zero,
I'd expect to see something in the first bucket, and in any case both _count and _sum are > 0.
Any ideas about how to debug this?

UPDATE: 
while typing this question, I did get non-zero in the buckets:

XXXX_processing_time_seconds_bucket{prog="XXXX.mtail",le="2"} 1
XXXX_processing_time_seconds_bucket{prog="XXXX.mtail",le="5"} 1
XXXX_processing_time_seconds_bucket{prog="XXXX.mtail",le="10"} 1
XXXX_processing_time_seconds_bucket{prog="XXXX.mtail",le="20"} 1
XXXX_processing_time_seconds_bucket{prog="XXXX.mtail",le="40"} 1
XXXX_processing_time_seconds_bucket{prog="XXXX.mtail",le="80"} 1
XXXX_processing_time_seconds_bucket{prog="XXXX.mtail",le="160"} 1
XXXX_processing_time_seconds_bucket{prog="XXXX.mtail",le="+Inf"} 1
XXXX_processing_time_seconds_sum{prog="XXXX.mtail"} 7
XXXX_processing_time_seconds_count{prog="XXXX.mtail"} 6

But if the _count is 6 and the _sum is 7, why don't I see more in the buckets?

This is effectively what the mtail program looks like:

hidden gauge start_processing_time by pid
histogram XXXX_processing_time_seconds buckets 1,2,5,10,20,40,80,160             

/^(?P<date>\w+\s+\d+\s+\d+:\d+:\d+)  / {
    strptime($date, "Jan _2 15:04:05")

    /YYYY\[(?P<pid>\d+)\]: / {
   
        /Starting/ {
            start_processing_time[$pid] = timestamp()
        }
   
        /Finished/ {
            start_processing_time[$pid] > 0 {
                timestamp() > start_processing_time[$pid] {
                    XXXX_processing_time_seconds = timestamp() - start_processing_time[$pid]
                }
                del start_processing_time[$pid]
            }
        }
    }
}

Chris

Michael Hobbs

unread,
Jul 1, 2019, 10:03:03 AM7/1/19
to Chris Card, mtail-users
Is it possible it's due to this bug:
https://github.com/google/mtail/issues/226

It's fixed but not yet released (I think). Can you try adding a bucket of 0 at the start of your range and see if that gets you the results you expect?

Thanks,
Mike

--
You received this message because you are subscribed to the Google Groups "mtail-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mtail-users...@googlegroups.com.
To post to this group, send email to mtail...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mtail-users/061b08fc-1b89-4c9d-b8e7-90ce3f65e44c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Chris Card

unread,
Jul 1, 2019, 11:57:11 AM7/1/19
to mtail-users
Thanks Mike, I'll try that

Chris Card

unread,
Jul 2, 2019, 3:58:20 AM7/2/19
to mtail-users
Adding a 0 bucket seems to do the trick, thanks!
To unsubscribe from this group and stop receiving emails from it, send an email to mtail...@googlegroups.com.

Jamie Wilkinson

unread,
Jul 4, 2019, 3:58:18 AM7/4/19
to Chris Card, mtail-users
v3.0.0-rc32 will have the fix in it.

To unsubscribe from this group and stop receiving emails from it, send an email to mtail-users...@googlegroups.com.

To post to this group, send email to mtail...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages