Histogram in Prometheus for existing buckets of data in JSON file

[charle...@gmail.com]

I have an application which would produce metrics in JSON file frequently. And, I want to consume the data into Prometheus.

Currently, I am struggling with the histogram in prometheus.

The snippet from the JSON file:

"timings" : {
 "request_start_to_finish" : {
 "0.001s" : 262,
 "0.002s" : 186,
 "0.003s" : 268,
 "0.004s" : 193,
 "0.005s" : 265,
 "0.006s" : 297,
 "0.007s" : 189,
 "0.008s" : 267,
 "0.009s" : 261,
 "0.01s" : 316
 }
}

Using Python, I would extract the data

 bucketList = [0,0,0,0,0,0,0,0,0,0]

 for k, v in response['session_metrics'][sessionID]['sourceIds'][sourceID]['requests']['client']['timings']['request_start_to_finish'].items():
 if any(re.findall('0.001', k, re.IGNORECASE)): bucketList[0] = v
 if any(re.findall('0.002', k, re.IGNORECASE)): bucketList[1] = v
 if any(re.findall('0.003', k, re.IGNORECASE)): bucketList[2] = v
 if any(re.findall('0.004', k, re.IGNORECASE)): bucketList[3] = v
 if any(re.findall('0.005', k, re.IGNORECASE)): bucketList[4] = v
 if any(re.findall('0.006', k, re.IGNORECASE)): bucketList[5] = v
 if any(re.findall('0.007', k, re.IGNORECASE)): bucketList[6] = v
 if any(re.findall('0.008', k, re.IGNORECASE)): bucketList[7] = v
 if any(re.findall('0.009', k, re.IGNORECASE)): bucketList[8] = v
 if any(re.findall('0.01', k, re.IGNORECASE)): bucketList[9] = v

 bucketList[1] = bucketList[1] + bucketList[0]
 bucketList[2] = bucketList[2] + bucketList[1]
 bucketList[3] = bucketList[3] + bucketList[2]
 bucketList[4] = bucketList[4] + bucketList[3]
 bucketList[5] = bucketList[5] + bucketList[4]
 bucketList[6] = bucketList[6] + bucketList[5]
 bucketList[7] = bucketList[7] + bucketList[6]
 bucketList[8] = bucketList[8] + bucketList[7]
 bucketList[9] = bucketList[9] + bucketList[8]
 
 metric.add_sample('bms_client_req_duration_milliseconds_bucket', value=bucketList[0], labels={ 'le': '0.001'})
 metric.add_sample('bms_client_req_duration_milliseconds_bucket', value=bucketList[1], labels={ 'le': '0.002'})
 metric.add_sample('bms_client_req_duration_milliseconds_bucket', value=bucketList[2], labels={ 'le': '0.003'})
 metric.add_sample('bms_client_req_duration_milliseconds_bucket', value=bucketList[3], labels={ 'le': '0.004'})
 metric.add_sample('bms_client_req_duration_milliseconds_bucket', value=bucketList[4], labels={ 'le': '0.005'})
 metric.add_sample('bms_client_req_duration_milliseconds_bucket', value=bucketList[5], labels={ 'le': '0.006'})
 metric.add_sample('bms_client_req_duration_milliseconds_bucket', value=bucketList[6], labels={ 'le': '0.007'})
 metric.add_sample('bms_client_req_duration_milliseconds_bucket', value=bucketList[7], labels={ 'le': '0.008'})
 metric.add_sample('bms_client_req_duration_milliseconds_bucket', value=bucketList[8], labels={ 'le': '0.009'})
 metric.add_sample('bms_client_req_duration_milliseconds_bucket', value=bucketList[9], labels={ 'le': '0.01'})
 metric.add_sample('bms_client_req_duration_milliseconds_bucket', value=bucketList[9], labels={ 'le': '+Inf'})
 metric.add_sample('bms_client_req_duration_milliseconds_count', value=bucketList[9], labels={})

In the /metrics endpoint, I would get this.

# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="2",minor="7",patchlevel="14",version="2.7.14"} 1.0
# HELP bms_client_req_duration_milliseconds request duration histogram
# TYPE bms_client_req_duration_milliseconds histogram
bms_client_req_duration_milliseconds_bucket{le="0.001"} 346.0
bms_client_req_duration_milliseconds_bucket{le="0.002"} 445.0
bms_client_req_duration_milliseconds_bucket{le="0.003"} 732.0
bms_client_req_duration_milliseconds_bucket{le="0.004"} 1153.0
bms_client_req_duration_milliseconds_bucket{le="0.005"} 1430.0
bms_client_req_duration_milliseconds_bucket{le="0.006"} 1725.0
bms_client_req_duration_milliseconds_bucket{le="0.007"} 1873.0
bms_client_req_duration_milliseconds_bucket{le="0.008"} 1996.0
bms_client_req_duration_milliseconds_bucket{le="0.009"} 2291.0
bms_client_req_duration_milliseconds_bucket{le="0.01"} 2788.0
bms_client_req_duration_milliseconds_bucket{le="+Inf"} 2788.0
bms_client_req_duration_milliseconds_count 2788.0

However, when trying to following query in Prometheus, I would get the following NaN.

histogram_quantile(0.5, sum(rate(bms_client_req_duration_milliseconds_bucket [5m])) by (le))

Prometheus Details:

Version	2.3.1
Revision	188ca45bd85ce843071e768d855722a9d9dabe03
Branch	HEAD
BuildUser	root@82ef94f1b8f7
BuildDate	20180619-16:06:30
GoVersion	go1.10.3

If there is any other way to actually scrape / consume such data in Prometheus, please advise. Thank you.

[Brian Brazil]

That all looks correct. Are the values from the code increasing over time?

--

Brian Brazil

www.robustperception.io

[charle...@gmail.com]

Yes, the values would be increasing over time.

[Brian Brazil]

Hmm, what is the sum(rate(...)) returning?

Brian

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d6a18f08-6691-4e24-be3e-5e28ab6698ed%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io