Problem with my setup, python exporter accumulates to many metrics.
cesa...@gmail.com
I have configure these metrics:
prometheusmetrics = {
'status': Gauge("p72_jenkins_build_by_project_result","result of each job execution",labelnames=["project","app","job","number","status"]),
'time': Gauge("p72_jenkins_build_by_project_timestamp","timestamp of each job execution",labelnames=["project","app","job","number"]),
'ed': Gauge("p72_jenkins_build_by_project_estimatedDuration","Estimated duration of each job execution",labelnames=["project","app","job","number"]),
'health': Gauge("p72_jenkins_pipeline_health","Health of application pipelines",labelnames=["project","app","job"]),
'lastbuild': Gauge("p72_jenkins_pipeline_LastBuild","Health score of application pipelines",labelnames=["project","app","job"]),
'firstbuild': Gauge("p72_jenkins_pipeline_FirstBuild","Health score of application pipelines",labelnames=["project","app","job"]),
'lastcompleted': Gauge("p72_jenkins_pipeline_LastCompletedBuild","Health score of application pipelines",labelnames=["project","app","job"]),
'lastfailed': Gauge("p72_jenkins_pipeline_LastFailedBuild","Health score of application pipelines",labelnames=["project","app","job"]),
'lastsuccess': Gauge("p72_jenkins_pipeline_LastSuccessBuild","Health score of application pipelines",labelnames=["project","app","job"])
}
and the metrics are published using the endpoint in the python prometheus client
start_http_server(8889)
The issue is that metrics start accumulating and doesn't have a TTL so everytime time prometheus scrape it collect all these old metrics plus new ones add the timestamp os the scrape and each time it does the same so I have old metrics repeated to infinity with different timestamps.
And because of these huge amount of data graphing things in grafana is becoming a dificult task for example:
count(count_over_time(p72_jenkins_build_by_project_status[5m]))by(status) - count(count_over_time(p72_jenkins_build_by_project_status[5m] offset 5m))by(status)
currently the endpoint is showing more than 90,000 lines of data.
How do you handle this? It will be great to make disapear old already collected data but I don't know how, I know there isn't a TTL.