Prometheus automated bench-marking project

[HARSH AGARWAL]

Hi everyone,

This topic is to tell about the updates to Prometheus automated benchmarking tool and get feedback from the community.

Prombench has been integrated with prow. This means any maintainer can trigger benchmark by typing /benchmark pr or /benchmark release 2.3.0 in a PR conversation.

This action scales up a GKE cluster by adding new NodePools and adds prombench components in a new namespace(prombench-<PR_NUMBER>).

A detailed explanation of the design can be found here.

Grafana is also automatically deployed, making it easier for maintainers to compare metrics of two releases.

The dashboard used for Grafana includes brian's 2.3.x prombench-dashboard and some node metrics.

A working demo can be found at the following links:

GitHub PR showing trigger commands for prombench:: https://github.com/sipian/prometheus/pull/12

Prometheus-meta (Prometheus server monitoring the Prometheus servers being benchmarked):: http://35.229.83.49:9090/targets

Grafana:: http://35.229.121.161:3000 (Username: admin, Password: admin)

The code for the project:

Prombench:: https://github.com/sipian/prombench/tree/prombench
Benchmark plugin for prow:: https://github.com/sipian/test-infra/tree/prometheus-prow/prow/plugins/benchmark
 

The next features on my TODO list are:

1. Replace exposed service from LoadBalancer to Ingress
   
2. Add alerting rules and post alerts as GitHub comments on being triggered
   
3. Post a GitHub comment if the maintainer forgets to comment /benchmark delete
   
4. Enable GitHub login in grafana & add grafana-kubernetes-app

Feedback & suggestions are welcome.

Thanks,
Harsh.

[Ben Kochie]

That's fantastic!

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/00ad6d5c-fd35-40c2-b983-0b1bde4212c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[HARSH AGARWAL]

Just a heads-up.
Since, the project is under active development, the links to the demo deployments might not be active all the time.

[HARSH AGARWAL]

Quick update about this tool

• Grafana and Prometheus-meta (Prometheus server monitoring the Prometheus servers being benchmarked) are exposed on nginx-ingress reverse-proxy
  
• One instance of Prometheus-meta monitors all instances of prombench and one instance of Grafana graphs the metrics(can switch between instances via grafana template variables).
• Grafana allows anonymous access to its dashboards.
  
• The prombench cluster can be deployed using the gke tool developed (https://github.com/sipian/prombench/tree/automate-prombench-gke)
  

Thanks,
Harsh.