[SIG-scale] Meeting Notes 2024-06-27 & 2024-06-06

[Andrew Burden]

6/27/24

Recording: SIG - Performance and scale 2024-06-27 

Attendees: rhallisey, alaypatel07

Agenda:

• Density test is working again

• https://kubevirt.io/ci-performance-benchmarks/weekly/periodic-kubevirt-e2e-k8s-1.29-sig-performance/vm/index.html

• Metrics test fixes

• https://github.com/kubevirt/kubevirt/pull/12096 

• Ryan Hallisey US & Alay Patel US to review

• Make a 1.3 release blocker

• How can we catch changes that break the performance metrics

• Unit tests? Func tests?

• Test metric range, which is triggered only on changes to the path: pkg/monitoring/metrics/common/client/metrics.go

• 1.3

• Create 1.3 performance benchmarks

• Kwok integration

• Memory cpu utilization for control plane https://github.com/kubevirt/kubevirt/pull/12116 

• PR to add tests: https://github.com/kubevirt/kubevirt/pull/12117 

• Will need an issue for CI configuration for a job to run kwok test

• Metrics capture for virt-handlers

• Find which handlers are outliers

• Find an average

• Metric: Average mem usage of handlers

== Apologies, looks like this was missed when it was posted ==

6/6/24

Recording: https://youtu.be/k6tLmNdgFdU 

Attendees: rhallisey, alaypatel07

Agenda:

• Job is failing

• Check with Brian

• Install kwok when running `make cluster up` https://github.com/kubevirt/kubevirtci/pull/1178

• Add a density test to the periodic job

• Need to track virt-controller & virt-api resource usage

• Scale experiment with 5.4k VMI

• Virt-controller memory increases to 3.9 GB

• Profile virt-controller

• Transition PhaseTransition List to a watch

• https://kubevirt.io/ci-performance-benchmarks/weekly/periodic-kubevirt-e2e-k8s-1.29-sig-performance/vm/index.html

• Losing more metrics after a PR around May 19, 2024

• https://github.com/kubevirt/kubevirt/pull/11601

Also from April, https://github.com/kubevirt/kubevirt/pull/11184