[SIG-scale] Community meeting 4/29

[Ryan Hallisey US]

Unfortunately, there is not a recording for the first meeting.  In the future, we'll record the meetings and post them to youtube.

Notes:
Attendees: rhallisey, rmohr, nmimurag, kwiesmueller, jhopper, gmaltby, rbeisner, fanzhang

KubeVirt maintains its own api-server, scheduler, and workloads independent of Kuberentes, so it should have its own scale and performance standards, tools, and best practices.

Scope

Define and drive scalability and performance goals within KubeVirt.  This includes documenting, testing, and measuring performance and scalability for a KubeVirt release.

Goals

• Analyze KubeVirt component scalability

• Learn how KubeVirt can more efficiently use Kubernetes

• Document, test, and maintain the performance and scalability standards of KubeVirt components and Virtual Machines across KubeVirt releases

• Raise awareness to work focusing on performance and scale

• Meet bi-monthy to discuss new solutions, patterns, and progress

Topics

• Improve virt-launcher create-to-ready time to be as close as possible to pod create-to-ready time

• Achieve equal create-to-ready time between the virt-launcher pod and VMI
  

• Write-up some best practices

• Build an expected staRrt time matrix based on performance classes/types

• Maintain performance under high pod counts and high VMI counts

• Create a scale test framework

  • kubemark - https://pkg.go.dev/k8s.io/kubernetes/pkg/kubemark

• Performance reports in CI i.e. this change provides +- performance

• Create a API for large scale creation/deletion

• Virtual Machine Pools (design doc)

Meetings:
4/29/21

Agenda:

• Intro to SIG-scale

  • Where are we now?

    • How many nodes?

      • 280 (2 virt-api, 2 virt-controllers)

    • How many VMs?

      • ~1000

    • Create-to-ready throughput for 500 VMs

      • ~50 VMs / minute 

  • Where can we go?

    • 1000 nodes?

      • API responsiveness: 99% of all API calls return in less than 1s 

      • VM startup time: 99% of VMs (with pre-pulled images) start within 60s.

      • Pod startup time: 99% of VMs (with pre-pulled images) start within 25s.

        • The Kubernetes Pod startup time metric is 5s, but KubeVirt’s pods have VMs!

    • 200 VMs / minute throughput?

  • What can we build or change?

• Areas people are already working on or interested in 

  • [rhallisey] - Virtual Machine Pools

  • [nmimurag] [+1 kwiesmueller] - Control plane continuous performance and scalability evaluation (reviews and comments are welcome)

  • [rmohr] [+1 kwiesmueller] interested in API calls on scale

    • How many k8s calls we make when we create a VM

  • [rhallisey] - test kubevirt with kubemark