Application aware quota

[Barak Mordehai]

I am excited to introduce a new repository to the community that addresses an important aspect of managing virtual resources in a multi-tenant environment. This repository presents a new operator which reimplements the standard resourceQuota of Kubernetes.

Motivation:
During the migration process, there is a hidden cost involved in creating a new virt-launcher pod before terminating the old one. This differs from how standard pods are moved and can result in unexpected resource utilization from the user's perspective. This poses a challenge where the hidden cost of live migration can prevent live migrating a VM for reasons not reflected to the VM creator / namespace owner

When upgrading KubeVirt, it is necessary to migrate VMIs to upgrade the virt-launcher pod. However, migrating a virtual machine in the presence of a resource quota can cause the migration to fail and subsequently cause the upgrade to fail.

background:
This repository represent a new implementation to Resource Quota with additional logic which recognize the virtual resources required when running a VM and hide the additional resources needed for migration as this is an implementation detail and users shouldn't be charged for the additional resources required for infrastructure
How to use it:
1) fork: https://github.com/kubevirt/applications-aware-quota
2) run `make cluster-up && make cluster-sync`

An example for the new resourceQuota could be found in:
https://github.com/kubevirt/applications-aware-quota/blob/main/manifests/examples/arq-example.yaml

Currently we have 2 useful configurations:

DedicatedVirtualResources -  Calculate vmi.requests.memory as the vmi's ram size and vmi.requests.cpu as number of threads of vmi n this configuration no memory.request/limits and cpu.request/limits won't be included 

VmiPodUsage (The default configuration) - Calculate usage of launcher like any other pod 

Important: Both Configurations hide migration additional resources

To change Configuration:
`kubectl edit aaq` 

set:
```

spec:
  configuration:
    vmiCalculatorConfiguration:
      configName: DedicatedVirtualResources

```

Thanks Barak.

[Barak Mordehai]

Forgot to mention that before deploying kubevirtci you should set:
export KUBEVIRT_PROVIDER=k8s-1.27
As this operator utilizes the scheduling gates feature which is enabled by default since 1.27.

Another thing worth mentioning is that the final goal is to make this operator generic so more apps could use it
(ArgoCD,Isto...)