SIG-node CI jobs using kubetest2

[Naman Lakhwani]

Hi everyone, I'm a Linux foundation fall’21 mentee and my mentorship project is to test the sig-node-related code using kubetest2. For this, we've successfully added the kubetest2 version, of most of the node-e2e pre-submit jobs and now started adding the same for periodic ones.

Bootstrap has been deprecated so we have used pod-utils for the newly added prow-jobs [test-infra/pod-utilities.md at master · kubernetes/test-infra (github.com)] for kubetest2. A detailed explanation along with an example of a CI job moving from using bootstrap to pod utils can be found here [Example of a CI job moving from using bootstrap to pod utils (github.com)]

In kubetest2, we have a noop deployer and a node tester, so a typical node test command looks like “kubetest2 noop --test=node -- <tester-args>”. All the tester args for node, in kubetest2 could be found here → kubetest2/node.go at master · kubernetes-sigs/kubetest2 (github.com)

Below are some points which we kept in mind while configuring jobs for kubetest2.

1. Experimental image have the kubetest2 binaries, therefore, suffix to spec.containers.image is -experimental and not -master

2. Kubernetes repo is present at the same location therefore, “--repo-root=. ”

3. No change in “--gcp-zone” & “--gcp-project”

4. “--test-args” of kubetest2 is equivalent to “--node-test-args” of non-kubetest2.

5. This arg for non-kubetest2: 

“--test_args=--nodes=8 --focus="\[NodeConformance\]" --skip="\[Flaky\]|\[Slow\]|\[Serial\]"” 

is now supplied via three different flags of the kubetest2 node tester. i.e. 

- --parallelism=8

- --focus-regex=\[NodeConformance\]  // No single/double string quotes.

- --skip-regex=\[Flaky\]|\[Slow\]|\[Serial\] // No single/double string quotes.

6. “- --node-args=--image-config-file=/workspace/…” of non-kubetest2 jobs, is simply “--image-config-file=/home/prow/go/src/k8s.io/…”.

7. “--timeout” can be provided via “decoration-config.timeout”

8. "--deployment", "--node-tests", "--provider" are not required to be passed anymore.

Below is the list of added kubetest2 pre-submit jobs, along with the respective links to test-grid, all jobs config could be found here [ test-infra/sig-node-presubmit.yaml at master · kubernetes/test-infra (github.com)]

• Pull-kubernetes-node-e2e-kubetest2 [presubmits-kubernetes-nonblocking Test Grid (k8s.io)]

• Pull-kubernetes-node-e2e-containerd-kubetest2 [presubmits-kubernetes-nonblocking Test Grid (k8s.io)]

• Pull-kubernetes-node-e2e-containerd-features-kubetest2 [presubmits-kubernetes-nonblocking Test Grid (k8s.io)]

• Pull-kubernetes-node-e2e-alpha-kubetest2 [presubmits-kubernetes-nonblocking Test Grid (k8s.io)]

• Pull-kubernetes-node-kubelet-serial-kubetest2 [sig-node-presubmits Test Grid (k8s.io)]

• Pull-kubernetes-node-kubelet-serial-containerd-kubetest2 [sig-node-presubmits Test Grid (k8s.io)]

• Pull-kubernetes-node-kubelet-eviction-kubetest2 [sig-node-presubmits Test Grid (k8s.io)]

• Pull-kubernetes-node-kubelet-serial-cpu-manager-kubetest2 [sig-node-presubmits Test Grid (k8s.io)]

• Pull-kubernetes-node-kubelet-serial-topology-manager-kubetest2 [sig-node-presubmits Test Grid (k8s.io)]

• Pull-kubernetes-node-crio-cgrpv2-e2e-kubetest2 [sig-node-presubmits Test Grid (k8s.io)]

• Pull-kubernetes-node-crio-e2e-kubetest2 [sig-node-presubmits Test Grid (k8s.io)]

Note: all pre-submit jobs added for kubetest2 are optional and explicitly need “/test pull-kubernetes-node-....-kubetest2” to run. 

Kind regards

Naman Lakhwani