Envtest for pods and setting PodIP via status update request

[Stephen Connolly]

I have a controller that is interacting with the services running inside a statefulset of pods.

When each pod starts the service within each it needs interaction with the pod before it can transition to fully ready.

I am trying to write some tests for my controller using envtest... but even though I have a non-containerised version of the service running locally on localhost, I cannot update the status of the pod to have Phase Running or to have PodIP of 127.0.0.1.

Has anyone else encountered this issue before?

Or do I have to try and have my tests use a fully fake client rather than rely on envtest? (IIRC the fake doesn't implement watches as fully so I would have even more of a gap between the fake and the real system... which would be bad)

Thanks in advance

[Camila Macedo]

Hi Stephen,

Testing controller interactions with pods, especially within a StatefulSet, can indeed be challenging.

EnvTest Examples:

I would like to suggest to check out examples generated with the DeployImage plugin [1] in Kubebuilder’s testdata, which can help manage status conditions set by the controller in tests. These examples illustrate how to use Envtest to observe status updates. Here are some useful links:

• Controller Implementation: busybox_controller.go [2], where conditions are set by the controller.
• Controller Test for Status Conditions: busybox_controller_test.go [3], which includes assertions on status updates influenced by the controller logic.

PS: Regarding the fake client, as you noted, it’s generally not recommended due to its limitations and it can be removed/deprecated any time on the future. More info: controller-runtime issue #768.

To address your need:

Based on your requirements, it sounds like you may benefit more from creating a full e2e integration test of your project rather than just testing the controller implementation. Kubebuilder scaffolds basic e2e tests to help users get started, allowing you to run kubectl commands that set predefined conditions (for example to create the Pods with the status and etc that you need) and observe the reconciliation of your project.

Here are a few relevant resources for e2e tests as examples:

4. Default Scaffolded e2e Tests: project e2e tests [4].

5. Kubebuilder Project e2e Tests: e2e tests implemented in Kubebuilder [5], used to validate its features and solutions.

6. GitHub Action Workflow for e2e Tests: test-e2e.yml [6], which provides a comprehensive CI setup to run these tests on any pull request. So that, it is possible to validate if the project will still working well after the changes.

I hope this helps! Let me know if you need further clarification or if there’s anything specific you’re aiming to test.
Also, please feel free to reach out via the Slack channel—join us on the #kubebuilder [7] channel.

[1] - https://book.kubebuilder.io/plugins/available/deploy-image-plugin-v1-alpha
[2] - https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v4-with-plugins/internal/controller/busybox_controller.go
[3] - https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v4-with-plugins/internal/controller/busybox_controller_test.go#L139-L146
[4] - https://github.com/kubernetes-sigs/kubebuilder/tree/master/testdata/project-v4-with-plugins/test/e2e
[5] - https://github.com/kubernetes-sigs/kubebuilder/tree/master/test/e2e
[6] - https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v4-with-plugins/.github/workflows/test-e2e.yml
[7] - https://communityinviter.com/apps/kubernetes/community#kubebuilder

Best regards,

Camila Macedo

[Alvaro Aleman]

> PS: Regarding the fake client, as you noted, it’s generally not recommended due to its limitations and it can be removed/deprecated any time on the future. More info: controller-runtime issue #768.

Small clarification: There are no plans to remove or deprecate the fakeclient. It does not perfectly mimic the behavior of a real apiserver but it is reasonably close. Tests that use it are much faster than tests that use envtest or kind.

--
You received this message because you are subscribed to the Google Groups "kubebuilder" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubebuilder...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/kubebuilder/0e9342ac-643e-4234-a919-acfd4ed283fbn%40googlegroups.com.