[kubernetes/kubernetes] PVC/PV conformance testing discussion (#65155)

[Michelle Au]

Is this a BUG REPORT or FEATURE REQUEST?:
@kubernetes/sig-storage-feature-requests

As a follow up to the conformance testing discussion in kubernetes/features#498, we had a meeting to brainstorm ideas around conformance testing of PVCs/PVs. Some ideas we came up with:

• The core controller paths with PV controller (both static and dynamic provisioning), attach/detach, kubelet volume manager mount/unmount can be part of a core Kubernetes suite using a mock CSI plugin. This would only test the Kubernetes control path and the PVC/PV lifecycle management. Even something like default StorageClasses could be tested if we made the mock CSI driver a default StorageClass (and disabled the provider-specific default StorageClass if preinstalled)
• Data path tests (ie reading/writing), data persistence during disruption/errors (rescheduling pods on other nodes) requires a real volume plugin implementation and is a good candidate for a "persistent volumes profile" conformance suite. The behavior here is very volume-plugin dependent, and not all providers may have a "default" volume plugin that they include. So this may actually be more of a volume plugin conformance suite, instead of a provider suite, where each volume plugin can be certified to meet some standard behavior (ie attach/detach handling in error conditions, read only behavior, permission handling, etc). Providers could potentially list which persistent volume plugins they include out of the box.

Some action items/questions:

• [sig-architecture] Is having a core conformance suite that tests the control path for volume handling with a mock plugin that doesn't actually run a real user experience useful? It would indicate that your provider at least runs all the necessary controllers.
• [sig-architecture] Is having a profile conformance suite that requires providers to include support for at least one volume plugin out of the box useful? This would map to user experience much more than the former.
• [sig-storage] For sure we should have a CSI plugin conformance standard (that is being handled separately outside of Kubernetes). What about in-tree plugins?
• [@msau42] Once the above questions are answered, come up with a plan for:
  • what tests can go in each suite
  • what work needs to be done to modify existing tests, add new tests to the conformance suites
  • multi-release plan for adding those tests

cc @WilliamDenniss @bgrant0607 @AishSundar @pospispa @saad-ali

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[Michelle Au]

Discussed a bit with @saad-ali. We brainstormed having 3 different suites for persistent volumes.

1. Core conformance suite that tests PVC/PV lifecycle management with a mock driver. All distributions must support this.
   • Tests that Create, Delete, Attach, Detach, Mount, Unmount, Resize are called on appropriate nodes at appropriate times in the volume lifecycle
   • Generic PV features that can be tested here: PVC/PV prebinding/binding, dynamic provisioning, default StorageClasses, reclaim policy, access modes, StorageProtection, volume topology (and integration with Pod topology), mount options, block volumes, mount and attach handling in node conditions (nodes are deleted, notready, drained, tainted)
2. Persistent volume profile suite that tests the data and control path. Distributions must bundle one or more real persistent volume drivers out of the box. All the tests in the core conformance suite must pass (with some exceptions depending on subcategory), but with a real volume driver instead of a mock. In addition to control path tests, data path and data persistence are also tested. Tests could be further split up into smaller subprofiles:
   a. Single writer filesystem volume
   * A pod writes data to a volume. Force the pod to be rescheduled on a different node. The data should be accessible from the new node.
   * Try to create multiple pods on different nodes, subsequent pods should fail.
   b. Multi writer filesystem volume
   * Tests that data persists across multiple pods scheduled on different nodes at the same time.
   c. Local persistent filesystem volume
   * Tests data locality of the pod to the volume, and that the pod is always scheduled to the same node.
   d. Repeat a-c for block volumes
   e. Generic features that can be tested across all subprofiles: readOnly, subpath.
   f. There are some features like dynamic provisioning and volume resizing that may not be supported by all plugins. We can still have suites for these "optional" features and if the provided plugin also supports these, then they could add the extra designation to it. (ie, supports single writer + dynamic provisioning + default storageclass + resizing)
3. Driver conformance suite that tests the data and control path. Drivers could run against the same tests above, and be certified to work in Kubernetes. The difference from 2) is that they don't need to be bundled into the distribution, so various storage vendors that are independent of Kubernetes platforms can also meet Kubernetes conformance.

[Aish Sundar]

@msau42 are you still blocked on answers from sig-architecture to decide on further course of action. If so will you be taking up the ques with sig-architecture?

[Michelle Au]

The main question I have: Is there value to having a core suite that tests only the control path with a mock driver? Just because the tests pass with a mock driver, doesn't mean that they will pass with a real volume driver. You could have OS/kernel dependencies that will prevent a real volume plugin from working. We can discuss this at the next conformance meeting and go from there.

The real value to the user is the persistent volume profile suite because it tests both control + data path integration of specific volume drivers in a particular platform. We can definitely focus on this first.

[Michelle Au]

Conclusion from the comforance wg meeting:

• It's not valuable to test only the control plane without a real driver
• Let's focus on a Persistent Storage profile suite that is optional and not included into the base conformance suite
  • Define features that all persistent volume plugins must support
  • Define optional features that are not required for conformance, but are still valuable for users

[Aish Sundar]

/cc @mgdevstack @akurylec @joseluis

[Aish Sundar]

#65793

[Michelle Au]

Summary of the current idea for storage conformance tests:

Have 3 different conformance suites, and some examples for possible test cases in each:

1. Core conformance: All Kubernetes distros must pass
   • In-tree ephemeral volumes like emptydir, secrets, configmaps, downward api, etc (these already exist, maybe more can be added)
   • Hostpath
   • PVC/PV features that don't require a specific volume plugin: pre-binding, PV protection, Retain reclaim policy, etc.
2. Persistent volume plugin profile: Every persistent volume plugin must pass. Kubernetes distros may bundle in 1+ persistent volume plugins to be persistent volume certified (optional).
   • Basic read/write tests
   • Data persistence across pod restarts, various node failure conditions
   • Proper mount cleanup
   • PVC protection
3. Persistent volume plugin optional features: Each persistent volume plugin may additionally support these optional features. They're not required for conformance, but are important features that users will consider when choosing which volume plugin is appropriate for their needs.
   • Dynamic provisioning
   • Default storage class
   • Resizing
   • Snapshots
   • RWO vs RWX
   • Block
   • Fsgroup/permissions

More detailed test case tracking is available here. This sheet is open to anyone in kubernetes-dev.

[Brian Grant]

/conformance

[Brian Grant]

Catching up. Keep in mind that it needs to be possible to run the conformance suite against managed services that may have limited configurability.

[Brian Grant]

I'm not sure it makes sense to have a storage-specific suite as part of Kubernetes certification. As with "node conformance" tests, the volume plugin tests sound useful independently of certification.

We haven't sorted out profiles yet, but there should not to be dozens of them, or else the numbers of possible combinations of different profiles would defeat the workload portability goal. (As well as being confusing for users.)

My current thinking is that we should bundle a common set of optional functionality into the first profile. For example, maybe it could contain RBAC, LoadBalancer Services, and single writer + dynamic provisioning + default storageclass.

[Michelle Au]

it needs to be possible to run the conformance suite against managed services that may have limited configurability.

Can the tests themselves be configurable? For example, each provider can configure the test to drop in configuration for their supported volume plugin? This becomes especially important for providers that don't have in-tree volume types and need to use Flex or CSI drivers.

we should bundle a common set of optional functionality into the first profile. For example, maybe it could contain RBAC, LoadBalancer Services, and single writer + dynamic provisioning + default storageclass.

I think that could make sense for something like a "stateful workload" profile.

[Brian Grant]

@msau42 Re. test configurability: I think it would be much better for users if there were a StorageClass that represented a consistent set of features/behaviors across providers, and that the conformance tests validated that behavior, rather than creating a configuration that existed just for the purpose of the tests. I also don't yet have a sense for how many providers will or won't enable users to choose and install CSI drivers.

[Michelle Au]

It makes sense to test for a portable, out of the box experience with no extra steps needed. I think in that case, we could easily promote some of the existing StatefulSet tests that use default StorageClass into a conformance profile suite and write new tests around pod and PVC lifecycles.

I'll pursue the rest of the items outside of the conformance area and instead as part of a new volume plugin feature validation suite.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[Michelle Au]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

—

[Michelle Au]

/remove-lifecycle stale
/lifecycle frozen

[Vallery Lancey]

@msau42 is there a proposal that sig-architecture should look at?
/priority backlog

[Michelle Au]

There's no proposal. We were last stuck on how to handle optional features in conformance. Profiles were discussed as one option. I would be happy to work with someone on a proposal if there has been any progress on this front. Cc @johnbelamaric

[John Belamaric]

Nothing concrete around profiles or optional features yet, but I expect to have some ideas/plans on paper for discussion in the next week or so.

[Michelle Au]

@johnbelamaric I had another thought regarding storage tests in conformance. Would it be useful to be able to test CSI conformance in the Kubernetes conformance suite using a non-cloud provider driver such as the CSI hostpath driver? It has no dependencies on cloud providers, and the tests we currently have deploy fixed images of the hostpath csi driver. By doing this, we at least can provide some confidence about basic CSI support in Kubernetes distros.

We would still want separate optional profiles/tests to test real storage implementations.

[John Belamaric]

Yes, exercising the CSI integration independently from cloud providers is
valuable, for sure.

On Thu, Jul 11, 2019 at 12:05 AM Michelle Au <notifi...@github.com>
wrote:

> @johnbelamaric <https://github.com/johnbelamaric> I had another thought

> regarding storage tests in conformance. Would it be useful to be able to
> test CSI conformance in the Kubernetes conformance suite using a non-cloud
> provider driver such as the CSI hostpath driver? It has no dependencies on
> cloud providers, and the tests we currently have deploy fixed images of the
> hostpath csi driver. By doing this, we at least can provide some confidence
> about basic CSI support in Kubernetes distros.
>
> We would still want separate optional profiles/tests to test real storage
> implementations.
>
> —

> You are receiving this because you were mentioned.

> Reply to this email directly, view it on GitHub

> <https://github.com/kubernetes/kubernetes/issues/65155?email_source=notifications&email_token=ACIHRM3DPM2BLQCFXJE7IC3P6ZMKVA5CNFSM4FFIVGXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZU4BGQ#issuecomment-510247066>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACIHRM3RQ3AZA3YGYDS4HETP6ZMKVANCNFSM4FFIVGXA>
> .

[kfox1111]

I like the idea of testing with a dummy driver. How dummy it is though may matter. For example, currently minikube is broken with regards to csi drivers:
kubernetes/minikube#4072

Testing with the csi-driver-hostpath driver though uncovers the issue.

[Michelle Au]

I think it can't be completely mocked out like the mock driver, but hostpath seems more reasonable. Hostpath does have the limitation though of all the volumes being restricted to a single node, so we won't be able to add multi-node test cases that test things like data persistence.

But that's more of an evaluation of driver capabilities, more than testing whether or this k8s cluster makes all the proper csi calls

[Patrick Ohly]

Would it be useful to be able to test CSI conformance in the Kubernetes conformance suite using a non-cloud provider driver such as the CSI hostpath driver?

A test that depends on the hostpath driver might not meet the current conformance tests requirements because it depends on a privileged pod.

A cluster is not required to support those (at least that is my understanding), or if it does, then how to authorize those pods to run is not standardized and thus a test might become non-portable.

In the current E2E test, we get by with a special cluster role binding that is required on some clusters and ignored on others: https://github.com/kubernetes/kubernetes/blob/c495744436fc94ebbef2fcbeb97699ca96fe02dd/test/e2e/testing-manifests/storage-csi/hostpath/hostpath/e2e-test-rbac.yaml#L1-L31

—
You are receiving this because you are on a team that was mentioned.

Reply to this email directly, view it on GitHub, or unsubscribe.

[Michelle Au]

I spoke to @johnbelamaric earlier about the privileged requirement and concluded that we can relax it if necessary. So I don't consider requiring privileged an issue.