[kubernetes/kubernetes] Ensure that host disk space used by Docker volume is accounted for (#52032)

[Saad Ali]

From Docker docs:

If you start a container with a volume that does not yet exist, Docker creates the volume for you. The following example mounts the volume myvol2 into /app/ in the container.

$ docker run -d \
  -it \
  --name devtest \
  -v myvol2:/app \
  nginx:latest

Use docker inspect devtest to verify that the volume was created and mounted correctly. Look for the Mounts section:

"Mounts": [
    {
        "Type": "volume",
        "Name": "myvol2",
        "Source": "/var/lib/docker/volumes/myvol2/_data",
        "Destination": "/app",
        "Driver": "local",
        "Mode": "",
        "RW": true,
        "Propagation": ""
    }
],

This shows that the mount is a volume, it shows the correct source and destination, and that the mount is read-write.

If the docker volume is not backed by a docker volume driver then docker creates a directory on the host machine to back the volume.

Kubernetes is implementing a mechanism to control the amount of host disk space a container can use. See the Local Ephemeral Storage Resource Management design in kubernetes/community#991.

Purpose of this bug is to make sure that Kubernetes accounts for docker volumes backed by host disk as part of ephemeral-storage.

/assign @jingxu97

CC @Random-Liu @kubernetes/sig-storage-bugs @kubernetes/sig-node-bugs

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[Kubernetes Submit Queue]

[MILESTONENOTIFIER] Milestone Labels Complete

@jingxu97 @saad-ali

Issue label settings:

• sig/node sig/storage: Issue will be escalated to these SIGs if needed.
• priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.
• kind/bug: Fixes a bug discovered during the current release.

Additional instructions available here

[Lantao Liu]

@saad-ali This is not a regression. Do we want this be a 1.8 issue?

[Yu-Ju Hong]

@saad-ali This is not a regression. Do we want this be a 1.8 issue?

+1. Doesn't seem like a 1.8 issue.

[Saad Ali]

My understanding is that the ephemeral-storage limit is being introduced in 1.8? @jingxu97?

[Lantao Liu]

Also ref kubernetes-incubator/cri-containerd#186 (comment).

Background

We don't use docker volume in Kubernetes. However, Dockerfile does support VOLUME , user could create VOLUME and write data into it in the Dockerfile. When docker runs the container, it will:

1. create a host volume;
2. copy data from image into the volume;
3. bind mount the volume back to the image.

I believe there are users relying on this behavior, at least, mongo:latest and the microservices demo do reply on the VOLUME.

Actually, not only Docker image. Volumes is also supported in OCI image spec 1.0. It means that all OCI runtimes also need to support it.

Problem

Today kubelet is not aware of these host volumes, and not doing accounting for it. It means that application could use arbitrarily use host disk and not being monitored.

Possible solutions

• Option 1: Tell all OCI runtimes to use rootfs instead of creating host volumes.
  • Cons:
    • Volumes will also be readonly on readonly rootfs (regression), at least the microservices demo does rely on it;
    • Performance regression (overlay filesystem copy-up).
• Option 2: Change CRI to let runtime return host volumes for containers.
  • Cons: Need to figure out whether this aligns with our overall design of Volumes in Kubernetes.
• Option 3: Let container runtime return OCI image config, and let Kubelet do the volume mount.
  • Cons:
    • Making assumption on image format.
    • If we want to support copying data from container rootfs to volume, kubelet needs access to container rootfs, which is tricky;
    • More burden on kubelet
• Option 4: Update pod spec based on image config in admission controller.
  • Cons:
    • Making assumption on image format.
    • Even more tricky;
• Option 5: Explicitly tell user not to use it, and use Kubenetes volumes instead.
  • Cons:
    • Takes time.
    • Not user friendly, because they are using it (need to figure out how many users are using it).

Seems like option 2 is a better near term solution; And option 4/5 is better long term solution.

[Vish Kannan]

Hostpath volumes are not supported because of the inability to identify sharing patterns. Docker volumes described in this context is hostpath volumes to me. @jingxu97 correct me if I got it wrong.

[Yu-Ju Hong]

Hostpath volumes are not supported because of the inability to identify sharing patterns. Docker volumes described in this context is hostpath volumes to me. @jingxu97 correct me if I got it wrong.

k8s doesn't allow sharing of this types of volumes -- it doesn't pass -v when creating a container.

This means that so far these volumes are just private, per-container volumes created by docker and will be destroyed along with the container. This is similar to the emptyDir volumes in kubernetes. It's not that useful unless we explicitly support docker volumes.

The volume section in the OCI image spec is very vague. It seems like this should be the user's choice to decide what kind of volumes to create for the container instance at run time.

I like Option 4 but it seems less viable at this moment. How about a variant of Option 3?
After pulling down the image, kubelet can inspect the image to discover these "volumes". It can then create emptyDir or host volumes based on the default policy or pod spec. Next, it can pass these "special mounts" to the runtime via CRI and runtime would be responsible for copying the data if necessary.

We still have the problem of recording the volumes and surfacing them properly.

[Lantao Liu]

I like Option 4 but it seems less viable at this moment. How about a variant of Option 3?
After pulling down the image, kubelet can inspect the image to discover these "volumes". It can then create emptyDir or host volumes based on the default policy or pod spec. Next, it can pass these "special mounts" to the runtime via CRI and runtime would be responsible for copying the data if necessary.
We still have the problem of recording the volumes and surfacing them properly.

I also prefer letting kubelet to make the decision, so that user could define what the underlying storage should be for the image defined volumes. By default we could use EmptyDir, and could define policy for image defined Volumes.

For other images which don't care about this, they could just return empty list.

[Yu-Ju Hong]

My understanding is that the ephemeral-storage limit is being introduced in 1.8? @jingxu97?

Is this feature alpha or beta in 1.8? I think it's ok not to account these volumes for an alpha feature even if we do decide to do so.
I'd suggest punting this to the next release or later given that the volumes are not well-supported in kubernetes.

[Kubernetes Submit Queue]

[MILESTONENOTIFIER] Milestone Labels Complete

@jingxu97 @saad-ali

Issue label settings:

• sig/node sig/storage: Issue will be escalated to these SIGs if needed.
• priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.
• kind/bug: Fixes a bug discovered during the current release.

Additional instructions available here The commands available for adding these labels are documented here

[Saad Ali]

My understanding is that the ephemeral-storage limit is being introduced in 1.8? @jingxu97?

Is this feature alpha or beta in 1.8? I think it's ok not to account these volumes for an alpha feature even if we do decide to do so.
I'd suggest punting this to the next release or later given that the volumes are not well-supported in kubernetes.

It is alpha so I'm ok with that. Removing from 1.8 milestone.

[Jing Xu]

hostpath is by definition is persistent volume which has different lifecyle of pod. The docker volumes also exist outside the lifecycle of a given container, so I think they are like hostpath volumes. For our current local ephemeral storage isolation feature, we do not management hostpath persistent volumes.
For persistent volume, we suggest to use new feature of local volume which is normally set up on a separate partition for easy management. If user choose the share local volume on root file system, we cannot manage the use of the local volumes at least for now.

[Yu-Ju Hong]

The docker volumes also exist outside the lifecycle of a given container, so I think they are like hostpath volumes. For our current local ephemeral storage isolation feature, we do not management hostpath persistent volumes.

Today we practically treat them as private volumes of the container -- there is no sharing and the volume gets removed along with the container. As mentioned #52032 (comment), there is no way for pods/containers to share these volumes using kubernetes API.

Whether we want to support these volumes and how to support them is a separate question.

[Lantao Liu]

hostpath is by definition is persistent volume which has different lifecyle of pod. The docker volumes also exist outside the lifecycle of a given container, so I think they are like hostpath volumes.

That is not the case for image specified volumes. For image specified volume, docker creates it when creating the container, and delete it when deleting the container (Kubelet chooses to do that specifying RemoveVolumes)

So I believe the image volumes are still in the scope of Local Ephemeral Storage Resource Management. However, we could not manage it before we change CRI and kubelet, so I don't think we could not fix this in 1.8.

However, we may want to revisit this in 1.9.

[Derek Carr]

fwiw, this has been a discussion for us recently.

for clusters where we needed to control this, we can optionally run the following:
https://github.com/projectatomic/docker-novolume-plugin

but its aggressive in that it denies the ability to start a container that uses an image volume.

for cri-o, we want the runtime to support "ignore" image volumes, which i think is the right long term behavior for kubernetes.

https://github.com/kubernetes-incubator/cri-o/blob/66d96e77e33bf773781e2a8313e8a73b9c679caa/docs/crio.8.md#global-options

see --image-volumes parameter for configuring cri-o daemon.

fyi @mrunalp @sjenning

[Lantao Liu]

for cri-o, we want the runtime to support "ignore" image volumes, which i think is the right long term behavior for kubernetes.

The problem is that there are images relying on the image volume, e.g.:

1. Some assumes that image volume should be writable even if rootfs is read-only;
2. Some assumes that their data baked in the image should be copied to the volume mounted.

Given so, we may still need to support this unless we could tell user not to use this.
Since image volume is part of OCI image spec now, we may want to properly support it. @yujuhong 's comment #52032 (comment) sounds like a good option to me.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

Closed #52032.

[wenlxie]

@Random-Liu @saad-ali
Seems this feature is still not include into ephemeral-storage feature. Any plan to enhance this?
https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how-pods-with-ephemeral-storage-limits-run

[Lantao Liu]

/reopen
/remove-lifecycle rotten

[Kubernetes Prow Robot]

@Random-Liu: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Kubernetes Prow Robot]

Reopened #52032.

[fejta-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

—
You are receiving this because you are on a team that was mentioned.

Reply to this email directly, view it on GitHub, or unsubscribe.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.

/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[Kubernetes Prow Robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

—

[Kubernetes Prow Robot]

Closed #52032.

[wu0407]

/reopen

[Kubernetes Prow Robot]

@wu0407: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

—

[Bo Liu]

@Random-Liu any updates ?