Re: [kubernetes/kubernetes] ConfigMaps and Secrets mounted with subPath do not update when changed (#50345)

[Jordan Liggitt]

cc @kubernetes/sig-storage-bugs

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[Josh Berkus]

@maeb do you see this getting done in the next 5 days for 1.10?

[Josh Berkus]

This issue and its related PRs will be removed from 1.10 at Code Freeze on Monday, unless they are all updated with status/approved-for-milestone and with a progress update. If that's fine, do nothing; if you are still targeting 1.10, please let us know what's going on and update the labels. Thanks!

[Josh Berkus]

As a reminder, this issue is about to be kicked out of 1.10 tracking in 6 hours, because it doesn't have the required labels. If this is actually an 1.10 issue, please update it! @maeb @thesandlord

[Michelle Au]

I think we just need to document this as a known limitation.

Because subpaths are bind mounted by docker, if it was a symlink, then it gets resolved to the actual path during the bindmount.

[Michelle Au]

/assign

[Tim Pepper]

@msau42 if that's the decision, please remove v1.10 milestone designation

[k8s-ci-robot]

Closed #50345 via kubernetes/website#7533.

[Ming Fang]

why is this close?
what is the fix or is there going to be a fix?

[Michelle Au]

This is a known limitation of using subpath with atomic volume types and has been documented. There is no fix planned.

[Ming Fang]

@msau42 thanks for the clarification.

For anyone affected by this, I just learned about this https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#add-configmap-data-to-a-specific-path-in-the-volume and it seems to do the same thing as subPath but actually works.

For the original example above, the solution would look like this

        volumeMounts:
          - name: my-config
            mountPath: /usr/src/app/config
      volumes:
      - name: my-config
        configMap:
          name: my-config
          items:
          - key: config.json
            path: config.json

[Laci Kundra]

@mingfang you are right.

This configMap / items works great, it's a symlink, so it get updated when the ConfigMap updates.
However, it does not solve the other problem: let's say if I want to mount nginx.conf to /etc/nginx/nginx.conf the directory /etc/nginx becomes empty (removing all other default configs), and will only contain this (finally symlinked) nginx.conf :(

I would love to have a solution that combines symlinks (for hot reload), and keeping original directory contents...

[Vincent Berruchon]

@kundralaci it seems to be now possible , look at that : #44815 (comment)
I've just tested a few minutes ago and seems working great.

[Pieter Temmerman]

I bumped into this issue because I was having the same frustraing problem.
However, I think the following could be a legit workaround:

Designate a folder where you will mount your desired configmaps, for example:

/configmaps/

Make the mountPath of each configmap (or secret) a subfolder of this folder.
Going further on previous examples with nginx, your mount would look like this:

`

• name: nginx-configmap # contains nginx.conf
  mountPath: /configmaps/nginx
  `
  Now, to tie this all together, create a symlink from /etc/nginx/nginx.conf to /configmaps/nginx/nginx.conf. This symlink can exist before the configmap is even mounted.

Whenever you update the configmap, the /configmap/nginx folder is remounted, and your symlink is now pointing to the updated version.

[Jarred Nicholls]

@ptemmer It's a decent workaround no doubt. Just more boilerplate to handle on container start.

[Jeff Harris]

@ptemmer

[Pieter Temmerman]

@jeff-1amstudios In the Dockerfile of the image

[王海波]

we do it like this:
initContainers: - name: init-myservice image: openresty:1.13.6.1 volumeMounts: - mountPath: /tmpconfig name: tmpconfig - mountPath: /configmap/usr/local/openresty/nginx/conf/ name: nginx-conf command: ['sh', '-c', 'mkdir /tmpconfig/test; cp /usr/local/openresty/nginx/conf/* /tmpconfig/test;ln -sf /configmap/usr/local/openresty/nginx/conf/* /tmpconfig/test/;'] containers: - image: openresty:1.13.6.1 name: openresty command: ['sleep', '100000'] volumeMounts: - mountPath: /usr/local/openresty/nginx/conf/ name: tmpconfig subPath: test - mountPath: /configmap/usr/local/openresty/nginx/conf/ name: nginx-conf volumes: - configMap: defaultMode: 420 items: - key: openresty-nginx.conf path: nginx.conf name: nginxconf name: nginx-conf - emptyDir: {} name: tmpconfig

it work for us but not clean enough

[Punit Agrawal]

+1 just saw this issue with nginx configmap as well. k8s 1.10.x

[Jeff Valore]

Any way that this issue can be reopened?
I ran into it with AWS EKS, and mounting a ConfigMap like

apiVersion: v1
kind: ConfigMap
metadata:
  name: site-config
  ...
data:
  site_config.yml: |
    ...some raw config string...

---

apiVersion: apps/v1
kind: Deployment

spec:
  ...
  template:
    ...
    spec:
      containers:
        ...
          volumeMounts:
            - name: config-volume
              mountPath: /usr/src/app/site_config.yml
              subPath: site_config.yml
      volumes:
      - name: config-volume
        configMap:
          name: site-config

You can update the ConfigMap, then wait some time (I waited 30min).
Then kubectl exec -ti {pod} -- bash into the running pod, cat /usr/src/app/site_config.yml and still see the old ConfigMap value.

Deleting the pod to force recreation results in having the correct file contents mounted.

—
You are receiving this because you are on a team that was mentioned.

Reply to this email directly, view it on GitHub, or unsubscribe.

[aaronsuns]

There are so many people encounter exactly the same problem as I have, there are existing files on a existing folder, I want to add more files from a configmap/secret, it works as subPath, but it not auto-updated when configmap/secret is changed. would be good to get that work, not lots of symlink or initContainer work around.

[debu99]

We have same issue because of the configmap is not updated

        message: 'OCI runtime create failed: container_linux.go:349: starting container
          process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58:
          mounting \\\"/var/lib/kubelet/pods/7873e103-a195-11ea-ab9c-0acac043f5ec/volume-subpaths/app-configvolume-1/appconfig/1\\\"
          to rootfs \\\"/var/lib/docker/overlay2/feaf7a8d06777072d8c89fe2105e075ff23ce88fdd42c95fd6b5862d1b1b097c/merged\\\"
          at \\\"/var/lib/docker/overlay2/feaf7a8d06777072d8c89fe2105e075ff23ce88fdd42c95fd6b5862d1b1b097c/merged/app/app.yml\\\"
          caused \\\"no such file or directory\\\"\"": unknown'

[Leon Keijser]

@wanghaibo using your example for our Prometheus instance. Works perfectly, thanks!

[Michelle Au]

Does the alternative of using configmap items or projected volumes work?

[Diego Carvallo]

It would help if the subPath limitation was also documented in the ConfigMaps section where no limitation is mentioned and took me hours to get to this thread and understand what was the underlying issue: https://kubernetes.io/docs/concepts/configuration/configmap/#mounted-configmaps-are-updated-automatically

[aaronsuns]

It's so sad that this one is closed without any improvement. So we have to use symlink to workaround this problem?

[Diego Carvallo]

Using symlinks didn't work for me. I tried creating the symlinks on an initContainer for a config file being mounted from a ConfigMap, but later when it's the turn for the app container to run (Elasticsearch in my case) it overwrites the symlinks with it's default config files. And I have no way of having this app reading from a different path it would not overwrite.

[Erik Sundell]

Attempted issue summary

The subPath option is the k8s way to mount a single file in a folder that is already populated with other files. ConfigMap and Secret resources are documented to automatically update, but has a disclaimer that usage of subPath will disable such automatic updating.

A workaround strategy (#50345 (comment)) and an implementation of it (#50345 (comment)) has been described, and involves mounting the secret/configmap without subPath in a different folder and manually creating a symlink from an initContainer ahead of time to that folder.

The issue is closed as its considered a documented limitation rather than a bug.

Future?

I guess the constructive thing may be to read about KEPs here: https://github.com/kubernetes/enhancements/tree/master/keps, and explore if a KEP is already around regarding removing this limitation. I didn't find such proposal at this time.

[Michelle Au]

The main challenge is technical. The subpath implementation uses bind-mounts for security reasons, and bind mount will stay with the original inode. I do not think we can remove the bind mount implementation, so the main alternatives to explore would be if we can get projected or configmap.items to provide single files instead of the whole directory.