Re: [kubernetes/kubernetes] Secrets with specific permissions (defaultMode or mode) not being applied in Kubernetes 1.4.0 (#34982)
Tim Hockin
@kubernetes/sig-storage-bugs because volume
@kubernetes/sig-node-bugs because kubelet
This is probably straight-forward, just needs someone to look at it.
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
leopoldodonnell
In my case the lifecycle approach ran into race condition issues. If it doesn't work for you, consider using an init container. This solved my problem with a secret git-secret with a key ssh
# FIXME: Remove this when defaultMode is working initContainers: - name: fix-perms image: busybox command: - sh - -c - /bin/chmod 400 /etc/git-secret/ssh volumeMounts: - name: git-secret mountPath: /etc/git-secret/ssh subPath: ssh securityContext: runAsUser: 0 volumes: - name: git-secret secret: secretName: the-secret-name defaultMode: 0400
dankirkpatrickmp2
Joshperry--you're getting the files mounted with octal permissions "0620"...0620 is 400 in decimal. The mode/defaultMode field is treating "400" as a decimal number...and you're getting the expected behavior for that.
Joshua Perry
Wow, @dankirkpatrickmp2. Thank you much, we flubbed that one alright.
Joshua Perry
I've changed my config to use octal 0400 (also tried 256 decimal) and it is now working properly. So perhaps not an upgrade issue, at least for me, from ~1.3.
fejta-bot
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Prevent issues from auto-closing with an /lifecycle frozen comment.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale
Tim Hockin
This appears fixed (by my repro test)
Tim Hockin
Closed #34982.
Richmond
I am having this issue now.
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T12:22:21Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.2-gke.1", GitCommit:"4ce7af72d8d343ea2f7680348852db641ff573af", GitTreeState:"clean", BuildDate:"2018-01-31T22:30:55Z", GoVersion:"go1.9.2b4", Compiler:"gc", Platform:"linux/amd64"}
Tim Hockin
On Wed, Feb 21, 2018 at 11:03 PM, Richmond <notifi...@github.com> wrote:
> I am having this issue now.
>
> Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T12:22:21Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
> Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.2-gke.1", GitCommit:"4ce7af72d8d343ea2f7680348852db641ff573af", GitTreeState:"clean", BuildDate:"2018-01-31T22:30:55Z", GoVersion:"go1.9.2b4", Compiler:"gc", Platform:"linux/amd64"}
>
> —
> Reply to this email directly, view it on GitHub
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AFVgVJRmtH8WDRf_-l6YmqpbMVxWDqkHks5tXREzgaJpZM4KZHug>
> .
Richmond
I followed this post (http://pauldone.blogspot.jp/2017/06/deploying-mongodb-on-kubernetes-gke25.html).
We have the same issue with this SO post (https://stackoverflow.com/questions/48816359/kubernetes-mongodb-non-root-keyfile-secret-permission-denied).
Michael Green
@richmondwang @leopoldodonnell @schasse @rainder @thockin There was no bug It seems like it was an issue with how the key was being generated by the post everyone is referencing.
https://github.com/MichaelScript/kubernetes-mongodb
If you run the example I created and kubectl logs mongod-0 you can see that the file now has the correct permissions and is mounted correctly.
Michael Green
@thockin I think there might be an issue where if the secret has no data then it mounts it anyways (not sure if this is the specified behavior but it might be okay)
If this is the specified behavior then there is a bug where the file permissions aren't modified of the mounted file that has no data. Which is what was causing everyone here issues because it isn't obvious that the generated secret has no data without specifically checking for that. (i.e: kubectl get secrets my-secret -o yaml)
For example if you try to mount the valid secret:
apiVersion: v1
kind: Secret
metadata:
creationTimestamp: 2018-04-15T05:41:34Z
name: my-secret
namespace: default
resourceVersion: "173565"
selfLink: /api/v1/namespaces/default/secrets/shared-bootstrap-data
uid: a8558142-406f-11e8-9f89-08002714eb9a
type: Opaque
and then modify it's permissions with defaultMode (i.e: 400) the permissions will not match (I think it gets set to the default)
This is different from the original issue and more to do with the absence of data in the secret.
Michael Grosser
Reproduction as far as I can see.
defaultMode: 256- secret not empty
Result: 777 /test/secret
apiVersion: v1
kind: Namespace
metadata:
name: test-secret
---
apiVersion: v1
kind: Secret
metadata:
name: test-secret
namespace: test-secret
labels:
project: test-secret
stringData:
secret: "123"
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: test-secret
namespace: test-secret
spec:
replicas: 1
selector:
matchLabels:
project: test
template:
metadata:
labels:
project: test
spec:
containers:
- name: test
image: ubuntu
command: [ "sh", "-c", "while true; do stat -c \"%a %n\" /test/secret; sleep 60; done" ]
volumeMounts:
- name: secret
mountPath: /test
volumes:
- name: secret
secret:
defaultMode: 256
secretName: test-secret
Lephix
Permission has problem in our env. Following is the snippet. In the container the /home/xxx/.ssh directory and the file authroized_keys both have a 0777 permission.
volumeMounts:
- name: host-keys
mountPath: /home/xxx/.ssh/
- name: host-keys
secret:
secretName: ssh-host-keys
items:
- key: authorized_keys
path: authorized_keys
Thomas Buckley-Houston
There appears to be a recent and more significant reason why you can't change the permissions of a secret mount - a recent CVE issue required that all secret, configMap, downwardAPI and projected volumes mounts be read-only: #58720
So one approach to workaround this is to introduce yet another intermediate volume. You can then use an initContainer to copy from the secret mounted volume to the intermediate container. And then finally mount the intermediate volume that is not restricted to read-only on your application container.
Here is an example: https://github.com/kubernetes/charts/blob/37620d43064f7b24e6591c27fd4d110b30c83cce/stable/rabbitmq-ha/templates/statefulset.yaml#L30
Kilian Ciuffolo
I have the same problem and I am not sure if the docs are wrong or this is still a bug. Is mode: 384 supposed to set a file pointing to a secret to 0600 ? The behavior I am seeing says no.
Tim Hockin
I flagged this for followup, and finally had some time. My test shows it working -- someone tell me otherwise?
kubectl apply this file:
apiVersion: v1
kind: Secret
metadata:
name: test-secret-perms
stringData:
expect_0700: "should be 0700"
expect_0400: "should be 0400"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-secret-perms
spec:
replicas: 1
selector:
matchLabels:
app: test-secret-perms
template:
metadata:
labels:
app: test-secret-perms
spec:
containers:
- name: test
image: ubuntu
command: [ "sh", "-c", "while true; do ls -ldH /test/*; sleep 60; done" ]
volumeMounts:
- name: secret
mountPath: /test
securityContext:
fsGroup: 123
volumes:
- name: secret
secret:
secretName: test-secret-perms
defaultMode: 256
items:
- key: "expect_0700"
path: "expect_0700"
mode: 448
- key: "expect_0400"
path: "expect_0400"
# no mode
What I see in the logs is:
-r--r----- 1 root 123 14 Dec 21 22:44 /test/expect_0400
-rwxr----- 1 root 123 14 Dec 21 22:44 /test/expect_0700
0400 and 0700 are correct. The extra g+r is because I tested the fsGroup at the same time.
최준영
How can I get below(key, path)?
items: - key: "expect_0700" path: "expect_0700" mode: 448 - key: "expect_0400" path: "expect_0400"
my secret describe:
Name: test
Namespace: default
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
id_rsa: 3247 bytes
id_rsa.pub: 745 bytes
Are the keys id_rsa and id_rsa.pub?
최준영
최준영
@thockin
didn't work for me 😢
I applied your above yaml file.
Belows are output:
root@test-secret-perms-996fb7547-n7689:/test# ls -al
total 4
drwxrwsrwt 3 root 123 120 Jan 21 03:26 .
drwxr-xr-x 1 root root 4096 Jan 21 03:26 ..
drwxr-sr-x 2 root 123 80 Jan 21 03:26 ..2019_01_21_03_26_48.920981748
lrwxrwxrwx 1 root root 31 Jan 21 03:26 ..data -> ..2019_01_21_03_26_48.920981748
lrwxrwxrwx 1 root root 18 Jan 21 03:26 expect_0400 -> ..data/expect_0400
lrwxrwxrwx 1 root root 18 Jan 21 03:26 expect_0700 -> ..data/expect_0700
Which k8s version should I use?
my version:
kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Darrien Glasser
Setting the defaultMode didn't seem to work for me either. We needed a script to run at start, so I ended up copying the keys over to different dirs, and setting permissions to them in that script.
Barbaric, but I guess it works 😬
petrokashlikov
@thockin
didn't work for me 😢I applied your above yaml file.
Belows are output:root@test-secret-perms-996fb7547-n7689:/test# ls -al total 4 drwxrwsrwt 3 root 123 120 Jan 21 03:26 . drwxr-xr-x 1 root root 4096 Jan 21 03:26 .. drwxr-sr-x 2 root 123 80 Jan 21 03:26 ..2019_01_21_03_26_48.920981748 lrwxrwxrwx 1 root root 31 Jan 21 03:26 ..data -> ..2019_01_21_03_26_48.920981748 lrwxrwxrwx 1 root root 18 Jan 21 03:26 expect_0400 -> ..data/expect_0400 lrwxrwxrwx 1 root root 18 Jan 21 03:26 expect_0700 -> ..data/expect_0700Which
k8sversion should I use?my version:
kubectl version Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
I also faced similar issue and seems it is issue with representation, but actual permissions are fine.
I've set defaultMode: 256 for volume with ssh keys. When I look at /etc/ssh/keys folder which I mounted, permissions doesn't look right
lrwxrwxrwx 1 root root 27 Feb 15 23:29 ssh_host_rsa_key.pub -> ..data/ssh_host_rsa_key.pub lrwxrwxrwx 1 root root 23 Feb 15 23:29 ssh_host_rsa_key -> ..data/ssh_host_rsa_key lrwxrwxrwx 1 root root 31 Feb 15 23:29 ssh_host_ed25519_key.pub -> ..data/ssh_host_ed25519_key.pub lrwxrwxrwx 1 root root 27 Feb 15 23:29 ssh_host_ed25519_key -> ..data/ssh_host_ed25519_key
but you see that it is actually not files, but links to actual files and if you go to that directory, permissions are read only and from original /etc/ssh/keys dir you in fact can't delete files, so there is no write permissions as displayed
root@sftp-5f5cb76dd4-mljww:/etc/ssh/keys/..data# ls -lrt total 16 -r-------- 1 root root 754 Feb 15 23:29 ssh_host_rsa_key.pub -r-------- 1 root root 3401 Feb 15 23:29 ssh_host_rsa_key -r-------- 1 root root 110 Feb 15 23:29 ssh_host_ed25519_key.pub -r-------- 1 root root 419 Feb 15 23:29 ssh_host_ed25519_key
Vaisakh Rajagopal
I was beating my head on this to understand what happening. @thockin Explaned its correctly.
TLDR; But if you, @petrokashlikov can set the permission to the configMap or secret file make use of subPath which wil create the file with the permission specified in defaultMode
Lets take look
eg:-
.
.
volumes:
- name: test-vol
secret:
defaultMode: 0400
secretName: test-secret
.
.
containers:
- name: test-cont
volumeMounts:
- mountPath: /root/non_exiting_folder/secret_key_1_new_name
name: test-secret
subPath: scret_key_1
- mountPath: /root/non_exiting_folder/secret_key_2_new_name
name: test-secret
subPath: scret_key_2
now if you get into the pod and check the permission
root@***:/root/non_exiting_folder/# ls -la
total 16
drwxr-xr-x 2 root root 4096 Feb 27 07:22 .
drwx------ 1 root root 4096 Feb 27 07:23 ..
-r-------- 1 root root 3243 Feb 27 07:22 secret_key_1_new_name
-r-------- 1 root root 422 Feb 27 07:22 secret_key_2_new_name
-
drwxr-xr-x 2 root root 4096 Feb 27 07:22 .
'.' is current folder; which is no_existing_folder created by k8s have the default permission 0755 -
-r-------- 1 root root 3243 Feb 27 07:22 secret_key_1_new_name
-r-------- 1 root root 422 Feb 27 07:22 secret_key_2_new_name
The no_existing_folder created by k8s have the default permission 0755
The secrete file(renamed scret_key_1 -> secret_key_1_new_name) by k8s is having the defaultMode as specified in config
Note:- If the no defaultMode is specifed, secret key file permission will be 0644 by default
Now comming back to @thockin explantion.
You may have notice that ..data, ..2019_01_21_03_26_48.920981748 folders are missing when you use the subPath config
If we remove the subPath config
eg:
.
.
volumes:
- name: test-vol
secret:
defaultMode: 0400
secretName: test-secret
.
.
containers:
- name: test-cont
volumeMounts:
- mountPath: /root/non_exiting_folder/child_folder
name: test-secret
See the different by getting into the pod
root@***:/root/non_exiting_folder/# ls -la
total 16
total 8
drwxr-xr-x 3 root root 4096 Feb 27 07:39 .
drwx------ 1 root root 4096 Feb 27 07:39 ..
drwxrwxrwt 3 root root 120 Feb 27 07:38 child_folder
- ddrwxr-xr-x 3 root root 4096 Feb 27 07:39 .
'.' is current folder; which is no_existing_folder created by k8s. Permissions remains same default permission 0755 - drwxrwxrwt 3 root root 120 Feb 27 07:38 child_folder
child_folder created by k8s. but the folder permission is 0777 not default permission0755
/root/non_exiting_folder/ # cd child_folder/
/root/non_exiting_folder/child_folder/ # ls -la
total 4
drwxrwxrwt 3 root root 120 Feb 27 07:38 .
drwxr-xr-x 3 root root 4096 Feb 27 07:39 ..
drwxr-xr-x 2 root root 80 Feb 27 07:38 ..2019_02_27_07_38_59.023449857
lrwxrwxrwx 1 root root 31 Feb 27 07:38 ..data -> ..2019_02_27_07_38_59.023449857
lrwxrwxrwx 1 root root 13 Feb 27 07:38 scret_key_1-> ..data/scret_key_1
lrwxrwxrwx 1 root root 18 Feb 27 07:38 scret_key_2-> ..data/scret_key_2
~/.ssh/test #
Notice that when we removed the subPath config from yaml,
- new folder '..2019_02_27_07_38_59.023449857' is created
- new file 'data' symlinked to that timestamp(..2019_02_27_07_38_59.023449857) folder created
- '..2019_02_27_07_38_59.023449857' hold the mounted data and the files secrete_key_1, secrete_key_2 is symlinked with actual files through ..data(there by timestamp folder) respectively
If we check the file permission
- lrwxrwxrwx 1 root root 31 Feb 27 07:38 ..data -> ..2019_02_27_07_38_59.023449857
**..data ** symlink file have a permission 0777 and is symlinked with ..2019_02_27_07_38_59.023449857 folder. - drwxr-xr-x 2 root root 80 Feb 27 07:38 ..2019_02_27_07_38_59.023449857
but ..2019_02_27_07_38_59.023449857 folder has default permission(no defaultMode) 0755 and it holds actula data. - lrwxrwxrwx 1 root root 13 Feb 27 07:38 scret_key_1-> ..data/scret_key_1
scret_key_1 the secret key file have a permission 0777 and is symlinked with ..data/scret_key_1. As said earlier '..data' is a symlink to timestamp folder '..2019_02_27_07_38_59.023449857'
Checking permission for actual data
~/.ssh/test # cd ..data/
~/.ssh/test/..2019_02_27_07_38_59.023449857 # ls -la
total 8
drwxr-xr-x 2 root root 80 Feb 27 07:38 .
drwxrwxrwt 3 root root 120 Feb 27 07:38 ..
-rw-r--r-- 1 root root 3243 Feb 27 07:38 scret_key_1
-rw-r--r-- 1 root root 422 Feb 27 07:38 scret_key_2
~/.ssh/test/..2019_02_27_07_38_59.023449857 #
- -rw-r--r-- 1 root root 3243 Feb 27 07:38 scret_key_1
The secret key file scret_key_1 has permission specified in defaultMode: 0400 and it holds actula data.
So as i understand if k8s created file or folder automatically(like no_existng_folder in mountPath, no the end file) they will use the default Permission(folder 0755, secretFile 0644).
For the end file like the last part of mountPath, will give 0777(It wont respect defaultMode value unless subPath config is specified), though am not sure why 0777 ??
Alastair Firth
Many others referenced in a wall of text. I think a common confusion is that the files are linked, so ls will show the link permissions by default.
Simply add -L: ls -laL /path/to/directory/ to dereference the link