Re: [kubernetes/kubernetes] libpq-based PostgreSQL clients will no longer read a PGPASSFILE mounted from a secret (#57117)
fejta-bot
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
fejta-bot
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale
k8s-ci-robot
Closed #57117.
fejta-bot
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
David Arnold
David Arnold
/reopen
/remove-lifecycle rotten
k8s-ci-robot
@blaggacao: you can't re-open an issue/PR unless you authored it or you are assigned to it.
In response to this:
/reopen
/remove-lifecycle rotten
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
rata
@blaggacao The link to pqlib is using stat() and not lstat(). See https://github.com/postgres/postgres/blob/b90d97e081415768468295cc4d10d7ee21881964/src/interfaces/libpq/fe-connect.c#L6410
And note that, from the manage, it says:
lstat() is identical to stat(), except that if path is a symbolic link, then the link itself is stat-ed, not the file that it refers to.
So stat should return true to S_ISREG, as it is a regular file what this points to.
I'll try to reproduce and report back anyways :)
rata
@blaggacao @rhymoid I can't reproduce. What am I missing?
I tried creating a pod with this yaml
---
apiVersion: v1
kind: Pod
metadata:
name: rata-pod
spec:
containers:
- name: nginx
image: <some debian stretch base image>
volumeMounts:
- mountPath: /root/
name: foo
readOnly: true
resources:
requests:
cpu: 10m
memory: 32Mi
volumes:
- name: foo
secret:
secretName: rata-test-secret
defaultMode: 256
items:
- key: pgpass
path: .pgpass
And used this file:
127.0.0.1:666:db:rata:test
as pgpass. I run kubectl create secret generic rata-test-secret --from-file=pgpass to create the secret.
Then I run a kubectl apply of the pod and then exec to connect to it. And inside the pod run:
apt-get update
apt-get install postgresql-client-9.6
root@rata-pod:/# psql -h 127.0.0.1
psql: could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
root@rata-pod:/# ls -l ~/.pgpass
lrwxrwxrwx 1 root root 14 Jul 23 20:18 /root/.pgpass -> ..data/.pgpass
root@rata-pod:/# ls -l ~/..data/.pgpass
-r-------- 1 root root 26 Jul 23 20:18 /root/..data/.pgpass
root@rata-pod:/# export PGPASSFILE=/root/.pgpass
root@rata-pod:/# psql -h 127.0.0.1
psql: could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
I tried to export the env var as it was described in the original bug report. But the expected behavior still seems to happen.
Also, to be sure this was being used correctly I tried two more things.
First, also run this in the pod:
root@rata-pod:~# cp /root/.pgpass /tmp/pgpass
root@rata-pod:~# chmod 660 /tmp/pgpass
root@rata-pod:~# export PGPASSFILE=/tmp/pgpass
root@rata-pod:~# psql -h 127.0.0.1
WARNING: password file "/tmp/pgpass" has group or world access; permissions should be u=rw (0600) or less
psql: could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
Now that the file uses the wrong permissions, it says a warning about the permissions when running psql.
The second thing I tried is to delete the pod, modify the yaml with the defaultMode to be 432 (decimal for 660 in octal) and created it again. Then run:
apt-get update
apt-get install postgresql-client-9.6
root@rata-pod:~# psql -h 127.0.0.1
WARNING: password file "/root/.pgpass" has group or world access; permissions should be u=rw (0600) or less
psql: could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
So, when using the wrong permissions the proper warning is thrown. And following the code linked, https://github.com/postgres/postgres/blob/b90d97e081415768468295cc4d10d7ee21881964/src/interfaces/libpq/fe-connect.c#L6413-L6429, this is only thrown when the test to try it is a regular file works okay. And it makes sense it works okay, as it is using stat() and not lstat() as I said in my previous comment (the pointed file is a regular file, and that is what stat() checks). I've also checked and stat() is still being used in master (https://github.com/postgres/postgres/blob/master/src/interfaces/libpq/fe-connect.c#L6410).
So, the only options I can think are:
- Am I missing something?
- Step 5 in the original bug report that says that uses SQLAlchemy is wrong or checking for something
psqlis not. In that case, more details about what was done would be nice (I never used that before :)). Maybe the file was not in the home user that runs the pod? (in my example it was root)
So, I'd rather really understand the issue before trying way different approaches (like hard-links). But my guess is that no radical changes are needed, as it seems to be working with psql as expected. If some patch is needed (can't see why with this info, though), it should be a simple one. I GUESS =)
And too bad I haven't seen this before. But thanks a lot for the mention, as I've added the feature but I'm not on the sigs this bug report was opened to :-)
David Arnold
@rata Thanks! I indeed missed that intricate detail about stat vs lstat - I will dig into that, If my observation was an unexpected side effect and this bug is no, I'd would be more than happy to confirm that. Will have a deeper look this evening...
rata
@blaggacao Thank you. Let me know if the steps I followed are not clear. I'll be waiting for your confirmation =)
rata
@blaggacao could you confirm?
David Arnold
It will be until the weekend I can free up time to review this again, sry 👍
rata
No need to say sorry, you are helping a lot :)
rata
@blaggacao any news on this? I hope it worked :)
rata
Well, I consider this closed. Please let me know if there is a reason to think otherwise
David Arnold
I was blinded by the octal & decimal detail:
In my operator, I specified decimal 400 -> octal 620.
David Arnold
@rata With this line (DefaultMode: func(a int32) *int32 { return &a }(256), // octal 0400), the result is:
odoo@mausi-v10-server-65b4d9f448-sdvxg:/$ ls -la /run/secrets/odoo/pgpass
lrwxrwxrwx 1 root root 13 Sep 27 20:14 /run/secrets/odoo/pgpass -> ..data/pgpass
odoo@mausi-v10-server-65b4d9f448-sdvxg:/$ ls -la /run/secrets/odoo/..data/pgpass
-r--r----- 1 root odoo 184 Sep 27 20:14 /run/secrets/odoo/..data/pgpass
odoo@mausi-v10-server-65b4d9f448-sdvxg:/$ cp /run/secrets/odoo/..data/pgpass /tmp/pgpass
odoo@mausi-v10-server-65b4d9f448-sdvxg:/$ chmod 400 /tmp/pgpass
odoo@mausi-v10-server-65b4d9f448-sdvxg:/$ ls -la /tmp/pgpass
-r-------- 1 odoo odoo 184 Sep 27 20:16 /tmp/pgpass
odoo@mausi-v10-server-65b4d9f448-sdvxg:/$and failing (my startup script):
++ psql -h database postgres -c '\q'
WARNING: password file "/run/secrets/odoo/pgpass" has group or world access; permissions should be u=rw (0600) or less
Password:
WARNING: password file "/run/secrets/odoo/pgpass" has group or world access; permissions should be u=rw (0600) or less
psql: fe_sendauth: no password supplied
++ echo 'Postgres is unavailable - sleeping'
David Arnold
@rata I think the missing bit is that I run postgres as user != 1 and it seems that ownership is not consistent the one of that user, see:
-r--r----- 1 root odoo 184 Sep 27 20:14 /run/secrets/odoo/..data/pgpass
—
David Arnold
I need to use the following security context:
securityContext := &v1.PodSecurityContext{ RunAsUser: func(i int64) *int64 { return &i }(9001), RunAsNonRoot: func(b bool) *bool { return &b }(true), FSGroup: func(i int64) *int64 { return &i }(9001), }
rata
@blaggacao sorry, not sure I follow you. What is that code you are posting? That is not the yaml, right? I'm not sure what do you refer to.
But, I received notifications for comments you edited, that I do follow. And I think the root cause it's there. But, let me know if I'm missing something (I'm probably am :-D).
In your other comments, now not visible, you are missing one important thing, IIUC:
You pasted:
odoo@mausi-v10-server-65b4d9f448-sdvxg:/$ ls -la /run/secrets/odoo/pgpass
lrwxrwxrwx 1 root root 13 Sep 27 20:14 /run/secrets/odoo/pgpass -> ..data/pgpass
odoo@mausi-v10-server-65b4d9f448-sdvxg:/$ ls -la /run/secrets/odoo/..data/pgpass
-r--r----- 1 root odoo 184 Sep 27 20:14 /run/secrets/odoo/..data/pgpass
odoo@mausi-v10-server-65b4d9f448-sdvxg:/$ cp /run/secrets/odoo/..data/pgpass /tmp/pgpass
odoo@mausi-v10-server-65b4d9f448-sdvxg:/$ chmod 400 /tmp/pgpass
odoo@mausi-v10-server-65b4d9f448-sdvxg:/$ ls -la /tmp/pgpass
-r-------- 1 odoo odoo 184 Sep 27 20:16 /tmp/pgpass
odoo@mausi-v10-server-65b4d9f448-sdvxg:/$
++ psql -h database postgres -c '\q'WARNING: password file "/run/secrets/odoo/pgpass" has group or world access; permissions should be u=rw (0600) or less Password: WARNING: password file "/run/secrets/odoo/pgpass" has group or world access; permissions should be u=rw (0600) or less psql: fe_sendauth: no password supplied ++ echo 'Postgres is unavailable - sleeping'
You have group access read access for group in the file /run/secrets/odoo/pgpass, as you pasted here (let me copy it again, to be clear):
odoo@mausi-v10-server-65b4d9f448-sdvxg:/$ ls -la /run/secrets/odoo/..data/pgpass
-r--r----- 1 root odoo 184 Sep 27 20:14 /run/secrets/odoo/..data/pgpass
So, it is complaining for that:
WARNING: password file "/run/secrets/odoo/pgpass" has group or world access
You should change the mode bits so only the owner has access, IIUC. Have you tried using mode 256 (in decimal), as I pasted in my example? I think that, enough, will do the trick (if the psql command can read the file, of course).
And on the other issue you mention, can you please elaborate what is the problem with the atomic writer? I didn't understand :)
David Arnold
@rata Sorry, I tried to condense the issue.
A little context:
- I'm writing an operator
- It's there https://github.com/xoe-labs/Odoo-Operator
- I tried
256it's what currently is the working hypothesis. (see first code snipped in here)
Isn't the problem simply that it's owned by root instead of odoo which is the process incorporation despite of the provided PodSecurityContext?
-r--r----- 1 root odoo doesn't work, but
-r-------- 1 odoo odoo would.
Maybe it's not the atomic writer, but the link you posted. I'll check.
Any clues how to get k8s to construe the second line rather than the first.
It worked in your example, because you didn't use process incorporation, so root root was just fine.
rata
@blaggacao the link shows that if it's group readable it will fail, right?
The thing is:
- If it's group readable, it will fail
- It it's owned by root, only readable to the owner, a process running as
odoouser will not be able to read it
So, definitely not a problem when setting the mode of the file. Just that you want to change the ownership too.
Not sure there is currently any way to change the ownership :-(
As far as I can see in the current docs (https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/#configmapvolumesource-v1-core) there is not.
Maybe some hack, like changing it at startup (custom script or podStart should work, but not sure can be done as configmap should be mounted as not read-only) or using xattr can help meanwhile a proper solution is developed.
It can definitely be something to consider to add a way to set the ownership in some way. But that should be a new KEP, etc.
Would you like to contribute a patch? :)
David Arnold
Would you like to contribute a patch?
I'm coding in go since 3 months. I'm greener behind the ears than fresh salad.
David Arnold
It seems that his is the right issue: #2630
rata
Would you like to contribute a patch?
I'm coding in go since 3 months. I'm greener behind the ears than fresh salad.
Feel free to contribute when you feel ready :)
Maybe some hack, like changing it at startup (custom script or podStart should work, but not sure can be done as configmap should be mounted as not read-only) or using xattr can help meanwhile a proper solution is developed
Could you give one-two extra clues? I'm not following you, here...
Sorry I wasn't clear.
I mean postStart like it is shown here: https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/#define-poststart-and-prestop-handlers
So, maybe you can execute a chown as postStart. Or use xattr command. But you might need to mount the configmap as readOnly: false.
But definitely, IIUC, the issue you are having is that the user that executes can't read the file (and psql doesn't accept files with group readable bits). You the issue you point seems like the proper issue to look at :)
Let me know if I can help!
David Arnold
Finally, I used postStart but putting it into another directory to keep secret read only. Thanks!
rata
> Finally, I used `postStart` but putting it into another directory to keep secret read only. Thanks!