Permission denied for .git/refs/heads/awx_internal

[Andreas Bourges]

Hi,

...I ran into a really strange problem. I have 6 templates in a project, that are triggered via webhooks - everything working fine with them. Yesterday I added another Project/template that should be run periodically from AWX (no webhook). But whenever I try to launch the template, it tails with the following error:

PermissionError: [Errno 13] Permission denied: '/var/lib/awx/projects/_24__awx_vlan_deployment_061407147_pm/.git/refs/heads/awx_internal'

I tried different EE's - even with the default EE:latest is gives the same error.

Any hint on how to troubleshoot this problem?

Thanks,

Andreas

[AWX Project]

This should be a file on the web container. Can you log into it and check the permissions on the file?

-The AWX Team

[Andreas Bourges]

...seems my mail-response got lost - so I answer via web-interface:

----

Hi,

 

..thanks a lot for your input – web-container looks like this:

 

 

root@net04-mgmt-vm:~# kubectl exec awx-7dd9bf96c7-ml7zf -c awx-web -- /bin/ls -l /var/lib/awx/projects

total 20

drwxr-xr-x 5 awx root 4096 Jul 29 12:01 _10__awx_hello_world

-rwxr-xr-x 1 awx root 0 Jul 29 12:01 _10__awx_hello_world.lock

drwxr-xr-x 6 awx root 4096 Jul 19 08:24 _12__awx_vlan_deployment

-rwxr-xr-x 1 awx root 0 Jul 14 14:00 _12__awx_vlan_deployment.lock

drwxr-xr-x 5 awx root 4096 Jul 27 14:33 _21__device_backup_check

-rwxr-xr-x 1 awx root 0 Jul 27 14:33 _21__device_backup_check.lock

drwxr-xr-x 5 awx root 4096 Jul 27 16:15 _24__awx_vlan_deployment_061407147_pm

-rwxr-xr-x 1 awx root 0 Jul 27 16:14 _24__awx_vlan_deployment_061407147_pm.lock

drwxr-xr-x 4 awx root 4096 Jul 29 12:03 _8__awx_inventory

-rwxr-xr-x 1 awx root 0 Jul 14 14:00 _8__awx_inventory.lock

 

…and the project’s .git is root-owned – which is probably the problem:

 

root@net04-mgmt-vm:~# kubectl exec awx-7dd9bf96c7-ml7zf -c awx-web --  /bin/ls  -la /var/lib/awx/projects/_10__awx_hello_world/.git/refs/heads

total 12

drwxr-xr-x 2 root root 4096 Jul 29 12:20 .

drwxr-xr-x 5 root root 4096 Jul 29 12:01 ..

-rw-r--r-- 1 root root   41 Jul 29 12:01 main

 

However, I’m not allowed to change the ownership within the container:

 

root@net04-mgmt-vm:~# kubectl exec awx-7dd9bf96c7-ml7zf -c awx-web --  /bin/chown -R awx.root /var/lib/awx/projects/_10__awx_hello_world

chown: changing ownership of '/var/lib/awx/projects/_10__awx_hello_world/.git/config': Operation not permitted

chown: changing ownership of '/var/lib/awx/projects/_10__awx_hello_world/.git/logs/refs/remotes/origin/HEAD': Operation not permitted

chown: changing ownership of '/var/lib/awx/projects/_10__awx_hello_world/.git/logs/refs/remotes/origin': Operation not permitted

chown: changing ownership of '/var/lib/awx/projects/_10__awx_hello_world/.git/logs/refs/remotes': Operation not permitted

chown: changing ownership of '/var/lib/awx/projects/_10__awx_hello_world/.git/logs/refs/heads/main': Operation not permitted

 

I would suspect, that the git checkout is performed using the wrong (root) user – is there a place, where this can be configured?

 

Thanks,

 

Andreas

[AWX Project]

The awx user should be in the root group so, from a quick look, that should actually be ok as the files/dirs look rx by the root group. 

Can you confirm that the awx user is in the root group in both the web and task containers?
Also, are you using the projects_persistence setting?

-The AWX Team

[Andreas Bourges]

Hi,

...web-container - no:

root@net04-mgmt-vm:~# kubectl exec   awx-7dd9bf96c7-ml7zf -c awx-web --  cat /etc/group
root:x:0:
bin:x:1:
daemon:x:2:
sys:x:3:
[...]

task-container - no:

root@net04-mgmt-vm:~# kubectl exec   awx-7dd9bf96c7-ml7zf -c awx-task --  cat /etc/group
root:x:0:
bin:x:1:
daemon:x:2:
sys:x:3:

[...]

and yes - project_persistence is true:

root@net04-mgmt-vm:~/awx-operator# grep projects_persistence   awx-deploy.yml
  projects_persistence: true
root@net04-mgmt-vm:~/awx-operator#

Thanks,

andreas

[Andreas Bourges]

Hi,

...just double-checked and I can tell that the awx-user *is* in the root-group (sorry for the wrong information above):

root@net04-mgmt-vm:~/awx-operator# kubectl exec awx-7dd9bf96c7-rn25c -c awx-web -- /bin/id
uid=1000(awx) gid=0(root) groups=0(root),1000
root@net04-mgmt-vm:~/awx-operator# kubectl exec awx-7dd9bf96c7-rn25c -c awx-task -- /bin/id
uid=1000(awx) gid=0(root) groups=0(root),1000
root@net04-mgmt-vm:~/awx-operator#

but when looking at the directories, they're not group-writable:

root@net04-mgmt-vm:~/awx-operator# kubectl exec awx-7dd9bf96c7-rn25c -c awx-web -- /bin/ls -ld /var/lib/awx/projects/_21__device_backup_check/.git/refs/heads/
drwxr-xr-x 2 root root 4096 Jul 27 14:33 /var/lib/awx/projects/_21__device_backup_check/.git/refs/heads/
root@net04-mgmt-vm:~/awx-operator#

And that's why the "mkdir()" fails:

File "/usr/lib64/python3.9/os.py", line 225, in makedirs

  mkdir(name, mode)

PermissionError: [Errno 13] Permission denied: '/var/lib/awx/projects/_24__awx_vlan_deployment_061407147_pm/.git/refs/heads/awx_internal'

Thanks,

Andreas

[AWX Project]

This problem sounds similar to this opened issue https://github.com/ansible/awx-operator/issues/931

If not, this might be a good candidate to open a new issue for

AWX Team

[andyb...@googlemail.com]

--
You received this message because you are subscribed to a topic in the Google Groups "AWX Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/awx-project/eiPsf9dkfB4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/2b483842-9a40-46e7-92ae-f1750f7b3f81n%40googlegroups.com.

[Andreas Bourges]

OK - just verified. It boils down to the control-plane EE - if I use the default from  quay.io/ansible/awx-ee:latest, everything works fine. Have to have a look at the custom image - which should be based on the quay.io image :-(