awx-on-k3s changing permissions of '/var/lib/awx/projects': Operation not permitted

969 views
Skip to first unread message

HG

unread,
Sep 19, 2022, 8:37:44 AM9/19/22
to AWX Project
Hello all,
This time with version 0.29 of the operator and 0.29 awx-on-k3s.

AWX does not startup correctly:
awx           awx-operator-controller-manager-5cb6fc8f9-x2vn6   2/2     Running                 0             19m
awx           awx-postgres-13-0                                 1/1     Running                 0             13m
awx           awx-84f598c555-lvbss                              0/4     Init:CrashLoopBackOff   7 (58s ago)   13m

In the  awx_init logfile the following error appears:
2022-09-19T14:04:08.60017162+02:00 stderr F chmod: changing permissions of '/var/lib/awx/projects': Operation not permitted
2022-09-19T14:04:08.602818388+02:00 stderr F chgrp: changing group of '/var/lib/awx/projects': Operation not permitted

I have specified the following however for the project pv.
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: awx-projects-volume
spec:
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  capacity:
    storage: 2Gi
  storageClassName: awx-projects-volume
  hostPath:
    path: /u01/data/projects

Any clues?

m.ne...@cityscoot.eu

unread,
Sep 19, 2022, 3:41:57 PM9/19/22
to awx-p...@googlegroups.com
Hi,
I have the same experience along with some other members of the community.
The issue seems to be only on K3s https://github.com/ansible/awx-operator/issues/1055

Regards,
Antuelle78
> --
> You received this message because you are subscribed to the Google
> Groups "AWX Project" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to awx-project...@googlegroups.com
> <mailto:awx-project...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/awx-project/b768dda5-6d55-4779-9c14-19aac1d01125n%40googlegroups.com <https://groups.google.com/d/msgid/awx-project/b768dda5-6d55-4779-9c14-19aac1d01125n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>

kurokobo

unread,
Sep 19, 2022, 4:14:01 PM9/19/22
to awx-p...@googlegroups.com
Hi,

Thanks for using my repo.

Ensure the permission of "/u01/data/projects" on your K3s host.
My guide includes "sudo chmod 1000:0 /data/projects" to change permission for the project directory before attempting deployment to make chmod and chgrp possible by init container.

Try "sudo chmod 1000:0 /u01/data/projects" before "kubectl apply -k base".

Regards,

------- Original Message -------
--
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/b768dda5-6d55-4779-9c14-19aac1d01125n%40googlegroups.com.

m.ne...@cityscoot.eu

unread,
Sep 19, 2022, 4:27:13 PM9/19/22
to awx-p...@googlegroups.com
Hi kurokobo,

Thanks for the heads up. I am also looking at your repo.
I plan to write an ansible role to simplify the process.
Something similar for Ubuntu is available here: https://github.com/antuelle78/deploy-awx-k3s-ubuntu

Regards,
Antuelle78
> > <mailto:awx-project...@googlegroups.com>.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/awx-project/b768dda5-6d55-4779-9c14-19aac1d01125n%40googlegroups.com <https://groups.google.com/d/msgid/awx-project/b768dda5-6d55-4779-9c14-19aac1d01125n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "AWX Project" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to awx-project...@googlegroups.com
> <mailto:awx-project...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/awx-project/pX6VS-KYQlBs5ddfdjXzfCFliDWRcrGiqp6Q7RXYJSFOySGCORb90Ft5a4V5iPpsg1egf7Fbe8-wWST4d9XybxsYivnM52GIGjOLX3Gk5jI%3D%40protonmail.com <https://groups.google.com/d/msgid/awx-project/pX6VS-KYQlBs5ddfdjXzfCFliDWRcrGiqp6Q7RXYJSFOySGCORb90Ft5a4V5iPpsg1egf7Fbe8-wWST4d9XybxsYivnM52GIGjOLX3Gk5jI%3D%40protonmail.com?utm_medium=email&utm_source=footer>.
>

kurokobo

unread,
Sep 19, 2022, 5:19:12 PM9/19/22
to awx-p...@googlegroups.com
Hi Antuelle78,

Thanks for your interest in my repo :)
My workaround in my previous mail maybe not work for you since it is for HG and your situation seems to be a bit differ from HG's one.

Will reply on the issue ansible/awx-operator#1055

Regards,

------- Original Message -------
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/awx-project/pX6VS-KYQlBs5ddfdjXzfCFliDWRcrGiqp6Q7RXYJSFOySGCORb90Ft5a4V5iPpsg1egf7Fbe8-wWST4d9XybxsYivnM52GIGjOLX3Gk5jI%3D%40protonmail.com https://groups.google.com/d/msgid/awx-project/pX6VS-KYQlBs5ddfdjXzfCFliDWRcrGiqp6Q7RXYJSFOySGCORb90Ft5a4V5iPpsg1egf7Fbe8-wWST4d9XybxsYivnM52GIGjOLX3Gk5jI%3D%40protonmail.com?utm_medium=email&utm_source=footer.
>
>
> --
> You received this message because you are subscribed to the Google Groups "AWX Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/e75fe300-ce84-146b-c069-c4c273bbbf2f%40cityscoot.eu.

HG

unread,
Sep 20, 2022, 3:59:54 AM9/20/22
to awx-p...@googlegroups.com
Hi,

This issue does not appear with awx-operator 0.25 and awx-on-k3s 0.25.

But when I use 0.29 it fails.

awx-84f598c555-c4hn6                               0/4     Pending             0          0s
awx-84f598c555-c4hn6                               0/4     Pending             0          0s
awx-84f598c555-c4hn6                               0/4     Init:0/1            0          1s
awx-84f598c555-c4hn6                               0/4     Init:0/1            0          77s
awx-84f598c555-c4hn6                               0/4     Init:Error          0          79s
awx-84f598c555-c4hn6                               0/4     Init:Error          1 (3s ago)   80s
awx-84f598c555-c4hn6                               0/4     Init:CrashLoopBackOff   1 (12s ago)   91s

/u01/data/projects has:
 drwxrwxrwx. 2 root root  6 Sep 20 09:44 projects

The sudo chmod and chgrp are in both version:

0.29
./roles/installer/templates/deployments/deployment.yaml.j2:              chmod 775 /var/lib/awx/projects
./roles/installer/templates/deployments/deployment.yaml.j2:              chgrp 1000 /var/lib/awx/projects
 0.25
./roles/installer/templates/deployment.yaml.j2:              chmod 775 /var/lib/awx/projects
./roles/installer/templates/deployment.yaml.j2:              chgrp 1000 /var/lib/awx/projects

So it is unclear to me what changed.

Regards Hans


Op ma 19 sep. 2022 om 22:14 schreef 'kurokobo' via AWX Project <awx-p...@googlegroups.com>:
You received this message because you are subscribed to a topic in the Google Groups "AWX Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/awx-project/Wm917iZa_Tc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/pX6VS-KYQlBs5ddfdjXzfCFliDWRcrGiqp6Q7RXYJSFOySGCORb90Ft5a4V5iPpsg1egf7Fbe8-wWST4d9XybxsYivnM52GIGjOLX3Gk5jI%3D%40protonmail.com.

HG

unread,
Sep 20, 2022, 4:16:43 AM9/20/22
to AWX Project
The whole install starts failing with version 0.26 (awx-operator awx-on-k3s)

Op dinsdag 20 september 2022 om 09:59:54 UTC+2 schreef HG:

kurokobo

unread,
Sep 20, 2022, 6:58:57 AM9/20/22
to awx-p...@googlegroups.com
Hi HG,

> This issue does not appear with awx-operator 0.25 and awx-on-k3s 0.25.

Yes, correct. This issue came with 0.29.0.
In short, in 0.28.0 or earlier, chmod/chgrp are invoked by root, but in 0.29.0, chmod/chgrp are invoked by UID:1000.
So if your /u01/data/projects is owned by root:root, UID:1000 can't be invoke chmod/chgrp and this causes "operation not permitted".

For 0.29.0, sadly, it seems that allowing UID:1000 to chmod/chgrp for /var/lib/awx/projects _before_ deployment of awx pod by changing owner/perm for the actual directory on your K3s host is required in some situation.
This can be achieved by following workaround in my previous mail.

> Try "sudo chmod 1000:0 /u01/data/projects" before "kubectl apply -k base".

I think this issue should be fixed in upstream, but in 0.29.0, this workaround is required anyway.

Regards,

------- Original Message -------

m.ne...@cityscoot.eu

unread,
Sep 20, 2022, 7:45:18 AM9/20/22
to awx-p...@googlegroups.com
Hi kurokobo,

Your work around works as expected.
I tested with molecule using vagrant platform on Ubuntu 20.04/22.04 and CentOS Stream 8.

K3s: v1.25.0+k3s1
Operator: 0.29.0
AWX: 21.6.0

I even test upgrading from:

K3s: v1.21.9+k3s1
Operator: 0.28.0
AWX: 21.5.0

And had no issues.

The code is available here: https://github.com/antuelle78/awx-install-on-k3s

Regards,
Antuelle78



On 20/09/2022 12:58, "'kurokobo' via AWX Project" <awx-p...@googlegroups.com> wrote:
> Hi HG,
>
> > This issue does not appear with awx-operator 0.25 and awx-on-k3s 0.25.
>
> Yes, correct. This issue came with 0.29.0.
> For technical details, refer my comment on
> https://github.com/ansible/awx-operator/issues/1055#issuecomment-1251613435 <https://github.com/ansible/awx-operator/issues/1055#issuecomment-1251613435>
> > *0.29*
> > ./roles/installer/templates/deployments/deployment.yaml.j2: chmod 775
> > /var/lib/awx/projects
> > ./roles/installer/templates/deployments/deployment.yaml.j2: chgrp 1000
> > /var/lib/awx/projects
> > *0.25*
> > ./roles/installer/templates/deployment.yaml.j2: chmod 775
> > /var/lib/awx/projects
> > ./roles/installer/templates/deployment.yaml.j2: chgrp 1000
> > /var/lib/awx/projects
> >
> > So it is unclear to me what changed.
> >
> > Regards Hans
> >
> >
> > Op ma 19 sep. 2022 om 22:14 schreef 'kurokobo' via AWX Project
> > <awx-p...@googlegroups.com <mailto:awx-p...@googlegroups.com>>:
> >
> > Hi,
> >
> > Thanks for using my repo.
> >
> > Ensure the permission of "/u01/data/projects" on your K3s host.
> > My guide includes "sudo chmod 1000:0 /data/projects" to change
> > permission for the project directory before attempting deployment
> > to make chmod and chgrp possible by init container.
> >
> > Try "sudo chmod 1000:0 /u01/data/projects" before "kubectl apply
> > -k base".
> >
> > Regards,
> >
> > ------- Original Message -------
> > On Monday, September 19th, 2022 at 21:37, HG
> > <hanspet...@gmail.com <mailto:hanspet...@gmail.com>> wrote:
> >
> >> Hello all,
> >> I am following https://github.com/kurokobo/awx-on-k3s
> >> <https://github.com/kurokobo/awx-on-k3s>.
> >> <mailto:awx-project...@googlegroups.com>.
> >> To view this discussion on the web visit
> >> https://groups.google.com/d/msgid/awx-project/b768dda5-6d55-4779-9c14-19aac1d01125n%40googlegroups.com <https://groups.google.com/d/msgid/awx-project/b768dda5-6d55-4779-9c14-19aac1d01125n%40googlegroups.com?utm_medium=email&utm_source=footer>.
> >
> > --
> > You received this message because you are subscribed to a topic in
> > the Google Groups "AWX Project" group.
> > To unsubscribe from this topic, visit
> > https://groups.google.com/d/topic/awx-project/Wm917iZa_Tc/unsubscribe <https://groups.google.com/d/topic/awx-project/Wm917iZa_Tc/unsubscribe>.
> > To unsubscribe from this group and all its topics, send an email
> > to awx-project...@googlegroups.com
> > <mailto:awx-project...@googlegroups.com>.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/awx-project/pX6VS-KYQlBs5ddfdjXzfCFliDWRcrGiqp6Q7RXYJSFOySGCORb90Ft5a4V5iPpsg1egf7Fbe8-wWST4d9XybxsYivnM52GIGjOLX3Gk5jI%3D%40protonmail.com <https://groups.google.com/d/msgid/awx-project/pX6VS-KYQlBs5ddfdjXzfCFliDWRcrGiqp6Q7RXYJSFOySGCORb90Ft5a4V5iPpsg1egf7Fbe8-wWST4d9XybxsYivnM52GIGjOLX3Gk5jI%3D%40protonmail.com?utm_medium=email&utm_source=footer>.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "AWX Project" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to awx-project...@googlegroups.com
> > <mailto:awx-project...@googlegroups.com>.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/awx-project/CAM7K%2BruKfroO-FcXN88ksD0VHhqxh_bK6-60RDmqtOVQZw9m%3DQ%40mail.gmail.com <https://groups.google.com/d/msgid/awx-project/CAM7K%2BruKfroO-FcXN88ksD0VHhqxh_bK6-60RDmqtOVQZw9m%3DQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "AWX Project" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to awx-project...@googlegroups.com
> <mailto:awx-project...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/awx-project/SKs26hZQnZV2SFKggMDHbMLWNkIWO_3WiBcEJH6jCvl5IV4inTOnrihiMU-Bq3_aDwRgh2j3gc7Erntb8AAz4CE9E7e7dxl13NGmgDODyoQ%3D%40protonmail.com <https://groups.google.com/d/msgid/awx-project/SKs26hZQnZV2SFKggMDHbMLWNkIWO_3WiBcEJH6jCvl5IV4inTOnrihiMU-Bq3_aDwRgh2j3gc7Erntb8AAz4CE9E7e7dxl13NGmgDODyoQ%3D%40protonmail.com?utm_medium=email&utm_source=footer>.
>

HG

unread,
Sep 20, 2022, 8:00:27 AM9/20/22
to AWX Project
So the userid 1000 = runner at the host level should match the userid 1000 inside the containers.


Op dinsdag 20 september 2022 om 12:58:57 UTC+2 schreef kuro...@protonmail.com:
Reply all
Reply to author
Forward
0 new messages