Custom Execution Environment image from a registry w/ custom CA-signed HTTPS certiticate

[Vincent Rubiolo]

Greeting everyone,

I am attempting to reconfigure our AWX instance so that it can pull our custom Execution Environment image from our internal OCI registry instead of RH Quay.

This registry (oci-registry.lorient.iot) uses an HTTPS certificate signed by our custom authority so all my jobs currently fail with the following in the automation-job-XXX pod events:
Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  4s    default-scheduler  Successfully assigned awx-vincent/automation-job-5857-h4svm to awx
  Normal   Pulling    4s    kubelet            Pulling image "oci-registry.lorient.iot/awx/redpesk-ee"
  Warning  Failed     4s    kubelet            Error: ErrImagePull
  Warning  Failed     4s    kubelet            Failed to pull image "oci-registry.lorient.iot/awx/redpesk-ee": rpc error: code = Unknown desc = failed to pull and unpack image "oci-registry.lorient.iot/awx/redpesk-ee:latest": failed to resolve reference "oci-registry.lorient.iot/awx/redpesk-ee:latest": failed to do request: Head "https://oci-registry.lorient.iot/v2/awx/redpesk-ee/manifests/latest": x509: certificate signed by unknown authority
  Normal   BackOff    3s    kubelet            Back-off pulling image "oci-registry.lorient.iot/awx/redpesk-ee"
  Warning  Failed     3s    kubelet            Error: ImagePullBackOff
 
I have followed the documentation[1] to add our CA certificate to the pods environment and can see it is correctly present there. For instance, in the awx-ee pod, I have the associated new mounts:
[...]
    Mounts:
      /etc/pki/ca-trust/extracted from ca-trust-extracted (rw)
      /etc/pki/ca-trust/source/anchors/bundle-ca.crt from awx-bundle-cacert (ro,path="bundle-ca.crt")
     
Our CA certificate also appears to be taken into account:
(vrubiolo)$ k exec awx-7b98b865-xdmb7 -ti -c awx-ee -- /bin/bash
bash-4.4$ trust list |head
pkcs11:id=%71%30%BF%E2%FE%C3%DC%7C%9B%C2%39%E1%7F%44%71%B4%71%11%CD%51;type=cert
    type: certificate
    label: IoT.bzh_Internal_Certification_Authority
    trust: anchor
    category: authority
[...]

From what I see here, the issue is that the certificate needs to be present in the K8S cluster environment (where the image pull occurs) and not in one of the AWX containers.

Could sb shed more light on the process of using an EE from a custom registry whose certificate is signed by a custom CA?

Thanks for your help,

Vincent

[1] https://github.com/ansible/awx-operator#trusting-a-custom-certificate-authority

[Vincent Rubiolo]

Hi again,

Would someone have more insight on this issue?

Based on my research, it looks like the issue is the unability to specify custom CA certificates for the registry when one creates registry credentials in AWX (this is reflected in the UI). When adding such credentials, the associated imagePullSecret is created but the certificates are not added to the system.

I have found https://github.com/ansible/awx/issues/10252 which is exactly about this issue.

I am currently deploying over K3S so this can be managed via custom registry settings in K3S itself which I am going to look at: https://rancher.com/docs/k3s/latest/en/installation/private-registry/#with-tls

I was wondering what was the best practice on the AWX side for you this until the abovementioned ticket is addressed.

Thanks for your help,

Vincent