Cert Manager Secret Updates Issue

[tudor33sud]

Hello,

I am using cert-manager bot on AKS. My service principal key has expired, and cert manager is logging errors that token cannot be refreshed in order to update Azure DNS configs. 

I updated the secret referenced in the cluster-issuer.yml file: 

              # Secret with the password

              clientSecretSecretRef:

                key: password

                name: azuredns-config-password

The problem is that after I updated the secret, the value doesn't seem to get propagated. I tried deleting all pods so that they are re-created, but still cert manager logs the same output: 

E0730 06:07:53.106444       1 sync.go:282] cert-manager/challenges/finalizer "msg"="error cleaning up challenge" "error"="azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/7a8642d9-df34-4d04-a7c6-70cba429463c/resourceGroups/MC_rtdose-dev_rtdose-dev_eastus/providers/Microsoft.Network/dnsZones/e2e85f1f6db34acab769.eastus.aksapp.io/TXT/_acme-challenge.auth?api-version=2017-10-01: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {\"error\":\"invalid_client\",\"error_description\":\"AADSTS7000222: The provided client secret keys for app 'b7d221d8-8cce-4315-8f28-c5b2b0d879c7' are expired. Visit the Azure portal to create new keys for your app: https://aka.ms/NewClientSecret, or consider using certificate credentials for added security: https://aka.ms/certCreds. Trace ID: d5688bc0-d0ab-4326-a920-8b014d090800 Correlation ID: 70499d09-c3db-4d7e-a5f8-fa9d59c85450 Timestamp: 2024-07-30 06:07:52Z\",\"error_codes\":[7000222],\"timestamp\":\"2024-07-30 06:07:52Z\",\"trace_id\":\"d5688bc0-d0ab-4326-a920-8b014d090800\",\"correlation_id\":\"70499d09-c3db-4d7e-a5f8-fa9d59c85450\",\"error_uri\":\"https://login.microsoftonline.com/error?code=7000222\"} Endpoint https://login.microsoftonline.com/cfd26b50-fb8f-44cf-87b2-d5df3d15d884/oauth2/token?api-version=1.0" "dnsName"="auth.e2e85f1f6db34acab769.eastus.aksapp.io" "resource_kind"="Challenge" "resource_name"="wildcard-gcctj-291083326-3914469931" "resource_namespace"="rtdose-dev" "resource_version"="v1" "type"="DNS-01" 

Does anyone have any idea on how to restart / rollout changes so that the new secret gets propagated? I couldn't find any information on this. I assumed that just by modifying the secret value and restart cert manager the new value would be picked up.

Thanks a lot in advance for your help! 

[tudor33sud]

Labels:            app=cert-manager
                   app.kubernetes.io/component=controller
                   app.kubernetes.io/instance=cert-manager
                   app.kubernetes.io/name=cert-manager
                   app.kubernetes.io/version=v1.11.0 

Here's a copy of cert manager service labels, to get an idea of the cert-manager version.