account console partially loads

[Stephen Sill II]

I've got 4 deployments of keycloak 24.0.4:

1. locally in minikube

2. in a pre-staging namespace in a k8s cluster(GCP)

3. in a staging namespace in a k8s cluster(different cluster)(GCP)

4. in a prod namespace in a k8s cluster(same cluster as 3)(GCP)

Auth and admin console access work great in all of these.

In the two applications I maintain that auth against keycloak I provide a "Manage Account" button that opens the following url in a new browser tab, with hostname and realm appropriate for the environment:

https://KEYCLOAKHOSTNAME/realms/REALM/account

On keycloak deployments 1 and 4 this works great. on 2 and 4, I get the wierdest problem.  The page partially loads, i.e. the sidenav, and topnav are fine, but where the user profile info(first and last name, etc) it has a spinner as if it's loading something that just stays there forever.

Opening inspector in browser(tried multiple browsers) I see that the difference between one that works and one that doesn't seems to be a call to this url:
https://KEYCLOAKHOSTNAME/realms/REALM/account/?userProfileMetadata=true

This call fails with a 403.   I did a side by side comparison of the deployment/ingress/realm/client configurations between the 2 that work and the 2 that don't and I'm really at a loss.

These have worked before.  I recently updated to 24.0.4 from 24.0.2 but I don't remember if I specifically tested that page with 24.0.2 in anything other than my minikube deployed 24.0.2.

Any thoughts about what might cause this behavior?  I've been deploying keycloak releases since v17 so I've exhausted all the normal things like reading the release notes etc.

Thanks