Need help with KIALI OpenId with RBAC on AKS clusters

19 views
Skip to first unread message

Kulbhushan Azad

unread,
Mar 30, 2021, 3:48:38 PMMar 30
to kiali-dev
hi
i am facing issue while trying to setup KIALI with RBAC enabled using OpenID on AKS clusters.
Looking for any help or suggestion to get the RBAC working with openID

below is  my openId Configuration


spec:
  auth:
    openid:
      client_id: ****
      disable_rbac: false
      scopes:
      - openid
      - profile
      - email
      username_claim: preferred_username
    password: secret:kiali:oidc-secret
    strategy: openid

kiali pod logs:
2021-03-30T18:28:13Z INF Starting Metrics Server on [:9090]
2021-03-30T18:32:46Z INF Not handling OpenId code flow authentication: No nonce code present. Login window timed out.

2021-03-30T18:33:29Z DBG Rest perf config QPS: 175.000000 Burst: 200
2021-03-30T18:33:29Z TRC [Kiali Cache] cacheIstioTypes map[AuthorizationPolicy:true DestinationRule:true Gateway:true PeerAuthentication:true RequestAuthentication:true ServiceEntry:true Sidecar:true VirtualService:true]
2021-03-30T18:33:29Z INF Kiali Cache is active for namespaces [.*]
2021-03-30T18:33:29Z DBG Rest perf config QPS: 175.000000 Burst: 200
2021-03-30T18:33:29Z INF [Prom Cache] Enabled

on web page: after succesful authentication with IDP , page gets redirected to kiali 
{"error":"Token is not valid or is expired","detail":"Unauthorized"}

Marc Boorshtein

unread,
Mar 30, 2021, 3:55:09 PMMar 30
to Kulbhushan Azad, kiali-dev

i am facing issue while trying to setup KIALI with RBAC enabled using OpenID on AKS clusters.
Looking for any help or suggestion to get the RBAC working with openID


I don't think this will work because you can't generate an OIDC token that works with your cluster.  I think you'll need to use impersonation instead.
 

Kulbhushan Azad

unread,
Mar 30, 2021, 4:10:36 PMMar 30
to kiali-dev
ok so you are saying i can't use RBAC with AKS using openID approach ?
fyi i am able do get the auhtentication working fine and get the user claims etc without RBAC (disable_rbac: true )
But i have a strict need to have RBAC.

also , how can i  configure impersonation approach that you are suggesting ? any documentation/reference would really help

Edgar Hernandez Garcia

unread,
Mar 30, 2021, 11:56:24 PMMar 30
to Kulbhushan Azad, kiali-dev
Hi, Kulbhushan!

It's a known issue that RBAC is not working with Kiali and AKS, even if AzureAD integration is turned on. It's mentioned in a note at the end of the "Requirements" section on Kiali's OpenID docs: https://kiali.io/documentation/latest/configuration/authentication/openid/#_requirements. As mentioned in this page, the current workaround is to use kube-oidc-proxy (or something similar) to make mappings of AzureAD users/groups with regular Kubernetes ServiceAccounts.

it was mentioned that there is a chance that authentication with AzureAD may succeed by using the "accees_token" (Kiali is currently using the "id_token"). This goes in line with what is mentioned in step 4 of this Azure docs: https://docs.microsoft.com/en-us/azure/aks/concepts-identity#webhook-and-api-server. However, since I don't have an Azure account, I cannot try this.

I saw your messages on Slack -- sorry for not replying, I was doing some errands.
If you have some time to try the "access_token" approach, I can prepare a modified Kiali with some hard coded changes and see if authentication succeeds on Azure with RBAC turned on.
We may need several trials to find how to do the right communication with AzureAD/AKS... or... you know.... also iterate because of coding bugs.
If we can confirm that the "access_token" works, I will do the permanent changes that can be made available in the next Kiali release. I will need you to try the permanent changes.

Does this work for you? If so, I'll do the changes and let you know when you can try.


The "Impersonation" that Marc is suggesting is another alternative. In this case, you use Kiali's Header authentication to handle authentication externally by using a reverse proxy (see https://kiali.io/documentation/latest/configuration/authentication/header/) and inject impersonation headers. This probably can be achieved with oauth2_proxy, although I'm not sure how.


--
You received this message because you are subscribed to the Google Groups "kiali-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kiali-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kiali-dev/5e2290de-74d9-4bb4-a373-0e4004d00980n%40googlegroups.com.

Kulbhushan Azad

unread,
Mar 31, 2021, 12:48:27 AMMar 31
to Edgar Hernandez Garcia, kiali-dev
Thanks Edgar for the response
Yes i can try the "access_token" approach.
Pls let me know how and where to pick the changes.
--
Thanks
Kulbhusan

Edgar Hernandez Garcia

unread,
Mar 31, 2021, 2:38:02 PMMar 31
to Kulbhushan Azad, kiali-dev

Hi, again

Check if you can use this image: quay.io/edgarhz/kiali:azure-poc-20210331-1223

You can see the code changes here: https://github.com/kiali/kiali/pull/3838
Notice there are some log messages echoing the tokens (which are sensitive).

If login doesn't work, I'll appreciate if you provide the log messages (of course, with the sensitive data removed).
If you haven't done so already, I think you should follow this guide to enable AKS+AAD integration: https://docs.microsoft.com/en-us/azure/aks/managed-aad

There are some holidays this week. If I don't reply soon, I surely should reply on monday.
I wait for your feedback!

- Edgar

Kulbhushan Azad

unread,
Mar 31, 2021, 3:05:58 PMMar 31
to Edgar Hernandez Garcia, kiali-dev
ok Thanks
i will try and let you know the result
--
Thanks
Kulbhusan

Kulbhushan Azad

unread,
Mar 31, 2021, 5:46:30 PMMar 31
to Edgar Hernandez Garcia, kiali-dev
Below is


auth:
  openid:
    authentication_timeout: 300
    client_id: xxx
    client_secret: xxx
    issuer_uri: https://login.microsoftonline.com/xxx/v2.0

    scopes:
    - openid
    - profile
    - email
    username_claim: preferred_username



2021-03-31T19:25:55Z INF Not handling OpenId code flow authentication: No nonce code present. Login window timed out.
2021-03-31T19:26:08Z ERR This is the raw token response: {"token_type":"Bearer","scope":"openid profile email Directory.Read.All Group.Read.All","expires_in":3599,"ext_expires_in":3599,"access_token":"xxx"}
2021-03-31T19:26:08Z ERR Parsed response: IdToken = xxx
2021-03-31T19:26:08Z ERR Parsed response: AccessToken = xxx
2021-03-31T19:26:08Z DBG Rest perf config QPS: 175.000000 Burst: 200
2021-03-31T19:26:08Z TRC [Kiali Cache] cacheIstioTypes map[AuthorizationPolicy:true DestinationRule:true Gateway:true PeerAuthentication:true RequestAuthentication:true ServiceEntry:true Sidecar:true VirtualService:true]
2021-03-31T19:26:08Z INF Kiali Cache is active for namespaces [.*]
2021-03-31T19:26:08Z DBG Rest perf config QPS: 175.000000 Burst: 200
2021-03-31T19:26:08Z INF [Prom Cache] Enabled
2021-03-31T19:26:09Z ERR The cluster is rejecting the IdToken. Trying the AccessToken...
2021-03-31T19:26:09Z ERR Original httpStatus = 401
2021-03-31T19:26:09Z ERR Original errMsg = Token is not valid or is expired
2021-03-31T19:26:09Z ERR Original detailedError = Unauthorized
2021-03-31T19:26:09Z DBG Rest perf config QPS: 175.000000 Burst: 200
2021-03-31T19:26:09Z ERR The cluster is rejecting the AccessToken. Nothing else to do...

the AKS cluster is AD enabled
image.png


and i do have the cluster admin role to the cluster
image.png


So what could be the next thing to try or look ?
--
Thanks
Kulbhusan

Edgar Hernandez Garcia

unread,
Mar 31, 2021, 6:22:30 PMMar 31
to Kulbhushan Azad, kiali-dev
What's the message you are getting in the browser?

Is it the same bare "Token is not valid or is expired" / "Unauthorized"? or does it have more details about why it's not working?

Kulbhushan Azad

unread,
Mar 31, 2021, 8:04:43 PMMar 31
to Edgar Hernandez Garcia, kiali-dev
same as before

{"error":"Token is not valid or is expired","detail":"Unauthorized"}
--
Thanks
Kulbhusan

Edgar Hernandez Garcia

unread,
Apr 5, 2021, 12:07:26 PMApr 5
to Kulbhushan Azad, kiali-dev

Hi again!

Can you, please, check your "App Registration"?

I have read this small guide for App Registrations in AzureAD: https://docs.pivotal.io/p-identity/1-11/azure-oidc/config-azure.html#register-an-app
Also, I'm aware that when you enable AzureAD integration in AKS, an "App" is created automatically. This is what I understand from this comment of another user using AKS: https://github.com/kiali/kiali/issues/3294#issuecomment-704500897

A common mistake is that people tend to create a new "App" to use for Kiali. The "Token is not valid or is expired" error is caused by this very common misconfiguration.
However, the right configuration is to reuse the existing App created for the cluster. In the case of AKS, you should reuse the App that is automatically created when enabling AzureAD integration.

So, please, make sure you are reusing this automatically created App, rather than creating a new one. I want to make sure that the config is right before going further.

Kulbhushan Azad

unread,
Apr 5, 2021, 11:22:55 PMApr 5
to Edgar Hernandez Garcia, kiali-dev
Yes I do have a new app created to be used for Kiali.
I am not sure if Microsoft would share the access and details to the app they implicitly created with the AKS.
Also the issuer uri might be different and/or the callback url they might have different(something standard) unlike what kiali needs as callback

I would consult Microsoft and seek their opinion and guidance too on this.
--
Thanks
Kulbhusan

Edgar Hernandez Garcia

unread,
Apr 6, 2021, 2:08:51 PMApr 6
to Kulbhushan Azad, kiali-dev

I thought that the automatically created app was available in the Azure console. Isn't this the case?
If you cannot see that app in the console and make changes to it, that's definitely a blocker. That would make it impossible to get a token that is valid for the cluster.

Kulbhushan Azad

unread,
Apr 6, 2021, 3:57:50 PMApr 6
to Edgar Hernandez Garcia, kiali-dev
right , i don't have access to those client/server app , i believe they share among multiple tenants 
--
Thanks
Kulbhusan

Kulbhushan Azad

unread,
Apr 7, 2021, 9:32:34 AMApr 7
to Edgar Hernandez Garcia, kiali-dev
hi Edgar, 
some good news, after putting several hours on this i was able to get the right access token which the backend AKS cluster would accept.

so below is the url which i used in local to get the access token which works with the AKS 

https://login.windows.net/<tenant-id>/oauth2/authorize?resource=<resource-id>&client_id=<client-id>&response_type=token&redirect_uri=http://localhost:8080/dumpToken&response_mode=form_post&scope=openid&state=12345&nonce=678910

*Note the resource id is in public domain and shared across multiple tenant 

so if you could help incorporate that above url in kiali while calling the IDP for access token and send a patch then i can test the same.
i guess we can hardcode the resourceid for now and if it works then we can have it configurable in the kiali cr.

Let me know if you need the resourceid





--
Thanks
Kulbhusan

Edgar Hernandez Garcia

unread,
Apr 7, 2021, 11:14:59 AMApr 7
to Kulbhushan Azad, kiali-dev
Hi!

It's very good news.
OK, pass the resource-id and I'll hard code it for testing. It should be a quick change.

You could either reply to the mail-list or reply privately. Since the resource-id is public, I think it doesn't matter what you choose.

Regards!


Reply all
Reply to author
Forward
0 new messages