RBAC Issue
263 views
Skip to first unread message
Peter Macallan
Apr 30, 2021, 4:00:32 AM4/30/21
to Operator Framework
Hi
I'm new to the operator-sdk and I am facing some issues that I think are RBAC related. I've generated my first ansible operator using the sdk 1.6.2.
Once I deployed the operator, it starts working in its namespace "projectname-operator-system". When I check the "manager" container logs, it shows that the roles are being executed. However, they basically all contain the same error:
fatal: [localhost]: FAILED! => {"changed": false, "error": 403, "msg": "Failed to retrieve ↪requested object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\": ↪\"Failure\",\"message\":\"services \\\\\"csservice-sample-svc-ms-sigs-yara\\\\\" is forbid ↪den: User \\\\\"system:serviceaccount:projectname-operator-system:projectname-operator ↪-controller-manager\\\\\" cannot get resource \\\\\"services\\\\\" in API group \\\\\"\\\\ ↪\" in the namespace \\\\\"default\\\\\"\",\"reason\":\"Forbidden\",\"details\":{\"name\":\ ↪"csservice-sample-svc-ms-sigs-yara\",\"kind\":\"services\"},\"code\":403}\\n'", "reason": ↪"Forbidden", "status": 403}
I am not really sure why it has permission issues. It tries to "work" in namespace "default". However, in all my roles, I defined a custom namespace using: namespace: '{{ ansible_operator_meta.namespace }}'. The value of {{ ansible_operator_meta.namespace }} is "projectname-operator-system". Which is where the operator runs. Why does the operator try to work in namespace "default"?
Also, when I try to create the SA manually from the yaml file generated by the operator-sdk: cat config/rbac/service_account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: controller-manager
namespace: system
I get the following error: kubectl apply -f config/rbac/service_account.yaml
Error from server (NotFound): error when creating "config/rbac/service_account.yaml": namespaces "system" not found
There is the mentioning of a "system" namespace and does as such not exist on my k8s cluster.
There is the mentioning of a "system" namespace and does as such not exist on my k8s cluster.
It would be great to get some input to put the pieces together.
Thanks!
Camila Macedo
Apr 30, 2021, 6:58:51 AM4/30/21
to Peter Macallan, Operator Framework
Hi Peter,
Assuming that you are working with Golang based operators:
Markers define the RBAC roles, and when you run the `make manifests` then, the specific manifests will be automatically updated. See the Memcached example here[1]. Also, see its explanation in the tutorial[2]. See the markers[3] doc and its specific RBAC section.
By default, when you create a project and you will deploy it, and ns will be created for your project. Then, see that you cannot apply the `kubectl apply -f config/rbac/service_account.yaml ` directly without apply all required manifest. To make easier the process, SDK provides helpers in the makefile, which means that you will run `make install` and `make deploy` instead of doing it all manually.
See the quick-start[5] in SDK docs or in Kubebuilder[6]. Also, I'd like to recommend you follow up on the SDK tutorial kindly. If you did not so far, it will not take too long and will bring a better understanding.
PS.: You can use the Kuebuilder docs as well. Just replace the kubebuilder command instruction with operator-sdk. Both projects are integrated. More info: Can I use the Kubebuilder docs?[7] and What are the differences between Kubebuilder and Operator-SDK?[8]
I hope that the info shared help you out.
Cheers,
--
You received this message because you are subscribed to the Google Groups "Operator Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to operator-framew...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/eae5294d-ba2b-4671-8912-a88aea61f012n%40googlegroups.com.
Peter Macallan
Apr 30, 2021, 7:39:12 AM4/30/21
to Operator Framework
Dear Camila
Many thanks for your answer. I went through all the links, but I still cannot resolve the issue.
I must admin, I kind of expected that I cannot just execute `kubectl apply -f config/rbac/service_account.yaml` just like that. The confirmation helps though.
The ns that will created is following the naming scheme `projectname-operator-system`, right? Will only the operator run in this ns or also the resources to be deployed by the operator?
I followed the quickstart tutorial and executed the following commands:
- operator-sdk init --plugins=ansible --domain=example.com
- operator-sdk create api --group app --version v1 --kind CSService
- make docker-build docker-push
- make install
- make deploy
- and during troubleshooting as well:
- kubectl apply -f config/samples/app_v1_csservice.yaml
The manager container then shows the following errors:
{"level":"error","ts":1619781853.2686849,"logger":"logging_event_handler","msg":"","name":"csservice-sample","namespace":"default","gvk":"app. ↪example.com/v1, Kind=CSService","event_type":"runner_on_failed","job":"7504504064263669287","EventData.Task":"Create Nginx Sidecar configMap","Ev ↪entData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/cscommon/tasks/main.yml:3","error":"[playbook task failed]","stacktrace": ↪"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/za...@v0.2.0/zapr.go:132\ngithub.com/operator-framework/operator ↪-sdk/internal/ansible/events.loggingEventHandler.Handle\n\t/workspace/internal/ansible/events/log_events.go:94"}
{"level":"error","ts":1619781853.6888945,"logger":"runner","msg":"ansible-playbook 2.9.19\r\n config file = /etc/ansible/ansible.cfg\r\n con ↪figured module search path = ['/usr/share/ansible/openshift']\r\n ansible python module location = /usr/local/lib/python3.8/site-packages/an ↪sible\r\n executable location = /usr/local/bin/ansible-playbook\r\n python version = 3.8.3 (default, Aug 18 2020, 08:56:04) [GCC 8.3.1 2019 ↪1121 (Red Hat 8.3.1-5)]\r\nUsing /etc/ansible/ansible.cfg as config file\r\nSkipping callback 'actionable', as we already have a stdout callb ↪ack.\nSkipping callback 'awx_display', as we already have a stdout callback.\nSkipping callback 'counter_enabled', as we already have a stdou ↪t callback.\nSkipping callback 'debug', as we already have a stdout callback.\nSkipping callback 'dense', as we already have a stdout callbac ↪k.\nSkipping callback 'dense', as we already have a stdout callback.\nSkipping callback 'full_skip', as we already have a stdout callback.\nS ↪kipping callback 'json', as we already have a stdout callback.\nSkipping callback 'minimal', as we already have a stdout callback.\nSkipping ↪callback 'null', as we already have a stdout callback.\nSkipping callback 'oneline', as we already have a stdout callback.\nSkipping callback ↪ 'selective', as we already have a stdout callback.\nSkipping callback 'skippy', as we already have a stdout callback.\nSkipping callback 'st ↪derr', as we already have a stdout callback.\nSkipping callback 'unixy', as we already have a stdout callback.\nSkipping callback 'yaml', as ↪we already have a stdout callback.\n\r\nPLAYBOOK: playbook.yml *********************************************************\n1 plays in /opt/ans ↪ible/playbooks/playbook.yml\n\r\nPLAY [localhost] ***************************************************************\nMETA: ran handlers\n\r\nTA ↪SK [cscommon : Create Nginx Sidecar configMap] *******************************\r\ntask path: /opt/ansible/roles/cscommon/tasks/main.yml:3\nfa ↪tal: [localhost]: FAILED! => {\"changed\": false, \"error\": 403, \"msg\": \"Failed to retrieve requested object: b'{\\\"kind\\\":\\\"Status\ ↪\\",\\\"apiVersion\\\":\\\"v1\\\",\\\"metadata\\\":{},\\\"status\\\":\\\"Failure\\\",\\\"message\\\":\\\"configmaps \\\\\\\\\\\"csservice-sam ↪ple-nginx-sidecar-cfg\\\\\\\\\\\" is forbidden: User \\\\\\\\\\\"system:serviceaccount:projectname-operator-system:projectname-operator-c ↪ontroller-manager\\\\\\\\\\\" cannot get resource \\\\\\\\\\\"configmaps\\\\\\\\\\\" in API group \\\\\\\\\\\"\\\\\\\\\\\" in the namespace \ ↪\\\\\\\\\\"default\\\\\\\\\\\"\\\",\\\"reason\\\":\\\"Forbidden\\\",\\\"details\\\":{\\\"name\\\":\\\"csservice-sample-nginx-sidecar-cfg\\\", ↪\\\"kind\\\":\\\"configmaps\\\"},\\\"code\\\":403}\\\\n'\", \"reason\": \"Forbidden\", \"status\": 403}\n\r\nPLAY RECAP ********************* ↪************************************************\r\nlocalhost : ok=0 changed=0 unreachable=0 failed=1 skipped=0 ↪ rescued=0 ignored=0 \r\n\n","job":"7504504064263669287","name":"csservice-sample","namespace":"default","error":"exit status 2","stac ↪ktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/za...@v0.2.0/zapr.go:132\ngithub.com/operator-framework/ ↪operator-sdk/internal/ansible/runner.(*runner).Run.func1\n\t/workspace/internal/ansible/runner/runner.go:265"}
I can see that the following sa is used: system:serviceaccount:projectname-operator-system:projectname-operator-controller-manager and that it has to permissions on ns default.
However, the sa exists and seems to have a clusterolebinding that should grant access to ns default as well:
kubectl describe clusterrolebindings projectname-operator-manager-rolebind
ing
Name: projectname-operator-manager-rolebinding
Labels: <none>
Annotations: Role:
Kind: ClusterRole
Name: projectname-operator-manager-role
Subjects:
Kind Name Namespace
---- ---- ---------
ServiceAccount projectname-operator-controller-manager projectname-operator-system
Am I missing something else?
Thanks
Camila Macedo
Apr 30, 2021, 8:23:08 AM4/30/21
to Peter Macallan, Operator Framework
Hi Peter,
Following some tips for you are able to check it and I hope that will help you out. If not, could you please raise an issue in the repo providing all asked info in the template and the exact steps you're performing? That can allow it to be better checked.
- The steps shared are not exactly the same in the quick-start. Could you please follow up the quick-start first to see if you have all pre-requirements in place See: https://sdk.operatorframework.io/docs/building-operators/ansible/quickstart/ e.g. Note that you did not create the role or playbook for your operator such as `operator-sdk create api --group cache --version v1alpha1 --kind Memcached --generate-role.
I must admin, I kind of expected that I cannot just execute `kubectl apply -f config/rbac/service_account.yaml` just like that. The confirmation helps though.
- The project uses kustomize, and then, by using the makefile targets, it will edit the required manifest with the project name, etc. (e.g. https://github.com/operator-framework/operator-sdk/blob/master/testdata/ansible/memcached-operator/Makefile#L78-L80). In this way, try to apply directly might not work.
- As pre-requirement for the default/quick-start steps, you need to run the commands with a user authorized with
cluster-adminpermissions because it will use cluster roles. You can follow up the steps: https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control#iam-rolebinding-bootstrap
- Ensure that you are scaffolding the new project with the latest release by running `operator-sdk version`. In the latest release, it has a bug fixed in the scaffolds which might be related to your scenario. See: https://github.com/operator-framework/operator-sdk/releases/tag/v1.6.2
`
Cheers,
To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/5c74cdba-49f4-444c-8d3c-d0daa7b4355cn%40googlegroups.com.
Peter Macallan
Apr 30, 2021, 10:41:41 AM4/30/21
to Operator Framework
Hi Camila
I was able to isolate the issue, but I don't know why this is a problem: my operator works as long as I only define "kind: Deployment" in an ansible task. As soon as I want to define an ansible tasks for configmaps, ingresses, services, etc. it immediately fails saying:
E0430 14:39:31.273599 9 reflector.go:138] pkg/mod/k8s.io/clie...@v0.20.2/tools/cache/reflector.go:167: Failed to watch /v1, Kind=Service: ↪ failed to list /v1, Kind=Service: services is forbidden: User "system:serviceaccount:projectname-operator-system:projectname-operator-contr ↪oller-manager" cannot list resource "services" in API group "" at the cluster scope
No clue why. I will take a break now, it was quiet an extensive debugging session.. :)
thanks
Camila Macedo
Apr 30, 2021, 12:19:37 PM4/30/21
to Peter Macallan, Operator Framework
Hi Peter,
Then you need to update the file config/rbac/role.yaml e.g here[1] to give the permissions for these resources. More info: https://kubernetes.io/docs/reference/access-authn-authz/rbac/.
Cheers,
To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/02807752-973e-4922-8f6e-a6b6d10f3ccan%40googlegroups.com.
Peter Macallan
Apr 30, 2021, 1:47:09 PM4/30/21
to Operator Framework
Yes, that was the issue! There was no problem with the operator config after all :) Many thanks!!
Camila Macedo
Apr 30, 2021, 2:00:44 PM4/30/21
to Peter Macallan, Operator Framework
Hi Peter,
Really thank you for let us know.
I am very happy that all worked out well as expected.
Cheers,
To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/573da6a3-7e60-4302-ace4-669c7eca5bd8n%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages