Is it safe to change objects created by GKE?

[Ingo Gottwald]

We would like to change some things on the default GKE setup and the docs don't clarify whether it is safe to do so or if the next GKE update will fail after that or revert everything.

We're thinking about changing two things specifically:

1) The fluentd config map in order to parse a little more and use structured logging in our own containers. (while still letting them use stdout/stderr)
2) Change the dashboard and give it a read only scope with no access to secrets.

The 2nd is by far the most important:
Currently with k8s 1.6 via GKE we can restrict our users nicely with RBAC, but this does not limit the ability for users to use "kubectl proxy".
With "kubectl proxy" everybody gets access to the kubernetes-dashboard which by GKE default has the kube-system default token mounted, that can basically do anything.
The dashboard itself has no authn/authz. Therefore anybody can escalate their own privileges to "root" in the cluster and leave any RBAC restrictions behind.
This is nothing that we would be willing to launch in production.

Our solution to this would be to use a token with limited abilities mounted into the dashboard container, or if everything else fails, drop the UI for now.
But in those cases we would need to modify the deployment object created by GKE.

Will changes like these make our cluster go up in flames on the next GKE Master upgrade?

[Robert Bailey]

To ensure that your changes aren't overwritten, it'd be best to delete the GKE-managed addons (e.g. disable logging on your cluster) and install them yourself (e.g. create your own fluentd daemonset).

I don't think it is currently possible to disable the dashboard. 

 

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[Timo Reimann]

Robert, could you please clarify what the implications of swapping out the no-restrictions ServiceAccount associated with the Dashboard would be on the GKE side?

Specifically, if we replaced the existing ServiceAccount with one having less privileges, would the next GKE upgrade maintain this customization or revert it? Or, even worse, could it break the overall upgrade procedure?

As Ingo said, having a wide-open Dashboard without any ability for restriction seems like a fairly big security concern. We couldn't even block ingress access to the kube-system namespace via Network Policies because those aren't supported in GKE yet (to my knowledge).

Thanks.

On Tuesday, June 20, 2017 at 12:21:14 AM UTC+2, Robert Bailey wrote:

On Mon, Jun 19, 2017 at 12:52 AM, Ingo Gottwald <in.go...@gmail.com> wrote:

We would like to change some things on the default GKE setup and the docs don't clarify whether it is safe to do so or if the next GKE update will fail after that or revert everything.

We're thinking about changing two things specifically:

1) The fluentd config map in order to parse a little more and use structured logging in our own containers. (while still letting them use stdout/stderr)
2) Change the dashboard and give it a read only scope with no access to secrets.

The 2nd is by far the most important:
Currently with k8s 1.6 via GKE we can restrict our users nicely with RBAC, but this does not limit the ability for users to use "kubectl proxy".
With "kubectl proxy" everybody gets access to the kubernetes-dashboard which by GKE default has the kube-system default token mounted, that can basically do anything.
The dashboard itself has no authn/authz. Therefore anybody can escalate their own privileges to "root" in the cluster and leave any RBAC restrictions behind.
This is nothing that we would be willing to launch in production.

Our solution to this would be to use a token with limited abilities mounted into the dashboard container, or if everything else fails, drop the UI for now.
But in those cases we would need to modify the deployment object created by GKE.

Will changes like these make our cluster go up in flames on the next GKE Master upgrade?

To ensure that your changes aren't overwritten, it'd be best to delete the GKE-managed addons (e.g. disable logging on your cluster) and install them yourself (e.g. create your own fluentd daemonset).

I don't think it is currently possible to disable the dashboard. 

 

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

[Robert Bailey]

On Tue, Jun 20, 2017 at 2:13 AM, 'Timo Reimann' via Kubernetes user discussion and Q&A <kubernet...@googlegroups.com> wrote:

Robert, could you please clarify what the implications of swapping out the no-restrictions ServiceAccount associated with the Dashboard would be on the GKE side?

The cluster addons that are managed by the system can get replaced when the master gets upgraded, replacing any changes that you have made (the system treats them as being owned by the system). 

The dashboard addon is set to be "reconciled" (see here), which means that the addon manager will run kubectl apply on the addon every 60 seconds or so. So your changes may not persist long enough to be clobber by the next upgrade. 

 

Specifically, if we replaced the existing ServiceAccount with one having less privileges, would the next GKE upgrade maintain this customization or revert it? Or, even worse, could it break the overall upgrade procedure?

It's possible that the apply command will respect your changes, in which case I'd expect them to survive an upgrade unless we change the version of the dashboard addon that is running (at which point I'd expect them to get reverted). In either case, it won't break the upgrade procedure.  

As Ingo said, having a wide-open Dashboard without any ability for restriction seems like a fairly big security concern. We couldn't even block ingress access to the kube-system namespace via Network Policies because those aren't supported in GKE yet (to my knowledge).

You are correct that GKE does not currently support Network Policies. 

 

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

[Timo Reimann]

Thanks for the details.

I just verified that changing the Dashboard Service Account is possible without addon-manager clobbering my modifications.

My only concern left at this stage is the possible revert during a GKE upgrade. I suppose that changing the Dashboard version is rather common, so I'm worried that there will be a time window during upgrades when the Dashboard becomes wide-open again.

Do you think it'd be reasonable to address this issue? If so, is there already a ticket somewhere that I could track and/or help moving forward somehow?

[Robert Bailey]

On Wed, Jun 21, 2017 at 7:07 AM, 'Timo Reimann' via Kubernetes user discussion and Q&A <kubernet...@googlegroups.com> wrote:

Thanks for the details.

I just verified that changing the Dashboard Service Account is possible without addon-manager clobbering my modifications.

Excellent.

 

My only concern left at this stage is the possible revert during a GKE upgrade. I suppose that changing the Dashboard version is rather common, so I'm worried that there will be a time window during upgrades when the Dashboard becomes wide-open again.

Do you think it'd be reasonable to address this issue? If so, is there already a ticket somewhere that I could track and/or help moving forward somehow?

If you file an issue with cloud support you should be able to track it through them. I don't know of a public issue that you can track for this, but we are tracking it internally. 

 

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.