GKE tags and routes on nodes

[Romain Vrignaud]

Hello,

I'd like to have your best practices about networking and GKE as they still are painful points for us.

We have currently two points that are hard for us to manage :

  - we need to add some additional tags in our clusters nodes in order to manage firewalling rules. AFAIK it's not possible to add tag in cluster's node. This is a pain as we need to edit every node every time they are (re)-created (for instance after every node migration). As we have dynamic usage of clusters, we would like to avoid using clusters tag for firewalling as it would imply to change firewalling rules everytime we change (add/delete) a new cluster. We asked for similar feature on dataproc and it was added here : https://cloud.google.com/dataproc/docs/release-notes/service#january_27_2016

  - we are using tim advice here https://github.com/kubernetes/kubernetes/issues/14545#issuecomment-230690123 and setup GCE static routes to nodes to access clusters services range from VPN. But we need to update routes manually when we scale up and down clusters and when we add or remove clusters.

Both of these usage prevent us to use https://cloud.google.com/container-engine/docs/cluster-autoscaler.

Is there any workaround ? Is there any plan on roadmap to be able to specify additional tags on nodes and routes to clusters service range ? 

Regards,

[Brandon Philips]

On Tue, Sep 6, 2016 at 7:00 AM Romain Vrignaud <rvri...@gmail.com> wrote:

We have currently two points that are hard for us to manage :

  - we need to add some additional tags in our clusters nodes in order to manage firewalling rules. AFAIK it's not possible to add tag in cluster's node. This is a pain as we need to edit every node every time they are (re)-created (for instance after every node migration). As we have dynamic usage of clusters, we would like to avoid using clusters tag for firewalling as it would imply to change firewalling rules everytime we change (add/delete) a new cluster. We asked for similar feature on dataproc and it was added here : https://cloud.google.com/dataproc/docs/release-notes/service#january_27_2016

  - we are using tim advice here https://github.com/kubernetes/kubernetes/issues/14545#issuecomment-230690123 and setup GCE static routes to nodes to access clusters services range from VPN. But we need to update routes manually when we scale up and down clusters and when we add or remove clusters.

In both of these cases I think you will need to write a custom controller container that you run on the cluster as a singleton to reconcile this state. In the case of tags you will need to build a "re-tagger" and in the networking case something that talks to the GCE networking API and reconciles nodes to the routes.

Sorry there isn't an easier answer.

Brandon

[paul.t...@ingramcontent.com]

Is there any better answer to this at this point? We had worked through this by having GKE specific routes, keying off of the goog-gke-node tag, It looks like 1.6 GKE took that away... Now we don't have a reliable way to target gke clusters short of adding custom routes for every single cluster pool tag.

[ja...@peakactivity.com]

Create a new node pool with tags:
gcloud container node-pools create new-pool -m n1-standard-8 --cluster staging --num-nodes 3 --tags gke-staging --scopes "https://www.googleapis.com/auth/projecthosting,storage-rw"

cordon your nodes in your old pool, drain the nodes, delete the pool

[Peter]

Its probably worth noting that you can't change tags like you can labels (in gcloud beta currently). Once a tag is set, there's no way to change it as I have found out.

There's an issue here for it: https://issuetracker.google.com/issues/62417678

[Jason Crow]

It's easy enough to work around by creating a node pool and migrating workload over to it.

Jason Crow  Sr. Architect
1880 N Congress Ave, Suite 204  Boynton Beach, FL, 33426 +1-561-202-5576  ja...@peakactivity.com

--
You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/3779D8Mn-Jg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[pfi...@sopost.com]

It is doable if you manage your clusters manually. My scenario is that I have the tags set in terraform, but I changed how the tag got said value (to correct an error). Now terraform wants to recreate the entire cluster and node pool.

I had just assumed tags were changeable like labels, and with normal compute instances they are (or you can remove / add them after the fact). with it being unchangeable, it basically changes the way I need to write the terraform module. Just a gotcha for anyone else using terraform to be aware of.

[Jason Crow]

Ah yes I see your problem.

On Wed, Feb 7, 2018 at 10:18 AM <pfi...@sopost.com> wrote:

On Wednesday, 7 February 2018 14:20:33 UTC, Jason Crow  wrote:
> It's easy enough to work around by creating a node pool and migrating workload over to it.
>
>
>
>
>
>
>
>
> Jason Crow 
> Sr. Architect
>

> 1880 N Congress Ave, Suite 204 
> Boynton Beach, FL, 33426
> +1-561-202-5576 
> ja...@peakactivity.com
>
>
>
> On Wed, Feb 7, 2018 at 9:11 AM, Peter <pfi...@sopost.com> wrote:
> On Friday, 20 October 2017 02:23:39 UTC+1, ja...@peakactivity.com  wrote:
>
> > On Friday, June 23, 2017 at 10:52:04 AM UTC-4, paul.t...@ingramcontent.com wrote:
>
> > > On Wednesday, September 7, 2016 at 8:05:43 PM UTC-5, Brandon Philips wrote:
>
> > > > On Tue, Sep 6, 2016 at 7:00 AM Romain Vrignaud <rvri...@gmail.com> wrote:
>
> > > >
>
> > > > We have currently two points that are hard for us to manage :
>
> > > >   - we need to add some additional tags in our clusters nodes in order to manage firewalling rules. AFAIK it's not possible to add tag in cluster's node. This is a pain as we need to edit every node every time they are (re)-created (for instance after every node migration). As we have dynamic usage of clusters, we would like to avoid using clusters tag for firewalling as it would imply to change firewalling rules everytime we change (add/delete) a new cluster. We asked for similar feature on dataproc and it was added here : https://cloud.google.com/dataproc/docs/release-notes/service#january_27_2016
>
> > > >
>
> > > >
>
> > > >   - we are using tim advice here https://github.com/kubernetes/kubernetes/issues/14545#issuecomment-230690123 and setup GCE static routes to nodes to access clusters services range from VPN. But we need to update routes manually when we scale up and down clusters and when we add or remove clusters.
>
> > > >
>
> > > >
>
> > > > In both of these cases I think you will need to write a custom controller container that you run on the cluster as a singleton to reconcile this state. In the case of tags you will need to build a "re-tagger" and in the networking case something that talks to the GCE networking API and reconciles nodes to the routes.
>
> > > >
>
> > > >
>
> > > > Sorry there isn't an easier answer.
>
> > > >
>
> > > >
>
> > > > Brandon
>
> > >
>
> > > Is there any better answer to this at this point? We had worked through this by having GKE specific routes, keying off of the goog-gke-node tag, It looks like 1.6 GKE took that away... Now we don't have a reliable way to target gke clusters short of adding custom routes for every single cluster pool tag.
>
> >
>
> > Create a new node pool with tags:
>
> > gcloud container node-pools create new-pool -m n1-standard-8 --cluster staging --num-nodes 3 --tags gke-staging --scopes "https://www.googleapis.com/auth/projecthosting,storage-rw"
>
> >
>
> > cordon your nodes in your old pool, drain the nodes, delete the pool
>
>
>
> Its probably worth noting that you can't change tags like you can labels (in gcloud beta currently). Once a tag is set, there's no way to change it as I have found out.
>
>
>
> There's an issue here for it: https://issuetracker.google.com/issues/62417678
>
>
>
>
>
> --
>
> You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
>
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/3779D8Mn-Jg/unsubscribe.
>

> To unsubscribe from this group and all its topics, send an email to kubernetes-use...@googlegroups.com.
>
> To post to this group, send email to kubernet...@googlegroups.com.

>
> Visit this group at https://groups.google.com/group/kubernetes-users.
>
> For more options, visit https://groups.google.com/d/optout.

It is doable if you manage your clusters manually. My scenario is that I have the tags set in terraform, but I changed how the tag got said value (to correct an error). Now terraform wants to recreate the entire cluster and node pool.

I had just assumed tags were changeable like labels, and with normal compute instances they are (or you can remove / add them after the fact). with it being unchangeable, it basically changes the way I need to write the terraform module. Just a gotcha for anyone else using terraform to be aware of.

--
You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/3779D8Mn-Jg/unsubscribe.

To unsubscribe from this group and all its topics, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.