Tainting preemptible node pools on GKE

[Micke Lisinge]

With the possibility of creating preemptible node pools on GKE it would be amazing if you could also taint these nodes, maybe even enable setting taints for all pools just like you can label pools today.

I know taints are currently in alpha but it would be great if this could get taken into consideration for when taints reaches beta.

I know this could also be solved with node affinity and anti-affinity but it would be amazing if you would be able to taint nodes to only be used by pods with the proper toleration.

[David Oppenheimer]

Nodes running on preemptible VMs are labeled with cloud.google.com/gke-preemptible=true

(see documentation here)

The downside of using a taint instead (or in addition) is that then users would have to explicitly add tolerations to use these nodes. So the default out-of-the-box behavior would be more complicated (and, for people who didn't thoroughly read the documentation, confusing). 

I guess we could add an option to "gcloud beta container node-pools create" to tell GKE to add a taint in addition to a label, so users who wanted this behavior could request it.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[Micke Lisinge]

For sure, i'm not asking you to change the defaults.

Exactly, if you could add a "--node-taints" flag, just like the "--node-labels" flag works to "gcloud beta container node-pools create" then it would cover my use case.

Den måndag 21 november 2016 kl. 01:11:28 UTC+1 skrev David Oppenheimer:

Nodes running on preemptible VMs are labeled with cloud.google.com/gke-preemptible=true

(see documentation here)

The downside of using a taint instead (or in addition) is that then users would have to explicitly add tolerations to use these nodes. So the default out-of-the-box behavior would be more complicated (and, for people who didn't thoroughly read the documentation, confusing). 

I guess we could add an option to "gcloud beta container node-pools create" to tell GKE to add a taint in addition to a label, so users who wanted this behavior could request it.

On Wed, Nov 16, 2016 at 6:00 AM, Micke Lisinge <lis...@rabble.se> wrote:

With the possibility of creating preemptible node pools on GKE it would be amazing if you could also taint these nodes, maybe even enable setting taints for all pools just like you can label pools today.

I know taints are currently in alpha but it would be great if this could get taken into consideration for when taints reaches beta.

I know this could also be solved with node affinity and anti-affinity but it would be amazing if you would be able to taint nodes to only be used by pods with the proper toleration.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

[David Oppenheimer]

Thanks for the suggestion. I think it's a reasonable suggestion and I've filed an internal feature request to add this to gcloud, but I can't guarantee if/when it will be implemented.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

[Micke Lisinge]

Thanks David!

[je...@blendle.com]

Has this feature been implemented already? If it has, I haven't been able to find it in the docs or the CLI man pages.

We are looking for the exact opposite of this request: to taint regular nodes, so all deployments, except the ones that *require* a regular node, are automatically deployed to our preemptible nodes.

[je...@blendle.com]

Just to make my feature request a bit more clear:

I'd like to be able to taint node-pools, which we would use to taint the node-pool that consists of regular nodes in our cluster, just as David proposed.

[David Oppenheimer]

Sorry, not implemented yet. It's definitely on our TODO list, so watch the GKE release notes (or ping this thread periodically :-)

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

[je...@blendle.com]

For now, we solved this by running a simple looped script on our cluster:

while true; do
kubectl taint nodes -l type=regular dedicated=regular:NoSchedule
sleep 60
done

Make sure you create your GKE node-pool(s) with the right labels, and this will make sure they are tainted, even if new ones are started.

[Ajit Kumar]

Now taints are supported in GKE and you don't need to run kubectl taint nodes. Please follow https://cloud.google.com/container-engine/docs/node-taints for more information.

Thanks

Ajit

[donina....@gmail.com]

Hi there,
I'm having troubles using GKE with preemptible nodes/pool and node taint.

See below my cluster creation command

gcloud beta container --project "bleah" clusters create "c0b001" --region "europe-west1" --username="admin" --cluster-version "1.8.4-gke.0" --machine-type "n1-standard-2" --image-type "COS" --disk-size "80" --local-ssd-count "1" --scopes "https://www.googleapis.com/auth/compute.readonly","https://www.googleapis.com/auth/datastore","https://www.googleapis.com/auth/devstorage.read_write","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" --num-nodes "1" --network "default" --enable-cloud-logging --enable-cloud-monitoring --enable-autoscaling --enable-legacy-authorization --enable-network-policy --addons "HttpLoadBalancing","HorizontalPodAutoscaling" --min-nodes "1" --max-nodes "3" --enable-autoupgrade --enable-autorepair --maintenance-window "02:00" --labels "env=dev","cluster=c0b001"

gcloud beta container node-pools create preemptible-pool --preemptible --cluster=c0b001 --disk-size=50 --enable-autoupgrade --enable-autorepair --image-type=cos --machine-type=n1-standard-2 --node-labels=pool=preemptible,machinetype=n1-standard-2 --num-nodes=1 --region=europe-west1 --enable-autoscaling --max-nodes=3 --min-nodes=1 --enable-cloud-endpoints --scopes "https://www.googleapis.com/auth/compute.readonly","https://www.googleapis.com/auth/datastore","https://www.googleapis.com/auth/devstorage.read_write","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append"--node-taints=dedicated=stateless:NoSchedule

preemptible-pool never becomes ready and fluentd pods stuck in ContainerCreating phase. I assume this is caused by the taint but there is no way for me to specify tolerations on "gke managed" containers such as calico or fluentd.

My goal is allow only some particularly deployments/pod with tolerations config to be scheduled on this pool while all the others must be scheduled (by default) on default-pool nodes.

Thanks.

[David Oppenheimer]

What version are you using? I believe this was fixed in this PR.

My goal is allow only some particularly deployments/pod to be scheduled on this pool while all the others must be located on default-pool nodes.

Thanks.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

[Lorenzo Donina]

Hi there,

I’m using 1.8.4 latest with regional cluster deployments. 

Thanks

Sent from my iPhone, erroneous words are a feature, not a typo.

You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/KUm233PUp-I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

[David Oppenheimer]

Can you do "kubectl describe pod" on one of the fluentd pods and post it here?

On Sun, Dec 10, 2017 at 11:00 PM, Lorenzo Donina <donina....@gmail.com> wrote:

Hi there,

To unsubscribe from this group and all its topics, send an email to kubernetes-users+unsubscribe@googlegroups.com.

To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[Lorenzo D]

Now work as expected.

I tried several times in the last 4 days and nodes had never become "ready"

Sorry about that. 

To unsubscribe from this group and all its topics, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.