Run a job on all nodes in the cluster

[Mayank]

Hi All

Is there a way to create a kubernetes job that runs on all nodes in the cluster and then finishes without creating one job per node using node selector ? Or may be this is enhancement to say run this job on all hosts with regex *.ops.net and viola 

-Mayank

[David Oppenheimer]

Unfortunately I don't think it's possible. The documentation for DaemonSet says the RestartPolicy must be Always. If it allowed Never then it would do what you want.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[Brandon Philips]

I think this would be a nice thing to have. I have seen a few users wanting to do things like run a quick script against all nodes in a cluster that say tweaks a sysctl across the entire fleet. Or, gathers up some setting and pushes the results to some service.

I think it would be worthwhile to gather other use cases and write a proposal.

On Fri, Jan 20, 2017 at 1:01 AM 'David Oppenheimer' via Kubernetes user discussion and Q&A <kubernet...@googlegroups.com> wrote:

Unfortunately I don't think it's possible. The documentation for DaemonSet says the RestartPolicy must be Always. If it allowed Never then it would do what you want.

On Thu, Jan 19, 2017 at 2:37 PM, Mayank <krma...@gmail.com> wrote:

Hi All

Is there a way to create a kubernetes job that runs on all nodes in the cluster and then finishes without creating one job per node using node selector ? Or may be this is enhancement to say run this job on all hosts with regex *.ops.net and viola 

-Mayank

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

[David Oppenheimer]

Brandon, would you like to file an issue in kubernetes/kubernetes to start? FWIW the privileged run-to-completion node configuration script is a use case we have also seen at Google, but the semantics get a bit tricky. We could start with just a run-to-completion DaemonSet which I think covers the use cases you mentioned as well as the one from Mayank.

On Fri, Jan 20, 2017 at 10:26 AM, Brandon Philips <brandon...@coreos.com> wrote:

I think this would be a nice thing to have. I have seen a few users wanting to do things like run a quick script against all nodes in a cluster that say tweaks a sysctl across the entire fleet. Or, gathers up some setting and pushes the results to some service.

I think it would be worthwhile to gather other use cases and write a proposal.

On Fri, Jan 20, 2017 at 1:01 AM 'David Oppenheimer' via Kubernetes user discussion and Q&A <kubernetes-users@googlegroups.com> wrote:

Unfortunately I don't think it's possible. The documentation for DaemonSet says the RestartPolicy must be Always. If it allowed Never then it would do what you want.

On Thu, Jan 19, 2017 at 2:37 PM, Mayank <krma...@gmail.com> wrote:

Hi All

Is there a way to create a kubernetes job that runs on all nodes in the cluster and then finishes without creating one job per node using node selector ? Or may be this is enhancement to say run this job on all hosts with regex *.ops.net and viola 

-Mayank

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

[Tim Hockin]

Concretely the "tweak a sysctl" thing leaves machines that are
"dirty". Once you allow any users to do this, the machines become
less useful for anyone else who doesn't specifically tolerate that
tweak. Almost every sysctl represents a tradeoff. Optimize for
low-latency network? Pay higher CPU and memory costs. And so on.

>>>> an email to kubernetes-use...@googlegroups.com.
>>>> To post to this group, send email to kubernet...@googlegroups.com.

>>>> Visit this group at https://groups.google.com/group/kubernetes-users.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Kubernetes user discussion and Q&A" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to kubernetes-use...@googlegroups.com.
>>> To post to this group, send email to kubernet...@googlegroups.com.

>>> Visit this group at https://groups.google.com/group/kubernetes-users.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Kubernetes user discussion and Q&A" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to kubernetes-use...@googlegroups.com.
>> To post to this group, send email to kubernet...@googlegroups.com.

>> Visit this group at https://groups.google.com/group/kubernetes-users.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-use...@googlegroups.com.
> To post to this group, send email to kubernet...@googlegroups.com.

[Daniel Smith]

If you want to tweak something about a machine, you probably want to actually occasionally check to ensure that it stayed tweaked (no one put it back). So I guess I'd expect the current behavior of daemonset is actually correct for that.

>>>> an email to kubernetes-users+unsubscribe@googlegroups.com.
>>>> To post to this group, send email to kubernetes-users@googlegroups.com.

>>>> Visit this group at https://groups.google.com/group/kubernetes-users.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Kubernetes user discussion and Q&A" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to kubernetes-users+unsubscribe@googlegroups.com.
>>> To post to this group, send email to kubernetes-users@googlegroups.com.

>>> Visit this group at https://groups.google.com/group/kubernetes-users.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Kubernetes user discussion and Q&A" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to kubernetes-users+unsubscribe@googlegroups.com.
>> To post to this group, send email to kubernetes-users@googlegroups.com.

>> Visit this group at https://groups.google.com/group/kubernetes-users.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-users+unsubscribe@googlegroups.com.
> To post to this group, send email to kubernetes-users@googlegroups.com.

> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

[Brandon Philips]

On Fri, Jan 20, 2017 at 1:40 PM 'Tim Hockin' via Kubernetes user discussion and Q&A <kubernet...@googlegroups.com> wrote:

Concretely the "tweak a sysctl" thing leaves machines that are
"dirty".  Once you allow any users to do this, the machines become
less useful for anyone else who doesn't specifically tolerate that
tweak.  Almost every sysctl represents a tradeoff.  Optimize for
low-latency network?  Pay higher CPU and memory costs.  And so on.

I am not saying it is necessarily the right solution for users; just that I have seen people wanting to do `kubectl get nodes | while read host ; ssh $host echo foo > /proc/sys/bar`. It would be nice to at least bring that into the API fold and auditing.

Brandon

[Rodrigo Campos]

I also think having"state changes" it's a bad use case, as it's been pointed out (it can be changed again, etc. No "self healing" once you have state like that, that goes against most of the basic ideas of reconciliation loops in k8s).

But I want to add that, although that use case seems like a wrong idea, there might be other valid use cases (like gather some data from nodes once). Maybe it's a good idea to find them first :-)

On Friday, January 20, 2017, Brandon Philips <brandon...@coreos.com> wrote:

>>>> an email to kubernetes-users+unsubscribe@googlegroups.com.
>>>> To post to this group, send email to kubernetes-users@googlegroups.com.

>>>> Visit this group at https://groups.google.com/group/kubernetes-users.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Kubernetes user discussion and Q&A" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to kubernetes-users+unsubscribe@googlegroups.com.
>>> To post to this group, send email to kubernetes-users@googlegroups.com.

>>> Visit this group at https://groups.google.com/group/kubernetes-users.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Kubernetes user discussion and Q&A" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to kubernetes-users+unsubscribe@googlegroups.com.
>> To post to this group, send email to kubernetes-users@googlegroups.com.

>> Visit this group at https://groups.google.com/group/kubernetes-users.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-users+unsubscribe@googlegroups.com.
> To post to this group, send email to kubernetes-users@googlegroups.com.

> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

[Mayank]

Yeah, my use case is basically change the permissions of the hostPath so that my pods running as non root can access it. I dont want a pod running as daemon set for this purpose, since the pod will just be sitting there doing nothing, until a new host appears or a host is reimaged, etc. I believe there is no way to mount a hostPath with so that a non root user can access. I would prefer to run a job externally once in a while to ensure this once in a lifetime needed thing.

There have been other debugging use cases which , want you to run some commands on a subset of nodes and get their output and complete. I believe a first class kubernetes way of accomplishing those would be awesome, although most of those use cases are debugging scenarios. I do agree with the general sentiment that leaving a node dirty is a bad idea but debugging /testing scenarios or collecting some data by running some commands might be scenarios to consider. What do you all think ?

-Mayank

[Ben Kochie]

This sounds like a job that should happen at provisioning time, or by config management software.

[David Oppenheimer]

I would recommend filing a Github issue about this. Discussing on the mailing list is not good for tracking this long-term.

[Mayank]

i opened one yesterday 

https://github.com/kubernetes/kubernetes/issues/40290

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

[Justin Garrison]

I have an old issue open for this.

https://github.com/kubernetes/kubernetes/issues/36601

[Brandon Philips]

I made a cute integration with the ol' SSH standby tool Fabric to accomplish something like this:

https://github.com/coreos/coreos-kubernetes/pull/835

https://twitter.com/BrandonPhilips/status/831997779723513856

Brandon

[mrpanigale]

I feel this feature is even more important with the deprecation of Fleet.

"For me this is a very important issue but more in the broader scope of Jobs (not just scheduled).

CoreOS has deprecated Fleet (distributed systemd). With Fleet one could broadcast a global systemd unit to run on and update all machines in a cluster with a label selector.

Given a kubernetes cluster provisioned with no configuration management tool such as Puppet, Fleet or Chef it would be logical for kubernetes to support modification of its self using just kubernetes. This is why I personally would like job scheduling choice:

- Run or schedule a job on all machines

- Run or schedule a job on a sub-set of machines using a label selector

- Run or schedule a job on a single machine"

[David Oppenheimer]

Just to be clear, DaemonSet does allow all of these

> - Run or schedule a job on all machines

> - Run or schedule a job on a sub-set of machines using a label selector

> - Run or schedule a job on a single machine"

It just requires that the "job" be a "run forever" thing, not a "run to completion" thing (hence not technically a Kubernetes Job).

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

[mrpanigale]

Hi David,

I am aware of this already being achievable via Daemonsets, although last time I checked a Daemonset had to run on all nodes and did not honor a node selector (this may have been fixed).

Assuming that this has been fixed, it is often not desirable for a "Puppet" script to run indefinitely. So with this analogy I dont find it useful to package post deployment scripts as Daemonsets. 

For me the undesirable work around would be to deploy a higher-level  agent something like Fleet or Salt as a Daemonset and then run interpreted scripts remotely.

kind regards,

Andrew

[David Oppenheimer]

On Sun, Feb 26, 2017 at 12:53 AM, 'mrpanigale' via Kubernetes user discussion and Q&A <kubernet...@googlegroups.com> wrote:

Hi David,

I am aware of this already being achievable via Daemonsets, although last time I checked a Daemonset had to run on all nodes and did not honor a node selector (this may have been fixed).

It does honor node selector. If you find that's not working, let us know.

 

Assuming that this has been fixed, it is often not desirable for a "Puppet" script to run indefinitely. So with this analogy I dont find it useful to package post deployment scripts as Daemonsets. 

Right, understood.

 

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.