Job execution in Kubernetes as a black box

392 views
Skip to first unread message

Diego Rodríguez Rodríguez

unread,
Oct 10, 2016, 8:58:53 AM10/10/16
to Kubernetes user discussion and Q&A
Hello, I have a doubt about how Kubernetes' jobs are handled.

I have a queue to submit certain amount of incoming tasks (Celery and RabbitMQ take care of this). Each one of this tasks is, in fact, a workflow which will be executed in a worker (Celery worker with a DAG executor inside). Each step of the workflow is a Docker image with an input file and an output file.

My question is, if I submit jobs from the workflow engine directly to the Kubernetes API, what happens I at some point there are no more resources? Will the remaining tasks be kept or will they be lost? My goal is to treat Kubernetes' jobs as a black box to submit works to. This works are of a very different and heterogeneous nature and I wouldn't need to bother with what is inside them because they are dockerized and executed by Kubernetes at some point.

To sum up, I already have the layer of Celery workers with a DAG executor inside which knows the right order of the tasks and knows how to manage everything concerning the workflow. These components will submit jobs (through Kubernetes API) and will wait for them to be executed and then continue with the remaining tasks asking Kubernetes to run them until the workflow ends.

I have read about a somehow related issue in Github:
https://github.com/kubernetes/kubernetes/pull/17787
https://github.com/kubernetes/kubernetes/pull/18827

I couldn't determine if this is closed or it is coming in a future release.


Thanks in advance!

Daniel Smith

unread,
Oct 10, 2016, 1:24:46 PM10/10/16
to kubernet...@googlegroups.com
If the system lacks resources, Pods will remain "Pending" until resources become available. Cluster scalers may use pending pods as a signal that the cluster size should be increased.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

Diego Rodríguez Rodríguez

unread,
Oct 12, 2016, 8:39:09 AM10/12/16
to Kubernetes user discussion and Q&A
I have already configured my cluster to test what you have stated. What I have done so far is to create a ResourceQuota which takes care that there will not be more than 4 pods running. Then I ask for say 20 jobs to be completed.

What happens in reality is that the first 4 jobs are completed and then, even though the pods are completed and therefore resources are available, it takes some minutes (around 4-5 minutes) to have another 4 jobs being completed. Indeed, what you said is true, however it is no practical because a delay of minutes can not be assumed.
The waiting jobs events look like this:
`FailedCreate    Error creating: pods "d59fa9b2-6c6b-4dc5-b149-7f89b35421bf-10-" is forbidden: Exceeded quota: compute-resources, requested: pods=1, used: pods=4, limited: pods=4`
So, it fails because there are no resources but it is trying it again only after some minutes. This behavior is far from the desired one, which is relaying on Kubernetes for the execution of a set of tasks no matter the resources available, just getting them executed as soon as possible.

thanks


On Monday, 10 October 2016 19:24:46 UTC+2, Daniel Smith wrote:
If the system lacks resources, Pods will remain "Pending" until resources become available. Cluster scalers may use pending pods as a signal that the cluster size should be increased.
On Mon, Oct 10, 2016 at 5:58 AM, Diego Rodríguez Rodríguez <diego...@gmail.com> wrote:
Hello, I have a doubt about how Kubernetes' jobs are handled.

I have a queue to submit certain amount of incoming tasks (Celery and RabbitMQ take care of this). Each one of this tasks is, in fact, a workflow which will be executed in a worker (Celery worker with a DAG executor inside). Each step of the workflow is a Docker image with an input file and an output file.

My question is, if I submit jobs from the workflow engine directly to the Kubernetes API, what happens I at some point there are no more resources? Will the remaining tasks be kept or will they be lost? My goal is to treat Kubernetes' jobs as a black box to submit works to. This works are of a very different and heterogeneous nature and I wouldn't need to bother with what is inside them because they are dockerized and executed by Kubernetes at some point.

To sum up, I already have the layer of Celery workers with a DAG executor inside which knows the right order of the tasks and knows how to manage everything concerning the workflow. These components will submit jobs (through Kubernetes API) and will wait for them to be executed and then continue with the remaining tasks asking Kubernetes to run them until the workflow ends.

I have read about a somehow related issue in Github:
https://github.com/kubernetes/kubernetes/pull/17787
https://github.com/kubernetes/kubernetes/pull/18827

I couldn't determine if this is closed or it is coming in a future release.


Thanks in advance!

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

Daniel Smith

unread,
Oct 12, 2016, 1:28:45 PM10/12/16
to kubernet...@googlegroups.com, David Eads
Ah, you're blocked because the quota check reconciles slowly. The quick fix is probably to just get more quota.

+David who may know of an already existing issue about this.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

David Eads

unread,
Oct 12, 2016, 3:18:07 PM10/12/16
to Kubernetes user discussion and Q&A, de...@redhat.com, Derek Carr
Quota reconcilation of pods was supposed to run at a faster rate.  I'm not sure if 5 minutes is that faster rate.  

Derek: Do you recall where and how the fast pod replenishment is?

Derek Carr

unread,
Oct 12, 2016, 5:54:16 PM10/12/16
to Kubernetes user discussion and Q&A, de...@redhat.com, dec...@redhat.com
The quota controller observes pod deletions, and queues the pertinent quotas for processing in response.  The replenishment should be relatively quick now.

We actually exercise this behavior in e2e runs.
  1. What version of Kubernetes are you running?
  2. How many quotas?
  3. How many pods?
My guess is that there is a conflict of some kind during the quota sync, and the 5 minutes you are seeing is a full re-sync interval for the quota controller.

It would be good to try to determine what the conflict is so we can fix.

If you have a quick set of steps I can run to reproduce that would be great.

I recommend opening an issue with the above details and cc @derekwaynecarr

Thanks,
Derek

Diego Rodríguez Rodríguez

unread,
Oct 13, 2016, 9:49:29 AM10/13/16
to Kubernetes user discussion and Q&A, de...@redhat.com, dec...@redhat.com
I have already created an issue in Github.

Building on top of that, and closely related to https://github.com/kubernetes/kubernetes/issues/34363 and to https://github.com/kubernetes/kubernetes/pull/18827 I have the following question. Since I am planning to use Kubernetes as a executing backend for the workflow execution environment I am building, at some point we will need to know the current state of a job. Well, this time has come, and for now the only thing I am able to do is polling the API (kubernetes.io/docs/api-reference/extensions/v1beta1/operations/). Taking into account that this is unacceptable, and I only did it for testing purposes, what is the recommended way in Kubernetes for doing so? (Keep in mind that I should do it from inside the cluster, no kubectl command) If it is not possible right now, will it be possible soon? I am working on a quite big project and since it is starting, and the first problems we are facing are related to this issues, it is time to decide if I continue with Kubernetes or I have to start looking for alternatives.

Thanks for your commitment

Rodrigo Campos

unread,
Oct 13, 2016, 10:22:13 AM10/13/16
to kubernet...@googlegroups.com, de...@redhat.com, dec...@redhat.com
You can probably watch via the API and get notified, but Haven't used it.

But if you are building something not trivial, you may be better off creating a third party resource that handles this and creates kubernetes jobs under the hood. You will have more flexibility, etc.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

Diego Rodríguez Rodríguez

unread,
Oct 13, 2016, 10:43:26 AM10/13/16
to kubernet...@googlegroups.com
Thanks for the advice but I have already tried the `watch` parameter and as far as I understand it does not work as expected. It gives me the same output as if I did not use the `watch` parameter even if I provide the first `resourceVersion` in order to be sure that it is comparing with the first state of the job.

I am making reference to http://kubernetes.io/docs/api-reference/extensions/v1beta1/operations/ on the 'watch changes to an object of kind Job' section.

To be honest I have never used something like this `watch` parameter before, so I may be misunderstanding it, but I am afraid I am not.


--
You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/qd5Ua2UqQNo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-users+unsubscribe@googlegroups.com.

Daniel Smith

unread,
Oct 13, 2016, 1:13:42 PM10/13/16
to kubernet...@googlegroups.com
Watch definitely should do the thing you want.

In a loop:
1. List. Note the returned RV. This gives you the current state of the system.
2. Watch, passing the RV.
3. Process events until the watch closes or times out.

Diego Rodríguez Rodríguez

unread,
Oct 14, 2016, 4:36:47 AM10/14/16
to kubernet...@googlegroups.com
This is what I am doing already

def get_job_state(job_id):
    response = requests.get('{}{}/{}'.format('https://kubernetes',
                                             ENDPOINT, job_id),
                            headers={'Authorization': 'Bearer {}'.format(
                                TOKEN
                            )},
                            verify=CA_CERT_PATH)

    if response.ok:
        res_dict = json.loads(response.text)
        resource_version = res_dict['metadata']['resourceVersion']
        while not (res_dict['status'].get('succeeded', False) or
                   res_dict['status'].get('failed', False)):
            response = requests.get('{0}{1}/{2}?watch=true&resourceVersion={3}'
                                    .format('https://kubernetes',
                                            ENDPOINT, job_id,
                                            resource_version),
                                    headers={'Authorization': 'Bearer {}'.
                                             format(
                                                 TOKEN
                                             )},
                                    verify=CA_CERT_PATH)

            if response.ok:
                res_dict = json.loads(response.text)
            else:
                # handle this
But it is not working, it is doing just the same as when I was using it without watch and RV.
 

Diego Rodríguez Rodríguez

unread,
Oct 14, 2016, 8:15:32 AM10/14/16
to kubernet...@googlegroups.com
I have identified the problem. It has nothing to do with Kubernetes, it is about how Python's requests module handles this kind of connections. Further information can be found in:

https://github.com/kennethreitz/requests/issues/2433   ---> I have tested it, and it does not work for me.
https://cobe.io/blog/posts/kubernetes-watch-python/     ---> Talks about the problem and provides a link to the Python module they have created to wrap the k8s API calls

The module is https://bitbucket.org/cobeio/kube/overview and it might be of help for someone who runs in that kind of troubles, for me does not work because as I could read in their docs, they do not support k8s jobs yet.

Daniel Smith

unread,
Oct 14, 2016, 1:23:14 PM10/14/16
to kubernet...@googlegroups.com
Glad you figured out what was going on (I was about to say, I don't know python well but that code doesn't look like it could possibly work for a streaming connection).

We are working on generating a python client. I will make sure we try the watch feature.

Diego Rodríguez Rodríguez

unread,
Oct 14, 2016, 2:07:59 PM10/14/16
to kubernet...@googlegroups.com
It was just that, but I feel like I would need a library to rely on as soon as possible (I am doing everything by hand) because the actual ones are not mature enough. When do you plan to release it? I am very interested.

Daniel Smith

unread,
Oct 14, 2016, 2:15:01 PM10/14/16
to kubernet...@googlegroups.com
I would follow https://github.com/mbohlool/, he'll probably be the one making the repos.

On Fri, Oct 14, 2016 at 11:07 AM, Diego Rodríguez Rodríguez <diego...@gmail.com> wrote:
It was just that, but I feel like I would need a library to rely on as soon as possible (I am doing everything by hand) because the actual ones are not mature enough. When do you plan to release it? I am very interested.

ravi prasad l r

unread,
Oct 16, 2016, 2:59:52 PM10/16/16
to Kubernetes user discussion and Q&A
This thread brings up an important point about available kubernetes client bindings and their maturity in terms of full fledged support of k8s apis.
My experience has been with these :
- https://github.com/kubernetes/client-go/  : official Go binding with full support of k8s API; we just migrated one of the project which was using in-tree internal clientset to this.
- https://github.com/fabric8io/kubernetes-client : Java client with fluent DSL; support most of the k8s APIs, good watch client support, also supports higher level constructs(found only in kubectl) like rolling-upgrade, cascading deletes, out of the box.

I'm also interested in learning about any full fledged js/nodejs and python bindings with good watch api support.

-Ravi
Reply all
Reply to author
Forward
0 new messages