Does kubernetes plugin allow modifying POD resource limits?

[J]

I'm trying to schedule jenkins-slave PODs on a locally running cluster of mine which already has resource limits for cpu/memory for PODs & Containers.

Firstly where can we specify resource limits for PODs (UI or anywhere)? 

The values I do specify through the jenkins UI seem to be meant for containers but they are also effecting PODs limits and eventually never get propagated correctly as it keep sending something else. Not sure what I'm missing here.

Here is what I typically get in the logs:

INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0

Jan 26, 2017 12:36:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call

SEVERE: Error in provisioning; slave=KubernetesSlave name: jenkins-slave-abdd6fe41ba2, template=org.csanchez.jenkins.plugins.kubernetes.PodTemplate@6238b6d9

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://api.k8s.cluster/api/v1/namespaces/cluster/pods. Message: pods "jenkins-slave-abdd6fe41ba2" is forbidden: [maximum cpu usage per Pod is 2, but limit is 500000002., maximum memory usage per Pod is 8Gi, but limit is 8852078592., maximum cpu usage per Container is 2, but limit is 500M.]. Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=pods, name=jenkins-slave-abdd6fe41ba2, retryAfterSeconds=null, additionalProperties={}), kind=Status, message=pods "jenkins-slave-abdd6fe41ba2" is forbidden: [maximum cpu usage per Pod is 2, but limit is 500000002., maximum memory usage per Pod is 8Gi, but limit is 8852078592., maximum cpu usage per Container is 2, but limit is 500M.], metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={})

The corresponding UI settings look like this:

Just to add, I can set the correct numbers for the container limits and that leaves me with POD resource limit errors like so:

[maximum cpu usage per Pod is 2, but limit is 4., maximum memory usage per Pod is 8Gi, but limit is 8852078592.]

[Carlos Sanchez]

On Thursday, January 26, 2017 at 1:54:47 AM UTC+1, J wrote:

I'm trying to schedule jenkins-slave PODs on a locally running cluster of mine which already has resource limits for cpu/memory for PODs & Containers.

Firstly where can we specify resource limits for PODs (UI or anywhere)? 

I don't see any option in the k8s api to set pod limits

https://kubernetes.io/docs/api-reference/v1/definitions/#_v1_podspec

IIUC limits are set in containers only

https://kubernetes.io/docs/admin/limitrange/

 

The values I do specify through the jenkins UI seem to be meant for containers but they are also effecting PODs limits and eventually never get propagated correctly as it keep sending something else. Not sure what I'm missing here.

see https://kubernetes.io/docs/admin/limitrange/

 

Here is what I typically get in the logs:

INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0

Jan 26, 2017 12:36:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call

SEVERE: Error in provisioning; slave=KubernetesSlave name: jenkins-slave-abdd6fe41ba2, template=org.csanchez.jenkins.plugins.kubernetes.PodTemplate@6238b6d9

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://api.k8s.cluster/api/v1/namespaces/cluster/pods. Message: pods "jenkins-slave-abdd6fe41ba2" is forbidden: [maximum cpu usage per Pod is 2, but limit is 500000002., maximum memory usage per Pod is 8Gi, but limit is 8852078592., maximum cpu usage per Container is 2, but limit is 500M.]. Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=pods, name=jenkins-slave-abdd6fe41ba2, retryAfterSeconds=null, additionalProperties={}), kind=Status, message=pods "jenkins-slave-abdd6fe41ba2" is forbidden: [maximum cpu usage per Pod is 2, but limit is 500000002., maximum memory usage per Pod is 8Gi, but limit is 8852078592., maximum cpu usage per Container is 2, but limit is 500M.], metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={})

The corresponding UI settings look like this:

Just to add, I can set the correct numbers for the container limits and that leaves me with POD resource limit errors like so:

[maximum cpu usage per Pod is 2, but limit is 4., maximum memory usage per Pod is 8Gi, but limit is 8852078592.]

I believe your problem is that you need to name your container `jnlp`, otherwise another container will be automatically created to run the jenkins-slave process

https://github.com/jenkinsci/kubernetes-plugin#constraints

 

[J]

You're right, the api only seems to define container limits. While I still look for how the api infers these; I can usually define them in a yaml like this with a "type: Pod" label, and it puts limits on my namespace for both PODs and containers:

However I was able to successfully run a build and create PODs in a namespace without any set limits. 

Setting the container name to 'jnlp' seems to not let it work anywhere though with obscure errors (I couldn't make out anything from these..):

Jan 26, 2017 6:29:38 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision

INFO: Excess workload after pending Spot instances: 1

Jan 26, 2017 6:29:38 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply

INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0

Jan 26, 2017 6:29:38 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call

INFO: Created Pod: jenkins-slave-e6720c6adbb4

Jan 26, 2017 6:29:38 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call

INFO: Waiting for Pod to be scheduled (0/100): jenkins-slave-e6720c6adbb4

Jan 26, 2017 6:29:44 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call

SEVERE: Error in provisioning; slave=KubernetesSlave name: jenkins-slave-e6720c6adbb4, template=org.csanchez.jenkins.plugins.kubernetes.PodTemplate@67f9564d

java.lang.IllegalStateException: Containers are terminated with exit codes: {jnlp=0}

at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:600)

at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:532)

at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Jan 26, 2017 6:29:48 PM hudson.slaves.NodeProvisioner$2 run

WARNING: Provisioned agent Kubernetes Pod Template failed to launch

java.lang.IllegalStateException: Containers are terminated with exit codes: {jnlp=0}

at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:600)

at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:532)

at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

[Carlos Sanchez]

You can check in k8s why the container exited.

You need to make sure args is '${computer.jnlpmac} ${computer.name}'

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/991159ab-4712-4d9f-afe0-c07cf88d2f82%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[J]

That worked with actually setting the name to 'jnlp-slave'; in a resource unlimited namespace. Thanks

Will get back to you as soon as I find the api handles for POD resource limits, as I'd really like to be able to use my existing cluster namespace.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

[Carlos Sanchez]

On Thu, Jan 26, 2017 at 9:04 PM, J <javer...@gmail.com> wrote:

That worked with actually setting the name to 'jnlp-slave'; in a resource unlimited namespace. Thanks

if you don't set the name to jnlp you'll get 2 containers in your pod

 

Will get back to you as soon as I find the api handles for POD resource limits, as I'd really like to be able to use my existing cluster namespace.

it gets the limits from the container limits in the pod AFAIK

 

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e9f0a12d-a500-4904-a840-ec934eaa81df%40googlegroups.com.

[J]

A simple 'jnlp' wasn't working. The PODs go into error and terminate immediately so I couldn't look at any further logs there. Connection using jnlp-agent also just terminates immediately after connecting; so not sure what the actual problem is.