How to use containerTemplates in declarative kubernetes agent definition

1,780 views
Skip to first unread message

Matt Buckland

unread,
Sep 7, 2018, 6:56:16 PM9/7/18
to Jenkins Users
Hello list,

I'm running jenkins inside k8s:
jenkins/jenkins:lts
jenkins/jnlp-slave:3.23-1
kubernetes-plugin: 1.12.4

I'm using the following to define my agent:

  agent {
    kubernetes {
      label "${JOB_NAME}"
      inheritFrom 'default'
      containerTemplate {
        name 'jnlp'
        image '${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/jenkins/jnlp-slave:${AGENT_IMAGE_TAG}'
        resourceRequestCpu '200m'
        resourceLimitCpu '1'
        resourceRequestMemory '256Mi'
        resourceLimitMemory '1Gi'
      }



In the logs I have this message:

Sep 07, 2018 5:11:34 PM org.csanchez.jenkins.plugins.kubernetes.pipeline.KubernetesDeclarativeAgent getAsArgs
WARNING: containerTemplate option in declarative pipeline is deprecated, use containerTemplates



I've not been able to find any documentation or examples showing how to use this containerTemplates option (note the "s" on the end). I've tried things such as:

      containerTemplates [
        containerTemplate {
          name 'jnlp'
          image '${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/jenkins/jnlp-slave:${AGENT_IMAGE_TAG}'
          resourceRequestCpu '200m'
          resourceLimitCpu '1'
          resourceRequestMemory '256Mi'
          resourceLimitMemory '1Gi'
        }
      ]


and various other combinations, but I can't get it to work.

Anyone have any suggestions?

Many thanks,

Matt

Carlos Sanchez

unread,
Sep 8, 2018, 2:54:58 AM9/8/18
to Jenkins Users
https://github.com/jenkinsci/kubernetes-plugin/blob/master/README.md#declarative-pipeline

Note that it was previously possible to define containerTemplate but that has been deprecated in favor of the yaml format.

containerTemplates also works but yaml syntax is preferred



--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/c98259fd-1aa4-40a1-890d-cb157cf3bf5b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Matt Buckland

unread,
Sep 8, 2018, 6:06:13 AM9/8/18
to jenkins...@googlegroups.com
Hi Carlos,

Yes, I'm well aware that containerTemplate is deprecated. Does it follow
that containerTemplates is also deprecated? The warning message implies
that this is not the case, so it is somewhat unclear.

Regardless of this, I am still interested to know what the correct
syntax is, since yaml is expanding my env vars to null
(https://issues.jenkins-ci.org/browse/JENKINS-53472) I'm not able to use
that as a drop-in replacement. I can workaround this by generating a
static yaml file from a template with variables expanded by something
external, but it's not as neat.

Kind regards,

Matt

On 08/09/18 07:54, Carlos Sanchez wrote:
> https://github.com/jenkinsci/kubernetes-plugin/blob/master/README.md#declarative-pipeline
>
> Note that it was previously possible to define containerTemplate but
> that has been deprecated in favor of the yaml format.
>
> containerTemplates also works but yaml syntax is preferred
>
>
>
> On Sat, Sep 8, 2018 at 12:56 AM Matt Buckland <ma...@fascinatedcow.net
> <mailto:ma...@fascinatedcow.net>> wrote:
>
> Hello list,
>
> I'm running jenkins inside k8s:
> jenkins/jenkins:lts
> jenkins/jnlp-slave:3.23-1
> kubernetes-plugin: 1.12.4
>
> I'm using the following to define my agent:
>
>   agent {
>     kubernetes {
>       label "${JOB_NAME}"
>       inheritFrom 'default'
>       containerTemplate {
>         name 'jnlp'
>         image
> '${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/jenkins/jnlp-slave:${AGENT_IMAGE_TAG}
> <http://amazonaws.com/jenkins/jnlp-slave:$%7BAGENT_IMAGE_TAG%7D>'
>         resourceRequestCpu '200m'
>         resourceLimitCpu '1'
>         resourceRequestMemory '256Mi'
>         resourceLimitMemory '1Gi'
>       }
>
>
>
> In the logs I have this message:
>
> Sep 07, 2018 5:11:34 PM
> org.csanchez.jenkins.plugins.kubernetes.pipeline.KubernetesDeclarativeAgent
> getAsArgs
> WARNING: containerTemplate option in declarative pipeline is
> deprecated, use containerTemplates
>
>
> I've not been able to find any documentation or examples showing how
> to use this containerTemplates option (note the "s" on the end).
> I've tried things such as:
>
>       containerTemplates [
>         containerTemplate {
>           name 'jnlp'
>           image
> '${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/jenkins/jnlp-slave:${AGENT_IMAGE_TAG}
> <http://amazonaws.com/jenkins/jnlp-slave:$%7BAGENT_IMAGE_TAG%7D>'
>           resourceRequestCpu '200m'
>           resourceLimitCpu '1'
>           resourceRequestMemory '256Mi'
>           resourceLimitMemory '1Gi'
>         }
>       ]
>
>
> and various other combinations, but I can't get it to work.
>
> Anyone have any suggestions?
>
> Many thanks,
>
> Matt
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to jenkinsci-use...@googlegroups.com
> <mailto:jenkinsci-use...@googlegroups.com>.
> <https://groups.google.com/d/msgid/jenkinsci-users/c98259fd-1aa4-40a1-890d-cb157cf3bf5b%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Jenkins Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/jenkinsci-users/R_WpAZtl6qE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> jenkinsci-use...@googlegroups.com
> <mailto:jenkinsci-use...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-users/CALHFn6MkJMQsSudHjs55zADR%2BQWhJT_HCqZM_RBGKaDrScUmfQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/jenkinsci-users/CALHFn6MkJMQsSudHjs55zADR%2BQWhJT_HCqZM_RBGKaDrScUmfQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

Carlos Sanchez

unread,
Sep 8, 2018, 6:25:46 AM9/8/18
to Jenkins Users
yaml is not expanded with environment variables. You need to keep using containerTemplate (or 

containerTemplates {
  containerTemplate {
...
  }
} 

in the meantime

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/112abd15-c230-d306-ad50-0725bb28afd4%40fascinatedcow.net.

Matt Buckland

unread,
Sep 10, 2018, 4:53:01 AM9/10/18
to jenkins...@googlegroups.com
Thanks again Carlos. Unfortunately that syntax doesn't seem to work. I
have this:

environment {
IMAGE_TAG = "${env.IMAGE_TAG}"
JOB_NAME = "${env.JOB_NAME}"
AGENT_IMAGE_TAG = "${env.AGENT_IMAGE_TAG}"
ACCOUNT_ID = "${env.ACCOUNT_ID}"
}
agent {
kubernetes {
label "${JOB_NAME}"
inheritFrom 'default'
containerTemplates {
containerTemplate {
name 'jnlp'
image
'${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/jenkins/jnlp-slave:${AGENT_IMAGE_TAG}'
resourceRequestCpu '200m'
resourceLimitCpu '1'
resourceRequestMemory '256Mi'
resourceLimitMemory '1Gi'
}
}
}


but I get this when I try to run the pipeline. I think maybe generating
a static yaml is better.


[Pipeline] End of Pipeline
java.lang.UnsupportedOperationException: must specify $class with an
implementation of interface java.util.List
at
org.jenkinsci.plugins.structs.describable.DescribableModel.resolveClass(DescribableModel.java:474)
at
org.jenkinsci.plugins.structs.describable.DescribableModel.coerce(DescribableModel.java:401)
at
org.jenkinsci.plugins.structs.describable.DescribableModel.injectSetters(DescribableModel.java:360)
at
org.jenkinsci.plugins.structs.describable.DescribableModel.instantiate(DescribableModel.java:283)
at
org.jenkinsci.plugins.pipeline.modeldefinition.withscript.WithScriptDescriptor.newInstance(WithScriptDescriptor.java:74)
at
org.jenkinsci.plugins.pipeline.modeldefinition.agent.DeclarativeAgentDescriptor.instanceForDescriptor(DeclarativeAgentDescriptor.java:159)
at
org.jenkinsci.plugins.pipeline.modeldefinition.agent.DeclarativeAgentDescriptor$instanceForDescriptor$2.call(Unknown
Source)
at
org.jenkinsci.plugins.pipeline.modeldefinition.model.Agent.getDeclarativeAgent(Agent.groovy:81)
at sun.reflect.GeneratedMethodAccessor968.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at
org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at
com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
Caused: java.lang.IllegalArgumentException: Could not instantiate
{label=Redhat_SSO_Deploy, inheritFrom=default,
containerTemplates={containerTemplate={name=jnlp,
image=${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/jenkins/jnlp-slave:${AGENT_IMAGE_TAG},
resourceRequestCpu=200m, resourceLimitCpu=1,
resourceRequestMemory=256Mi, resourceLimitMemory=1Gi}}} for
KubernetesDeclarativeAgent(activeDeadlineSeconds?: int, cloud?: String,
containerTemplate?: ContainerTemplate(name?: String, image?: String,
alwaysPullImage?: boolean, args?: String, command?: String, envVars?:
TemplateEnvVar{ContainerEnvVar(key: String, value: String) |
KeyValueEnvVar(key: String, value: String) | PodEnvVar(key: String,
value: String) | SecretEnvVar(key: String, secretName: String,
secretKey: String)}[], livenessProbe?: ContainerLivenessProbe(execArgs:
String, timeoutSeconds: int, initialDelaySeconds: int, failureThreshold:
int, periodSeconds: int, successThreshold: int), ports?:
PortMapping(name?: String, containerPort?: int, hostPort?: int)[],
privileged?: boolean, resourceLimitCpu?: String, resourceLimitMemory?:
String, resourceRequestCpu?: String, resourceRequestMemory?: String,
shell?: String, ttyEnabled?: boolean, workingDir?: String),
containerTemplates?: ContainerTemplate(name?: String, image?: String,
alwaysPullImage?: boolean, args?: String, command?: String, envVars?:
TemplateEnvVar{ContainerEnvVar(key: String, value: String) |
KeyValueEnvVar(key: String, value: String) | PodEnvVar(key: String,
value: String) | SecretEnvVar(key: String, secretName: String,
secretKey: String)}[], livenessProbe?: ContainerLivenessProbe(execArgs:
String, timeoutSeconds: int, initialDelaySeconds: int, failureThreshold:
int, periodSeconds: int, successThreshold: int), ports?:
PortMapping(name?: String, containerPort?: int, hostPort?: int)[],
privileged?: boolean, resourceLimitCpu?: String, resourceLimitMemory?:
String, resourceRequestCpu?: String, resourceRequestMemory?: String,
shell?: String, ttyEnabled?: boolean, workingDir?: String)[],
defaultContainer?: String, idleMinutes?: int, inheritFrom?: String,
instanceCap?: int, label?: String, nodeSelector?: String,
serviceAccount?: String, workingDir?: String, yaml?: String, yamlFile?:
String)
at
org.jenkinsci.plugins.structs.describable.DescribableModel.instantiate(DescribableModel.java:286)
at
org.jenkinsci.plugins.pipeline.modeldefinition.withscript.WithScriptDescriptor.newInstance(WithScriptDescriptor.java:74)
at
org.jenkinsci.plugins.pipeline.modeldefinition.agent.DeclarativeAgentDescriptor.instanceForDescriptor(DeclarativeAgentDescriptor.java:159)
at
org.jenkinsci.plugins.pipeline.modeldefinition.agent.DeclarativeAgentDescriptor$instanceForDescriptor$2.call(Unknown
Source)
at
org.jenkinsci.plugins.pipeline.modeldefinition.model.Agent.getDeclarativeAgent(Agent.groovy:81)
at sun.reflect.GeneratedMethodAccessor968.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at
org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at
com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
at
org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:579)
at
org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.call(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:75)
at WorkflowScript.run(WorkflowScript:2)
at ___cps.transform___(Native Method)
at
com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
at
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
at
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
at sun.reflect.GeneratedMethodAccessor362.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at
com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:39)
at
com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
at
com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:28)
at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)
at com.cloudbees.groovy.cps.Next.step(Next.java:83)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
at
org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
at
org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
at
org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
at
org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
at
org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
at
org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
at
org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
at
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
at
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
at
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
at
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
at
org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
at
jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at
jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
Reply all
Reply to author
Forward
0 new messages