Jenkins pod provisioning takes too long

[Denis Mone]

Hello everyone.
At my company we have been using Jenkins for our CI infrastructure and recently we decided to start using k8s in order to provisions agents there.
So i have installed the kubernetes-plugin configured the access to our autopilot k8s cluster and created a pipeline job that builds a maven based project.
I am pasting the pipeline script below:

    podTemplate(cloud: 'autopilot',
    yaml: '''
        apiVersion: v1
        kind: Pod
        metadata:
          labels:
        type: maven
        spec:
          volumes:
        - name: deps
          persistentVolumeClaim:
            claimName: maven-repo
          serviceAccount: jenkins-admin
          containers:
          - name: maven
        image: maven:3.8.1-jdk-8
        command:
        - sleep
        args:
        - 99d
        tty: true
        resources:
        limits:
          cpu: 2000m
          memory: 4096Mi
        requests:
          cpu: 100m
          memory: 512Mi
        volumeMounts:
        - name: deps
          mountPath: /root/.m2/repository
        ''') {
    timestamps {
        node(POD_LABEL) {
          stage('Build a Maven project') {
          sh "env"
            checkout(scm: [$class: 'GitSCM',
                branches: [[name: "master"]],
                doGenerateSubmoduleConfigurations: false,
                extensions: [[$class: 'CleanBeforeCheckout']],
                submoduleCfg: [],
                userRemoteConfigs: [
                    [credentialsId: 'git-checkout-cred', name: 'origin', url: 'g...@github.com<my-git-repo>.git']]
                ]
            )
        container('maven') {
          sh "ls -la"
          sh "pwd"
          sh(script: 'mvn clean package -DskipTests')
        }
          }
        }
    }
    }

Note the i have pre-created the maven-repo pvc before invoking the pipeline.
The pipeline runs fine, it downloads the dependencies and build the project but the time it takes is high in proportion to the tasks it is executing.
For instance it the build itself takes a less than a minute (dependencies are cached) but the provisioning of the pod takes almost 3 minutes.

I am attaching some images that demonstrate the above statements

I am also attaching a build log.

What can i do in order to speedup the provisioning of the pods?

PS Jenkins control plane is on a VM outside of K8s.

[Ivan Fernandez Calvo]

Did you try to provision the same pod definition manually in the K8s cluster to check how much time takes k8s to create the pod? Which k8s provider do you use? I have a similar setup on GKE and it takes less than 30 seconds to provision an agent the fisrt time, when the Docker image is in the k8s cluster it takes lesst than 20 seconds.

[Denis Mone]

I have tried to provision the pod manually. It takes around 30 - 35 seconds.

We use GKE as well as the k8s provider.

Do you have specific configuration of the kubernetes-plugin you can share. I find it interesting that with the same setup we have such a big difference in provisioning times.

[kuisathaverat]

here you have some of the pod templates we use 

https://github.com/elastic/apm-pipeline-library/pull/1614

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/HTHsFE7kHxw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f6e4fecb-1106-4c0a-a577-5397927bcb70n%40googlegroups.com.

--

Un Saludo

Iván Fernández Calvo

https://www.linkedin.com/in/iv%C3%A1n-fern%C3%A1ndez-calvo-21425033