slaves not using all available cpus

[Santiago Baldassin]

Hi guys,

I'm writing to you because we are having issues making our jenkins slaves to use all available resources when running a build. 

We have a master-slave deployment with jenkins master running in a t2.large instance type in AWS (2 vCPU and 8GB of RAM) and the slaves running in m4.xlarge instance type (4 vCPUs and 16GB of RAM)

the build consist of creating a docker-compose cluster (8 containers, including elasticsearch, psql, and a apache storm application) and executing a python test against that cluster

We rule out any issue related to the tests themselves or docker/docker-compose by running the test manually on the jenkins slave. The thing is that when running the test manually in the slave, the test runs in 5 minutes taking all the 4 vCPU to 100% while running the the build takes 25 minutes and only 1 of the vCPU is taken to 100% 

Is there any specific setting that we need to setup in Jenkins so the slave use all the available CPUs?

The number of executors is 1

Thanks in advance

[Viacheslav Dubrovskyi]

25.08.2017 21:36, Santiago Baldassin пишет:

Hi guys,

I'm writing to you because we are having issues making our jenkins slaves to use all available resources when running a build. 

We have a master-slave deployment with jenkins master running in a t2.large instance type in AWS (2 vCPU and 8GB of RAM) and the slaves running in m4.xlarge instance type (4 vCPUs and 16GB of RAM)

the build consist of creating a docker-compose cluster (8 containers, including elasticsearch, psql, and a apache storm application) and executing a python test against that cluster

We rule out any issue related to the tests themselves or docker/docker-compose by running the test manually on the jenkins slave. The thing is that when running the test manually in the slave, the test runs in 5 minutes taking all the 4 vCPU to 100% while running the the build takes 25 minutes and only 1 of the vCPU is taken to 100%

Did you run it manually under the same user which used for jenkins?

Is there any specific setting that we need to setup in Jenkins so the slave use all the available CPUs?

IMHO no. All settings related to user used for run jenkins slave.

-- 
WBR,
Viacheslav Dubrovskyi

[Santiago Baldassin]

Hi,

On Saturday, August 26, 2017 at 4:15:09 PM UTC-3, Slava Dubrovskiy wrote:

25.08.2017 21:36, Santiago Baldassin пишет:

Hi guys,

I'm writing to you because we are having issues making our jenkins slaves to use all available resources when running a build. 

We have a master-slave deployment with jenkins master running in a t2.large instance type in AWS (2 vCPU and 8GB of RAM) and the slaves running in m4.xlarge instance type (4 vCPUs and 16GB of RAM)

the build consist of creating a docker-compose cluster (8 containers, including elasticsearch, psql, and a apache storm application) and executing a python test against that cluster

We rule out any issue related to the tests themselves or docker/docker-compose by running the test manually on the jenkins slave. The thing is that when running the test manually in the slave, the test runs in 5 minutes taking all the 4 vCPU to 100% while running the the build takes 25 minutes and only 1 of the vCPU is taken to 100%

Did you run it manually under the same user which used for jenkins?

Yes. Ran it manually using the same user and all cpus are takin to 100%. 

One thing I forgot to mention is that all these is ran through a Jenkinsfile which is pretty simple:

pipeline {

    agent any 

    stages {

        stage('Checkout') { 

            steps { 

                git branch: 'system_test', credentialsId: 'cred', url: 'https://github.com/repo.git'

            }

        }

        stage('System'){

            steps {

                sh "script/system"    

            }

        }

    }

}

At the same time we tried creating an old school job without using pipelines with bash step that runs script/system and in this case all cpus are taken to 100%

so running script/system as part of a Jenkinsfile pipeline, does not take the CPUs to 100% but running it with a jenkins bash step does

script/system is a bash script that does the following

docker-compose up

docker-compose exec container python3 -m unittest -s tests
docker-compose down

Thoughts?