| After upgrading from 1.6.2 to 1.7.0 no VMs were being created. Investigation showed the changes in https://issues.jenkins-ci.org/browse/JENKINS-49150 affected us. Updated the credential and verified the template successfully. Saved and then VMs were being created, but no machine actually succeeded in being deployed. Logs contain :
SEVERE: Failure creating provisioned agent 'jenkins-linux-8-core-build-slaveb939d10'
com.microsoft.azure.vmagent.exceptions.AzureCloudException: AzureVMCloud: createProvisionedAgent: Deployment Failed: Microsoft.Compute/virtualMachines:jenkins-linux-8-core-build-slaveb939d10 - Conflict - {error={code=OperationNotAllowed, message=Operation results in exceeding quota limits of Core. Maximum allowed: 350, Current in use: 344, Additional requested: 8. Please read more about quota increase at http://aka.ms/corequotaincrease.}}
at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:37)
at com.microsoft.azure.vmagent.AzureVMCloud.createProvisionedAgent(AzureVMCloud.java:612)
at com.microsoft.azure.vmagent.AzureVMCloud$4.call(AzureVMCloud.java:854)
at com.microsoft.azure.vmagent.AzureVMCloud$4.call(AzureVMCloud.java:832)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Apr 17, 2018 12:55:10 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists
INFO: AzureVMManagementServiceDelegate: virtualMachineExists: check for jenkins-linux-8-core-build-slaveb939d10
Apr 17, 2018 12:55:10 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists
INFO: AzureVMManagementServiceDelegate: virtualMachineExists: jenkins-linux-8-core-build-slaveb939d10 doesnt exist
Apr 17, 2018 12:55:10 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate terminateVirtualMachine
INFO: Clean operation starting for jenkins-linux-8-core-build-slaveb939d10 NIC and IP
Apr 17, 2018 12:55:10 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate removeIPName
INFO: Remove NIC jenkins-linux-8-core-build-slaveb939d10NIC
Apr 17, 2018 12:55:10 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate removeIPName
INFO: Remove IP jenkins-linux-8-core-build-slaveb939d10IPName
Apr 17, 2018 12:55:18 PM hudson.slaves.NodeProvisioner$2 run
WARNING: Unexpected exception encountered while provisioning agent jenkins-linux-8-core-build-slave
com.microsoft.azure.vmagent.exceptions.AzureCloudException: AzureVMCloud: createProvisionedAgent: Deployment Failed: Microsoft.Compute/virtualMachines:jenkins-linux-8-core-build-slaveb939d10 - Conflict - {error={code=OperationNotAllowed, message=Operation results in exceeding quota limits of Core. Maximum allowed: 350, Current in use: 344, Additional requested: 8. Please read more about quota increase at http://aka.ms/corequotaincrease.}}
at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:37)
at com.microsoft.azure.vmagent.AzureVMCloud.createProvisionedAgent(AzureVMCloud.java:612)
at com.microsoft.azure.vmagent.AzureVMCloud$4.call(AzureVMCloud.java:854)
at com.microsoft.azure.vmagent.AzureVMCloud$4.call(AzureVMCloud.java:832)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
This happened for all VMs being provisioned. After some minutes I disabled the plugin by changing the available labels to not include the requested label. I have also rolled back the plugin and this removed all my cloud configuration so had to recreate from scratch. Now the machines are starting up, but as soon as it's get added to jenkins, it gets (offline) (suspended) and deleted after a couple of minutes. Logs :
Apr 17, 2018 6:17:13 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate createDeployment
INFO: AzureVMManagementServiceDelegate: createDeployment: Creating a new deployment jenkins-linux-8-core-build-slave0417181713767 with VM base name jenkins-linux-8-core-build-slave5a151
r 17, 2018 6:17:14 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$DeploymentRegistrar registerDeployment
INFO: AzureVMAgentCleanUpTask: registerDeployment: Registering deployment jenkins-linux-8-core-build-slave0417181713767 in jenkins-slaves
Apr 17, 2018 6:17:26 PM com.microsoft.azure.vmagent.AzureVMCloud createProvisionedAgent
INFO: AzureVMCloud: createProvisionedAgent: Waiting for deployment jenkins-linux-8-core-build-slave0417181713767 to be completed
Apr 17, 2018 6:18:28 PM com.microsoft.azure.vmagent.AzureVMCloud createProvisionedAgent
INFO: AzureVMCloud: createProvisionedAgent: Deployment jenkins-linux-8-core-build-slave0417181713767 not yet finished (Running): Microsoft.Compute/virtualMachines:jenkins-linux-8-core-build-slave5a1510 - waited 60 seconds
Apr 17, 2018 6:19:16 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments
INFO: AzureVMAgentCleanUpTask: cleanDeployments: Checking deployment jenkins-linux-8-core-build-slave0417181713767
Apr 17, 2018 6:24:07 PM com.microsoft.azure.vmagent.AzureVMCloud createProvisionedAgent
INFO: AzureVMCloud: createProvisionedAgent: VM available: jenkins-linux-8-core-build-slave5a1510
Apr 17, 2018 6:24:07 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate parseResponse
INFO: AzureVMManagementServiceDelegate: parseDeploymentResponse:
found agent jenkins-linux-8-core-build-slave5a1510
OS type Linux
number of executors 1
Apr 17, 2018 6:24:31 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate setVirtualMachineDetails
INFO: Azure agent details:
nodeNamejenkins-linux-8-core-build-slave5a1510
adminUserName=be14c538-86a6-42db-b490-c3e936021e89
shutdownOnIdle=false
retentionTimeInMin=0
labels=linux-8core-scalable
Apr 17, 2018 6:24:31 PM com.microsoft.azure.vmagent.AzureVMCloud$4 call
INFO: Azure Cloud: provision: Adding agent jenkins-linux-8-core-build-slave5a1510 to Jenkins nodes
Apr 17, 2018 6:24:31 PM com.microsoft.azure.vmagent.AzureVMAgent createComputer
INFO: AzureVMAgent: createComputer: start for agent jenkins-linux-8-core-build-slave5a1510
Apr 17, 2018 6:24:31 PM com.microsoft.azure.vmagent.AzureVMCloudOnceRetentionStrategy start
INFO: AzureVMCloudOnceRetentionStrategy: start: azureComputer name jenkins-linux-8-core-build-slave5a1510
Apr 17, 2018 6:24:31 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch
INFO: AzureVMAgentSSHLauncher: launch: launch method called for agent jenkins-linux-8-core-build-slave5a1510
Apr 17, 2018 6:24:55 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher getRemoteSession
INFO: AzureVMAgentSSHLauncher: getRemoteSession: getting remote session for user jenkins to host jenkins-linux-8-core-build-slave5a1510.westeurope.cloudapp.azure.com:22
Apr 17, 2018 6:24:56 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher getRemoteSession
INFO: AzureVMAgentSSHLauncher: getRemoteSession: Got remote session for user jenkins to host jenkins-linux-8-core-build-slave5a1510.westeurope.cloudapp.azure.com:22
Apr 17, 2018 6:25:39 PM com.microsoft.azure.vmagent.AzureVMCloudOnceRetentionStrategy done
INFO: AzureVMCloudOnceRetentionStrategy: Tagging VM to delete when idle: jenkins-linux-8-core-build-slave5a1510
Apr 17, 2018 6:27:13 PM com.microsoft.azure.vmagent.AzureVMCloudOnceRetentionStrategy done
INFO: AzureVMCloudOnceRetentionStrategy: Tagging VM to delete when idle: jenkins-linux-8-core-build-slave5a1510
Apr 17, 2018 6:29:13 PM com.microsoft.azure.vmagent.AzureVMCloudOnceRetentionStrategy done
INFO: AzureVMCloudOnceRetentionStrategy: Tagging VM to delete when idle: jenkins-linux-8-core-build-slave5a1510
Apr 17, 2018 6:29:16 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists
INFO: AzureVMManagementServiceDelegate: virtualMachineExists: check for jenkins-linux-8-core-build-slave5a1510
Apr 17, 2018 6:29:16 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists
INFO: AzureVMManagementServiceDelegate: virtualMachineExists: jenkins-linux-8-core-build-slave5a1510 exists
Apr 17, 2018 6:29:16 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2 call
INFO: AzureVMAgentCleanUpTask: cleanVMs: deleting jenkins-linux-8-core-build-slave5a1510
Apr 17, 2018 6:29:16 PM com.microsoft.azure.vmagent.AzureVMAgent deprovision
INFO: AzureVMAgent: deprovision: Deprovision called for agent jenkins-linux-8-core-build-slave5a1510, for reason: Delete agent after build execution (when idle).
Apr 17, 2018 6:29:16 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists
INFO: AzureVMManagementServiceDelegate: virtualMachineExists: check for jenkins-linux-8-core-build-slave5a1510
Apr 17, 2018 6:29:16 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists
INFO: AzureVMManagementServiceDelegate: virtualMachineExists: jenkins-linux-8-core-build-slave5a1510 exists
Apr 17, 2018 6:29:28 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate terminateVirtualMachine
This means our azure infrastructure is not being utilised. |