[JIRA] (JENKINS-57204) Provision failures does not clean up deployments

8 views
Skip to first unread message

rtyler@brokenco.de (JIRA)

unread,
Apr 27, 2019, 11:27:02 AM4/27/19
to jenkinsc...@googlegroups.com
R. Tyler Croy created an issue
 
Jenkins / Bug JENKINS-57204
Provision failures does not clean up deployments
Issue Type: Bug Bug
Assignee: Azure DevOps
Components: azure-container-agents-plugin
Created: 2019-04-27 15:26
Priority: Critical Critical
Reporter: R. Tyler Croy

We're seeing a number of failures on ci.jenkins.io which stem from Azure Container Agents failing to provision. It looks like failed ACI deployments don't get cleaned up automatically by the plugin.

ACI is returning the following failure:

{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.","details":[{"code":"Conflict","message":"{\r\n \"error\": {\r\n \"code\": \"ServiceUnavailable\",\r\n \"message\": \"The requested resource is not available in the location 'eastus2' at this moment. Please retry with a different resource request or in another location. Resource requested: '2' CPU '4' GB memory 'Linux' OS\"\r\n }\r\n}"}]}

Which is annoying, but whatever. The bug here is that the plugin does not clean up this failed deployment, so we have exceeded our quota of Deployments of 800, all failures like this one!

I recall this exact same issue in the Azure VM Agents plugin at one point as well.

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

vscjenkins@microsoft.com (JIRA)

unread,
Apr 27, 2019, 9:24:03 PM4/27/19
to jenkinsc...@googlegroups.com
Azure DevOps started work on Bug JENKINS-57204
 
Change By: Azure DevOps
Status: Open In Progress

vscjenkins@microsoft.com (JIRA)

unread,
Apr 27, 2019, 9:24:03 PM4/27/19
to jenkinsc...@googlegroups.com
Azure DevOps assigned an issue to Jie Shen
Change By: Azure DevOps
Assignee: Azure DevOps Jie Shen

jieshe@microsoft.com (JIRA)

unread,
Apr 27, 2019, 9:57:02 PM4/27/19
to jenkinsc...@googlegroups.com
Jie Shen commented on Bug JENKINS-57204
 
Re: Provision failures does not clean up deployments

Thanks for reporting this issue. It is similar as JENKINS-48045 in VM agent plugin. I will make a PR soon to fix this.

jieshe@microsoft.com (JIRA)

unread,
Jul 25, 2019, 4:00:02 AM7/25/19
to jenkinsc...@googlegroups.com
Jie Shen resolved as Fixed
 
Change By: Jie Shen
Status: In Progress Resolved
Resolution: Fixed
Released As: 1.1.0

olivier@vernin.me (JIRA)

unread,
Oct 21, 2019, 6:58:03 AM10/21/19
to jenkinsc...@googlegroups.com
Change By: Olivier Vernin
Resolution: Fixed
Status: Resolved In Review
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

olivier@vernin.me (JIRA)

unread,
Oct 21, 2019, 7:00:05 AM10/21/19
to jenkinsc...@googlegroups.com

olivier@vernin.me (JIRA)

unread,
Oct 21, 2019, 7:02:03 AM10/21/19
to jenkinsc...@googlegroups.com
Olivier Vernin commented on Bug JENKINS-57204
 
Re: Provision failures does not clean up deployments

It appears this issue is back on ci.jenkins.io, for some reason, deployments are not cleaned up

2019-10-21 10:50:55.713+0000 [id=55] WARNING hudson.slaves.NodeProvisioner#lambda$update$6: Unexpected exception encountered while provisioning agent aci-maven
java.lang.Exception: Status code 409, {"error":{"code":"DeploymentQuotaExceeded","message":"Creating the deployment 'aci-maven-h3c19t5n' would exceed the quota of '800'. The current deployment count is '800', please delete some deployments before creating a new one. Please see https://aka.ms/arm-debug for usage details."}}
at com.microsoft.jenkins.containeragents.aci.AciService.createDeployment(AciService.java:141)
at com.microsoft.jenkins.containeragents.aci.AciContainerTemplate.provisionAgents(AciContainerTemplate.java:128)
at com.microsoft.jenkins.containeragents.aci.AciCloud$1.call(AciCloud.java:104)
Caused: java.lang.Exception
at com.microsoft.jenkins.containeragents.aci.AciCloud$1.call(AciCloud.java:140)
at com.microsoft.jenkins.containeragents.aci.AciCloud$1.call(AciCloud.java:80)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

jieshe@microsoft.com (JIRA)

unread,
Oct 23, 2019, 2:32:03 AM10/23/19
to jenkinsc...@googlegroups.com
Jie Shen commented on Bug JENKINS-57204

I was trying to fix this at https://github.com/jenkinsci/azure-container-agents-plugin/pull/42 . The fix has been released in 1.1.0. If it does not work, is there any log for why the cleanup action fails?

Reply all
Reply to author
Forward
0 new messages