[JIRA] (JENKINS-55158) Hanging builds after completion after update from 1.40.1 to 1.41

alain.odea@gmail.com (JIRA)

unread,

Dec 12, 2018, 12:55:03 PM12/12/18

to jenkinsc...@googlegroups.com

Alain O'Dea created an issue

Jenkins /

JENKINS-55158

Hanging builds after completion after update from 1.40.1 to 1.41

Issue Type:	Bug
Assignee:	FABRIZIO MANFREDI
Components:	ec2-plugin
Created:	2018-12-12 17:53
Environment:	Jenkins 2.138.2 Amazon EC2 plugin:ec2:1.40.1 -> 1.41 Amazon ECR plugin:amazon-ecr:1.6 Amazon Web Services SDK:aws-java-sdk:1.11.457 Ant Plugin:ant:1.8 -> 1.9 Apache HttpComponents Client 4.x API Plugin:apache-httpcomponents-client-4-api:4.5.5-3.0 Artifactory Plugin:artifactory:2.16.2 Authentication Tokens API Plugin:authentication-tokens:1.3 Autofavorite for Blue Ocean:blueocean-autofavorite:1.2.2 AWS CodePipeline Plugin:aws-codepipeline:0.38 AWS Elastic Beanstalk Deployment Plugin:awseb-deployment-plugin:0.3.19 Basic Branch Build Strategies Plugin:basic-branch-build-strategies:1.1.1 Bitbucket Branch Source Plugin:cloudbees-bitbucket-branch-source:2.2.12 -> 2.2.15 Bitbucket Pipeline for Blue Ocean:blueocean-bitbucket-pipeline:1.9.0 Blue Ocean Core JS:blueocean-core-js:1.9.0 Blue Ocean Pipeline Editor:blueocean-pipeline-editor:1.9.0 Blue Ocean:blueocean:1.9.0 bouncycastle API Plugin:bouncycastle-api:2.17 Branch API Plugin:branch-api:2.0.20 -> 2.1.2 CloudBees AWS Credentials Plugin:aws-credentials:1.24 CloudBees Docker Build and Publish plugin:docker-build-publish:1.3.2 Cobertura Plugin:cobertura:1.13 Code Coverage API Plugin:code-coverage-api:1.0.5 Command Agent Launcher Plugin:command-launcher:1.2 Common API for Blue Ocean:blueocean-commons:1.9.0 Conditional BuildStep:conditional-buildstep:1.3.6 Config API for Blue Ocean:blueocean-config:1.9.0 Config File Provider Plugin:config-file-provider:3.3 -> 3.4.1 Copy Artifact Plugin:copyartifact:1.41 Credentials Binding Plugin:credentials-binding:1.16 -> 1.17 Credentials Plugin:credentials:2.1.18 Dashboard for Blue Ocean:blueocean-dashboard:1.9.0 Design Language:jenkins-design-language:1.9.0 Discard Old Build plugin:discard-old-build:1.05 Display URL API:display-url-api:2.3.0 Display URL for Blue Ocean:blueocean-display-url:2.2.0 Docker Commons Plugin:docker-commons:1.13 Docker Pipeline:docker-workflow:1.17 Durable Task Plugin:durable-task:1.26 -> 1.28 Email Extension Plugin:email-ext:2.63 EnvInject API Plugin:envinject-api:1.5 Events API for Blue Ocean:blueocean-events:1.9.0 External Monitor Job Type Plugin:external-monitor-job:1.7 Favorite:favorite:2.3.2 Folders Plugin:cloudbees-folder:6.7 Generic Webhook Trigger Plugin:generic-webhook-trigger:1.46 -> 1.50 Git client plugin:git-client:2.7.4 Git Pipeline for Blue Ocean:blueocean-git-pipeline:1.9.0 Git plugin:git:3.9.1 GIT server Plugin:git-server:1.7 GitHub API Plugin:github-api:1.95 GitHub Branch Source Plugin:github-branch-source:2.4.1 GitHub Pipeline for Blue Ocean:blueocean-github-pipeline:1.9.0 GitHub plugin:github:1.29.3 Gradle Plugin:gradle:1.29 Handy Uri Templates 2.x API Plugin:handy-uri-templates-2-api:2.1.6-1.0 HTML Publisher plugin:htmlpublisher:1.17 HTTP Request Plugin:http_request:1.8.22 i18n for Blue Ocean:blueocean-i18n:1.9.0 Ivy Plugin:ivy:1.28 Jackson 2 API Plugin:jackson2-api:2.8.11.3 -> 2.9.7.1 Javadoc Plugin:javadoc:1.4 JavaScript GUI Lib: ACE Editor bundle plugin:ace-editor:1.1 JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin:jquery-detached:1.2.1 JDK Tool Plugin:jdk-tool:1.2 JIRA Integration for Blue Ocean:blueocean-jira:1.9.0 JIRA plugin:jira:3.0.3 -> 3.0.5 JSch dependency plugin:jsch:0.1.54.2 JUnit Plugin:junit:1.26.1 JWT for Blue Ocean:blueocean-jwt:1.9.0 LDAP Plugin:ldap:1.20 Mailer Plugin:mailer:1.22 Matrix Authorization Strategy Plugin:matrix-auth:2.3 Matrix Project Plugin:matrix-project:1.13 Maven Integration plugin:maven-plugin:3.2 Mercurial plugin:mercurial:2.4 Node Iterator API Plugin:node-iterator-api:1.5.0 Outbound WebHook for build events:outbound-webhook:0.2.0 OWASP Markup Formatter Plugin:antisamy-markup-formatter:1.5 PAM Authentication plugin:pam-auth:1.4 Parallel Test Executor Plugin:parallel-test-executor:1.11 Parameterized Trigger plugin:parameterized-trigger:2.35.2 Personalization for Blue Ocean:blueocean-personalization:1.9.0 Pipeline Graph Analysis Plugin:pipeline-graph-analysis:1.9 Pipeline implementation for Blue Ocean:blueocean-pipeline-api-impl:1.9.0 Pipeline SCM API for Blue Ocean:blueocean-pipeline-scm-api:1.9.0 Pipeline Utility Steps:pipeline-utility-steps:2.1.0 -> 2.2.0 Pipeline: API:workflow-api:2.30 -> 2.33 Pipeline: Basic Steps:workflow-basic-steps:2.11 -> 2.13 Pipeline: Build Step:pipeline-build-step:2.7 Pipeline: Declarative Agent API:pipeline-model-declarative-agent:1.1.1 Pipeline: Declarative Extension Points API:pipeline-model-extensions:1.3.3 Pipeline: Declarative:pipeline-model-definition:1.3.3 Pipeline: Groovy:workflow-cps:2.59 -> 2.61 Pipeline: Input Step:pipeline-input-step:2.8 Pipeline: Job:workflow-job:2.25 -> 2.30 Pipeline: Milestone Step:pipeline-milestone-step:1.3.1 Pipeline: Model API:pipeline-model-api:1.3.3 Pipeline: Multibranch with defaults:pipeline-multibranch-defaults:2.0 Pipeline: Multibranch:workflow-multibranch:2.20 Pipeline: Nodes and Processes:workflow-durable-task-step:2.22 -> 2.26 Pipeline: REST API Plugin:pipeline-rest-api:2.10 Pipeline: SCM Step:workflow-scm-step:2.7 Pipeline: Shared Groovy Libraries:workflow-cps-global-lib:2.12 Pipeline: Stage Step:pipeline-stage-step:2.3 Pipeline: Stage Tags Metadata:pipeline-stage-tags-metadata:1.3.3 Pipeline: Step API:workflow-step-api:2.17 Pipeline: Supporting APIs:workflow-support:2.21 -> 2.23 Plain Credentials Plugin:plain-credentials:1.4 Priority Sorter Plugin:PrioritySorter:3.6.0 Pub-Sub "light" Bus:pubsub-light:1.12 Resource Disposer Plugin:resource-disposer:0.12 REST API for Blue Ocean:blueocean-rest:1.9.0 REST Implementation for Blue Ocean:blueocean-rest-impl:1.9.0 Role-based Authorization Strategy:role-strategy:2.9.0 Run Condition Plugin:run-condition:1.2 S3 publisher plugin:s3:0.11.3-SNAPSHOT (private-144b8119-andrew.burke) SAML Plugin:saml:1.1.0 -> 1.1.1 SCM API Plugin:scm-api:2.3.0 Script Security Plugin:script-security:1.47 -> 1.49 Server Sent Events (SSE) Gateway Plugin:sse-gateway:1.16 SonarQube Scanner for Jenkins:sonar:2.8.1 Splunk Plugin Extension:splunk-devops-extend:1.7.1 Splunk Plugin:splunk-devops:1.7.1 SSH Credentials Plugin:ssh-credentials:1.14 Structs Plugin:structs:1.17 Terraform Plugin:terraform:1.0.9 Test Results Analyzer Plugin:test-results-analyzer:0.3.5 Timestamper:timestamper:1.8.10 Token Macro Plugin:token-macro:2.5 user build vars plugin:build-user-vars-plugin:1.5 Variant Plugin:variant:1.1 Veracode Jenkins Plugin:veracode-jenkins-plugin:18.5.5.7 Verafin Veracode Plugin:verafin-veracode-plugin:1.1 Web for Blue Ocean:blueocean-web:1.9.0 Windows Slaves Plugin:windows-slaves:1.3.1 Workspace Cleanup Plugin:ws-cleanup:0.36
Labels:	regression
Priority:	Minor
Reporter:	Alain O'Dea

When we updated the plug-ins on Jenkins we noticed that builds were running, would complete their work, but would then stay in the queue after completion with stripes. The queue would then build up and not be processed. It seems like Jenkins lost contact with agents.

We have over 200 agents at peak use and 7 EC2 agent profiles currently.

We don't know the cause, but upgrading to Jenkins LTS 1.150.1 did not fix it, but downgrading Amazon EC2 plugin from 1.41 to 1.40.1 restored service and allowed our queue to complete.