[JIRA] (JENKINS-59696) support-core and thus advisor require to restart the instance to be activated

0 views
Skip to first unread message

aheritier@apache.org (JIRA)

unread,
Oct 16, 2019, 3:31:02 AM10/16/19
to jenkinsc...@googlegroups.com
Arnaud Héritier updated an issue
 
Jenkins / Bug JENKINS-59696
support-core and thus advisor require to restart the instance to be activated
Change By: Arnaud Héritier
Summary: support-core requires and thus advisor require to restart the instance to be activated
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

aheritier@apache.org (JIRA)

unread,
Oct 16, 2019, 3:50:03 AM10/16/19
to jenkinsc...@googlegroups.com
Arnaud Héritier assigned an issue to Unassigned
Change By: Arnaud Héritier
Assignee: Arnaud Héritier

bmathus+ossjira@cloudbees.com (JIRA)

unread,
Oct 16, 2019, 10:14:02 AM10/16/19
to jenkinsc...@googlegroups.com
Baptiste Mathus updated an issue
Change By: Baptiste Mathus
Labels: foundation-triage

rcampbell@cloudbees.com (JIRA)

unread,
Oct 16, 2019, 12:38:03 PM10/16/19
to jenkinsc...@googlegroups.com
Ryan Campbell updated an issue
Change By: Ryan Campbell
Component/s: cloudbees-jenkins-advisor-plugin

aburdajewicz@cloudbees.com (JIRA)

unread,
Oct 23, 2019, 12:50:02 AM10/23/19
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ assigned an issue to Allan BURDAJEWICZ
Change By: Allan BURDAJEWICZ
Assignee: Allan BURDAJEWICZ

aburdajewicz@cloudbees.com (JIRA)

unread,
Oct 23, 2019, 12:50:03 AM10/23/19
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ commented on Bug JENKINS-59696
 
Re: support-core and thus advisor require to restart the instance to be activated

Tested the following which confirms that Support Core is the culprit:

  • Use fresh install of LTS 2.138.4
  • Installed all Support Core dependencies from the Plugin Manager (installing Metrics plugin and Credentials plugin is enough)
  • Restart Jenkins
  • Install Support Core from the Plugin Manager
  • The exception shows up

aburdajewicz@cloudbees.com (JIRA)

unread,
Oct 23, 2019, 1:22:03 AM10/23/19
to jenkinsc...@googlegroups.com

Cannot reproduced in version 2.201 (current latest) of core.
Can reproduce in version 2.193 of core.

This problem seems to be fixed (rather avoided) wince the fix https://github.com/jenkinsci/jenkins/commit/5169f14019be77d6b205b7c2d5fe608b92ed3446#diff-c06f61bb7c43d48aa0554204a18832e6. But we may want to understand why this fails in previous versions.

aburdajewicz@cloudbees.com (JIRA)

unread,
Oct 23, 2019, 1:24:02 AM10/23/19
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ edited a comment on Bug JENKINS-59696
Cannot reproduced in version 2.201 (current latest) of core.
Can reproduce in version 2.193 of core.

This problem seems to be _fixed_ (rather avoided) wince since the fix https://github.com/jenkinsci/jenkins/commit/5169f14019be77d6b205b7c2d5fe608b92ed3446#diff-c06f61bb7c43d48aa0554204a18832e6. But we may want to understand why this fails in previous versions.

aburdajewicz@cloudbees.com (JIRA)

unread,
Oct 23, 2019, 1:24:03 AM10/23/19
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ updated an issue
 
Change By: Allan BURDAJEWICZ
Attachment: 2.193.loading-issues.log

aburdajewicz@cloudbees.com (JIRA)

unread,
Oct 23, 2019, 2:37:03 AM10/23/19
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ edited a comment on Bug JENKINS-59696
Cannot reproduced in version 2.201 (current latest) of core.
Can reproduce in version 2.193 of core.

This problem seems to be _fixed_ (rather avoided) since in the fix https://github latest version . com/jenkinsci/jenkins/commit/5169f14019be77d6b205b7c2d5fe608b92ed3446#diff-c06f61bb7c43d48aa0554204a18832e6. But we may want to understand why this fails in previous versions to know if this is a bug in {{support-core}} or in {{core}} .

aburdajewicz@cloudbees.com (JIRA)

unread,
Oct 23, 2019, 3:35:02 AM10/23/19
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ edited a comment on Bug JENKINS-59696
Cannot reproduced Can reproduce in version core 2.201 but not consistently. Sometimes it happens, sometimes no. To reproduce it consistently, I launch the plugin install and refresh the {{$JENKINS_URL/support}} aggressively ( current latest but manually : ) of core ) .
Can reproduce in version 2

I wonder if there is a race condition somewhere . 193 Maybe one of the Support Core component (like the [SlowRequestChecker|https://github.com/jenkinsci/support- core -plugin/blob/support-core-2 .

This problem seems to be _fixed_ (rather avoided 62/src/main/java/com/cloudbees/jenkins/support/slowrequest/SlowRequestChecker.java] ) in starts before the latest version. But we want PluginManager has finished to understand why this fails in previous versions to know if this is a bug in load the plugin dynamically, which causes the same extension of {{ support-core ContentFilters }} or in to be loaded twice (In the ExtensionList, the 2 instances of {{ core ContentFilters }} are the same object) .

Maybe _SlowRequestChecker_ starts too soon.

aburdajewicz@cloudbees.com (JIRA)

unread,
Oct 23, 2019, 3:56:03 AM10/23/19
to jenkinsc...@googlegroups.com

Maybe we should add a condition on

{ContentFilters#get}

to check instance.getInitLevel().compareTo(InitMilestone.EXTENSIONS_AUGMENTED) >= 0)

aburdajewicz@cloudbees.com (JIRA)

unread,
Oct 24, 2019, 4:18:04 AM10/24/19
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ edited a comment on Bug JENKINS-59696
Maybe we should add a condition on Adding fine logging in ExtensionList, I found out that the extensions may be added twice when loading the { {DeadlockTrackChecker}} because of the call from [DeadlockTrackChecker|https://github.com/jenkinsci/support-core-plugin/blob/support-core-2.62/src/main/java/com/cloudbees/jenkins/support/timer/DeadlockTrackChecker.java#L30]. Which could be avoid there. But another call that could be causing this is the call to [ExtensionList.lookupSingleton|https://github.com/jenkinsci/support-core-plugin/blob/support-core-2.62/src/main/java/com/cloudbees/jenkins/support/filter/ ContentFilters .java # get L47] that can cause the {{ExtensionList } } to check be created is it does not exist yet:

 { code}
return ExtensionList.lookupSingleton(ContentFilters.class);
 { instance code}

A fix I have been testing is to do this instead:

{code}
return GlobalConfiguration . getInitLevel all (). compareTo get ( InitMilestone ContentFilters . EXTENSIONS_AUGMENTED class ) >= 0) ;
{code }

I have not been able to reproduce the issue using this method. This method goes through a different code path to get the component.

****

I have added logging to the ExtensionsList class in this branch: https://github.com/Dohbedoh/jenkins/commit/0bf0178e43b8b8b42c82ba0938b4c53c2ead0c6d

I noticed that when the issue happens:

* the extension gets created when loading the {{DeadlockTrackChecker } } going through: https://github.com/Dohbedoh/jenkins/blob/JENKINS-59696/core/src/main/java/jenkins/model/Jenkins.java#L2641.
* then the {{CompleteBatchJob}} refreshes the extensions and add it again [through the ExtensionList#refresh|https://github.com/Dohbedoh/jenkins/blob/JENKINS-59696/core/src/main/java/hudson/ExtensionList.java#L359-L360]

I have attached logs in [^extension-loading-with-issue.log]

aburdajewicz@cloudbees.com (JIRA)

unread,
Oct 24, 2019, 4:29:03 AM10/24/19
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ started work on Bug JENKINS-59696
 
Change By: Allan BURDAJEWICZ
Status: Open In Progress

aburdajewicz@cloudbees.com (JIRA)

unread,
Oct 24, 2019, 4:29:03 AM10/24/19
to jenkinsc...@googlegroups.com
Change By: Allan BURDAJEWICZ
Status: In Progress Review

aburdajewicz@cloudbees.com (JIRA)

unread,
Oct 24, 2019, 7:03:02 AM10/24/19
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ edited a comment on Bug JENKINS-59696
 
Re: support-core and thus advisor require to restart the instance to be activated
Can reproduce in core 2.201 but not consistently. Sometimes it happens, sometimes no. To reproduce it consistently, I launch the plugin install and refresh the main page or the {{$JENKINS_URL/support}} aggressively (but manually :) ).

I wonder if there is a race condition somewhere. Maybe one of the Support Core component (like the [SlowRequestChecker|https://github.com/jenkinsci/support-core-plugin/blob/support-core-2.62/src/main/java/com/cloudbees/jenkins/support/slowrequest/SlowRequestChecker.java]) starts before the PluginManager has finished to load the plugin dynamically, which causes the same extension of {{ContentFilters}} to be loaded twice (In the ExtensionList, the 2 instances of {{ContentFilters}} are the same object).

Maybe _SlowRequestChecker_ starts too soon.

aheritier@apache.org (JIRA)

unread,
Oct 25, 2019, 11:32:03 AM10/25/19
to jenkinsc...@googlegroups.com

If you are facing this issue and cannot restart your instance, disable the logger which is generating these logs up until you can restart.

 

 

aheritier@apache.org (JIRA)

unread,
Oct 25, 2019, 11:32:03 AM10/25/19
to jenkinsc...@googlegroups.com
Arnaud Héritier updated an issue
Change By: Arnaud Héritier
Attachment: Screenshot 2019-10-25 17.28.38.png

aheritier@apache.org (JIRA)

unread,
Nov 6, 2019, 4:47:02 AM11/6/19
to jenkinsc...@googlegroups.com
Change By: Arnaud Héritier
Status: Resolved Fixed but Unreleased

aheritier@apache.org (JIRA)

unread,
Nov 6, 2019, 4:47:04 AM11/6/19
to jenkinsc...@googlegroups.com
Change By: Arnaud Héritier
Status: In Review Resolved
Resolution: Fixed
Released As: support-core-2.63

aheritier@apache.org (JIRA)

unread,
Nov 6, 2019, 4:49:02 AM11/6/19
to jenkinsc...@googlegroups.com
 
Re: support-core and thus advisor require to restart the instance to be activated

I confirm. I tested with 2.138.4 and 2.190.2 and the problem isn't here.
With 2.190.2 I hit JENKINS-60073 / JENKINS-59775 which is annoying but not critical

aheritier@apache.org (JIRA)

unread,
Nov 6, 2019, 5:14:03 AM11/6/19
to jenkinsc...@googlegroups.com
Change By: Arnaud Héritier
Status: Fixed but Unreleased Closed
Reply all
Reply to author
Forward
0 new messages