[JIRA] (JENKINS-50790) GitPluginTest are failing in the ATH

2 views
Skip to first unread message

rarabaolaza@cloudbees.com (JIRA)

unread,
Apr 13, 2018, 12:14:02 PM4/13/18
to jenkinsc...@googlegroups.com
Raul Arabaolaza updated an issue
 
Jenkins / Task JENKINS-50790
GitPluginTest are failing in the ATH
Change By: Raul Arabaolaza
Summary: GitPluginTest is are failing in the ATH
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

rarabaolaza@cloudbees.com (JIRA)

unread,
Apr 13, 2018, 12:20:02 PM4/13/18
to jenkinsc...@googlegroups.com
Raul Arabaolaza commented on Task JENKINS-50790
 
Re: GitPluginTest are failing in the ATH

It seems the problem is a timeout trying to load the available plugins page, I am going to debug locally with the docker image to get an idea of what is happening, but seems not only related to git plugin but also other plugins in the last ATH runs

rarabaolaza@cloudbees.com (JIRA)

unread,
Apr 13, 2018, 12:30:02 PM4/13/18
to jenkinsc...@googlegroups.com
Raul Arabaolaza edited a comment on Task JENKINS-50790
It seems the problem is a timeout trying to load the available plugins page, I am going to debug locally with the docker image to get an idea of what is happening, but seems not only related to git plugin but also other plugins in the last ATH runs


 

Confirmed that is failing when trying to install plugins via {{@WithPlugins}} rule

rarabaolaza@cloudbees.com (JIRA)

unread,
Apr 13, 2018, 12:41:02 PM4/13/18
to jenkinsc...@googlegroups.com
Raul Arabaolaza edited a comment on Task JENKINS-50790
It seems the problem is a timeout trying to load the available plugins page, I am going to debug locally with the docker image to get an idea of what is happening, but seems not only related to git plugin but also other plugins in the last ATH runs

 

Confirmed that is failing when trying to install plugins via {{@WithPlugins}} rule , after adding the MockUpdateCenter and clicking check now it tries to visit the available plugins page and there is where the timeout happens, it seems it takes too much to load, need to properly confirm that but seems a performance issue

rarabaolaza@cloudbees.com (JIRA)

unread,
Apr 16, 2018, 5:13:01 AM4/16/18
to jenkinsc...@googlegroups.com

Confirmed the performance issue, by increasing the load timeout the tests pass, I have also checked that loading the list of available plugins takes a lot of time, that is the reason for the timeout, checking now if I can reduce that load time as I do not like to increase timeouts. The fact that it works perfectly fine without the local container may indicate that some extra resources are needed and not a change in the code.

cc Oliver Gondža

rarabaolaza@cloudbees.com (JIRA)

unread,
Apr 16, 2018, 5:20:01 AM4/16/18
to jenkinsc...@googlegroups.com
Raul Arabaolaza edited a comment on Task JENKINS-50790
Confirmed the performance issue, by increasing the load timeout the tests pass, I have also checked that loading the list of available plugins takes a lot of time, that is the reason for the timeout, checking now if I can reduce that load time as I do not like to increase timeouts. The fact that it works perfectly fine without the local container may indicate that some extra resources are needed and not a change in the code.
 

This is probably impacting other tests also

 cc [~olivergondza]

ogondza@gmail.com (JIRA)

unread,
Apr 16, 2018, 5:22:03 AM4/16/18
to jenkinsc...@googlegroups.com

Right, a lot of the tests started to fail once moved from jenkins.ci.cloudbees.com to ci.jenkins.io (ATH is run in container now): https://ci.jenkins.io/job/Core/job/jenkins/job/stable-2.107/lastCompletedBuild/testReport/.

 

If you do have the resources to investigate and fix, it would be great. If not, I am ok increasing the timeout to get reliable build.

rarabaolaza@cloudbees.com (JIRA)

unread,
Apr 16, 2018, 5:26:03 AM4/16/18
to jenkinsc...@googlegroups.com

I will try to investigate and fix if possible and will increase timeout as a last resort, thanks!

rarabaolaza@cloudbees.com (JIRA)

unread,
Apr 16, 2018, 7:32:01 AM4/16/18
to jenkinsc...@googlegroups.com

Trying to use `ElasticTime.factor` to increase the timeout without impacting the java code, if this works I am going to create a PR to the ath-container script to make sure the tests in the container are run with the appropriate factor

ogondza@gmail.com (JIRA)

unread,
Apr 16, 2018, 7:39:02 AM4/16/18
to jenkinsc...@googlegroups.com

Do we have an idea if everything is slowed down or just the plugin page? That would suggest if your approach is the correct one or not...

rarabaolaza@cloudbees.com (JIRA)

unread,
Apr 16, 2018, 7:42:01 AM4/16/18
to jenkinsc...@googlegroups.com

The ElasticTime.factor seems to fix the things for me locally, also GitPluginTest#create_tag_for_build is failing just because there is no git username and email defined inside the container.

I believe I can do a PR to solve this once I get rid of some urgent thing so probably nothing until tomorrow or next day 

rarabaolaza@cloudbees.com (JIRA)

unread,
Apr 17, 2018, 4:30:02 AM4/17/18
to jenkinsc...@googlegroups.com

Oliver Gondža You are right, for the moment the only slowness I have found is located in the plugin management page in the available section, so increasing all timeouts via ElasticTime.factor is probably not the best idea, I am going to just increase the page load timeout temporarily on that part and see if this still works, if it doesn't then I will keep on the ElasticTime.factor approach  

rarabaolaza@cloudbees.com (JIRA)

unread,
Apr 17, 2018, 7:49:03 AM4/17/18
to jenkinsc...@googlegroups.com
Change By: Raul Arabaolaza
Status: In Progress Review

rarabaolaza@cloudbees.com (JIRA)

unread,
Apr 19, 2018, 4:14:02 AM4/19/18
to jenkinsc...@googlegroups.com
 
Re: GitPluginTest are failing in the ATH

It seems the PR went fine and solved not only GitPluginTest but also other affected by the same problem, waiting for the PR to be merged to close this 

scm_issue_link@java.net (JIRA)

unread,
Apr 19, 2018, 4:41:02 AM4/19/18
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Raul Arabaolaza
Path:
src/main/java/org/jenkinsci/test/acceptance/FallbackConfig.java
src/main/java/org/jenkinsci/test/acceptance/po/PluginManager.java
http://jenkins-ci.org/commit/acceptance-test-harness/012364d1a5f6386fab49198c16d00e0cd309a7d9
Log:
JENKINS-50790 Temporarily override pageLoadTimeout for available plugins

This should fix most of GitPluginTest and some others that are failing due to the
available plugins taking too much time to load on ci.jenkins.io

scm_issue_link@java.net (JIRA)

unread,
Apr 19, 2018, 4:41:02 AM4/19/18
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Raul Arabaolaza
Path:
src/main/java/org/jenkinsci/test/acceptance/FallbackConfig.java
src/main/java/org/jenkinsci/test/acceptance/po/PluginManager.java

scm_issue_link@java.net (JIRA)

unread,
Apr 19, 2018, 4:41:02 AM4/19/18
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Oliver Gondža
Path:
src/main/java/org/jenkinsci/test/acceptance/FallbackConfig.java
src/main/java/org/jenkinsci/test/acceptance/po/PluginManager.java
src/test/java/plugins/GitPluginTest.java
http://jenkins-ci.org/commit/acceptance-test-harness/dad333092159cb368efc2f9869572f0a05d255ac
Log:
Merge pull request #428 from raul-arabaolaza/JENKINS-50790

JENKINS-50790 Temporarily override pageLoadTimeout for available plugins

scm_issue_link@java.net (JIRA)

unread,
Apr 19, 2018, 4:41:02 AM4/19/18
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Raul Arabaolaza
Path:
src/test/java/plugins/GitPluginTest.java
http://jenkins-ci.org/commit/acceptance-test-harness/81589c3da0b5788917dfc2faaaec78b89c4edf27
Log:
JENKINS-50790 Fix GitPluginTest#create_tag_for_build

The docker container where the ATH is run doe snot provide a global git
user or mail, so I added the Custom Name and Mail option to the test so
it can properly tag things

rarabaolaza@cloudbees.com (JIRA)

unread,
Apr 19, 2018, 4:56:02 AM4/19/18
to jenkinsc...@googlegroups.com
Change By: Raul Arabaolaza
Status: In Review Resolved
Resolution: Fixed

m.vilacides@gmail.com (JIRA)

unread,
Apr 19, 2018, 6:06:02 AM4/19/18
to jenkinsc...@googlegroups.com
Isa Vilacides updated an issue
Change By: Isa Vilacides
Attachment: Screen Shot 2018-04-19 at 12.00.12.png

m.vilacides@gmail.com (JIRA)

unread,
Apr 19, 2018, 6:07:02 AM4/19/18
to jenkinsc...@googlegroups.com
Isa Vilacides commented on Task JENKINS-50790
 
Re: GitPluginTest are failing in the ATH

Months ago I tried to fix the same problem with increasing the timeout and it didn't solve all the timeout issues, see the PR [here|https://github.com/jenkinsci/acceptance-test-harness/pull/408/files. I also see on your PR that after increasing the timeout, there are still tests failing with the same error, so I think that the root cause of the long page loading has not been determined and we haven't really tackled it. On investigation I realised that there is a JS error on "Available" tab that might be preventing the page from loading and causing all these issues.

It could also be that the click on the available tab happens before the advanced tab has finished loading and it seems like webdriver.get returns as soon as the page fires the load event but the page hasn’t actually finished loading.

rarabaolaza@cloudbees.com (JIRA)

unread,
May 8, 2018, 3:29:02 AM5/8/18
to jenkinsc...@googlegroups.com
Raul Arabaolaza reopened an issue
 

See here it seems that the ATH is still flaky, I suspect due to load
Change By: Raul Arabaolaza
Resolution: Fixed
Status: Resolved Reopened

rarabaolaza@cloudbees.com (JIRA)

unread,
May 8, 2018, 3:29:03 AM5/8/18
to jenkinsc...@googlegroups.com
Raul Arabaolaza updated an issue
Change By: Raul Arabaolaza
Sprint: Essentials - Milestone 1

rarabaolaza@cloudbees.com (JIRA)

unread,
May 8, 2018, 3:40:02 AM5/8/18
to jenkinsc...@googlegroups.com
Raul Arabaolaza commented on Task JENKINS-50790
 
Re: GitPluginTest are failing in the ATH

Jesse Glick As you are the one that did the original implementation of MockUpdateCenter can you imagine a reason why it should have a performance degradation under core >= 2.112 with heavy load?

rarabaolaza@cloudbees.com (JIRA)

unread,
May 8, 2018, 3:40:04 AM5/8/18
to jenkinsc...@googlegroups.com
Raul Arabaolaza edited a comment on Task JENKINS-50790
[~jglick] As you are the one that did the original implementation of [MockUpdateCenter|https://github.com/jenkinsci/acceptance-test-harness/commits/master/src/main/java/org/jenkinsci/test/acceptance/update_center/MockUpdateCenter.java] can you imagine a reason why it should could have a performance degradation under core >= 2.112 with heavy load?

rarabaolaza@cloudbees.com (JIRA)

unread,
May 8, 2018, 4:02:01 AM5/8/18
to jenkinsc...@googlegroups.com

The delay in the rendering of the available plugins page is obvious even running locally (not in the ATH, just starting the war file) when you do not have any plugin installed, that rendering time (which goes worst which heavy load) is the root cause of this issue,

rarabaolaza@cloudbees.com (JIRA)

unread,
May 8, 2018, 4:35:02 AM5/8/18
to jenkinsc...@googlegroups.com

I have performed a little experimentation on load time for the available plugins, when running the ATH and using the MockUpdateCenter the call to get all available plugins oscillates between 25 and 56 seconds running locally, when not using the MockUpdateCenter it takes less than 10 seconds.

I am starting to believe that the only way to solve this (as I can not increase the page load timeout forever) is to use groovy scripts to install plugins instead of doing directly by the UI.

rarabaolaza@cloudbees.com (JIRA)

unread,
May 8, 2018, 6:02:02 AM5/8/18
to jenkinsc...@googlegroups.com

I am experimenting with skipping the available plugins page and just request the installation via a REST call, initial experiments seem promising

rarabaolaza@cloudbees.com (JIRA)

unread,
May 8, 2018, 6:02:02 AM5/8/18
to jenkinsc...@googlegroups.com
Raul Arabaolaza started work on Task JENKINS-50790
 
Change By: Raul Arabaolaza
Status: Reopened In Progress

jglick@cloudbees.com (JIRA)

unread,
May 8, 2018, 7:26:02 AM5/8/18
to jenkinsc...@googlegroups.com

There have been past attempts to load plugins via REST rather than GUI, and that code is still there; you just need a switch to activate it. It does not work all that well as the plugin manager has some subtle logic which is not easily replicated. I would rather just track down the cause of delays and fix them. Run a profiler or whatever. Is the problem in JavaScript or the mock UC service itself?

rarabaolaza@cloudbees.com (JIRA)

unread,
May 8, 2018, 7:42:02 AM5/8/18
to jenkinsc...@googlegroups.com

What I can see is the http call to pluginManager/available performed by the UI when clicking on the Available tab the one that takes too much time. My understanding is that the endpoint calls the mock UC under the covers, as I have checked that same call with an instance run via java -jar (outside ATH) and it takes much less time ~ 10 seconds

rarabaolaza@cloudbees.com (JIRA)

unread,
May 8, 2018, 9:28:02 AM5/8/18
to jenkinsc...@googlegroups.com

Maybe you refer to this code path? I am not going that way, I am making a request to trigger the installation via UC instead of using the buttons in the `Available Plugins Page` and let it handle dependencies and all that stuff, not sure however about versioning constraints at this point

rarabaolaza@cloudbees.com (JIRA)

unread,
May 8, 2018, 10:36:02 AM5/8/18
to jenkinsc...@googlegroups.com

So after a conversation with Jesse Glick it seems the problem is in the core itself since 2.112, the addition of detached plugins causing an exponential explosion in a recursive call. He is working on a fix, in the meanwhile a temporary increase in the timeout should be enough to make everything stable again

rarabaolaza@cloudbees.com (JIRA)

unread,
May 10, 2018, 4:30:02 AM5/10/18
to jenkinsc...@googlegroups.com

Not sure if the fix in JENKINS-51205 is enough, I am still receiving email alerts about instability on git-plugin, however at this moment ci.jenkins.io is down so I can not check the status

rarabaolaza@cloudbees.com (JIRA)

unread,
May 10, 2018, 11:46:02 AM5/10/18
to jenkinsc...@googlegroups.com

It seems it worked, I am going to get this open for a pair of days and close if nothing new arises

jglick@cloudbees.com (JIRA)

unread,
May 10, 2018, 12:53:02 PM5/10/18
to jenkinsc...@googlegroups.com

Makes sense.

rarabaolaza@cloudbees.com (JIRA)

unread,
May 11, 2018, 3:37:02 AM5/11/18
to jenkinsc...@googlegroups.com
Change By: Raul Arabaolaza
Status: In Progress Review

rarabaolaza@cloudbees.com (JIRA)

unread,
May 14, 2018, 4:05:02 AM5/14/18
to jenkinsc...@googlegroups.com
Raul Arabaolaza commented on Task JENKINS-50790
 
Re: GitPluginTest are failing in the ATH

Last builds of git-plugin have no failures, so I am going to close this.

rarabaolaza@cloudbees.com (JIRA)

unread,
May 14, 2018, 4:07:02 AM5/14/18
to jenkinsc...@googlegroups.com
 

The fix for JENKINS-51205 solved this problem
Change By: Raul Arabaolaza
Status: In Review Resolved
Resolution: Fixed

ogondza@gmail.com (JIRA)

unread,
Oct 9, 2019, 7:52:02 AM10/9/19
to jenkinsc...@googlegroups.com
Oliver Gondža updated an issue
Change By: Oliver Gondža
Component/s: acceptance-test-harness
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages