[JIRA] (JENKINS-39505) JClouds plugin fails on Jenkins v2.28

JClouds plugin fails on Jenkins v2.28

Issue Type:	Bug
Assignee:	Fritz Elfert
Components:	jclouds-plugin
Created:	2016/Nov/04 12:41 PM
Priority:	Major
Reporter:	Georg Brunmayr

We use the JClouds plugin to provision OpenNebula instances via aws-ec2 api
Since update to 2.28 the connection fails.

From the stack trace i guess this is related to JENKINS-38736

Cannot connect to specified cloud, please check the identity and credentials: Unable to create injector, see the following errors:

1) Overriding @Provides methods is not allowed.
@Provides method: org.jclouds.logging.config.LoggingModule.createLoggerFactory()
overridden by: jenkins.plugins.jclouds.compute.JCloudsCloud$1.createLoggerFactory()
at com.google.inject.internal.ProviderMethodsModule.getProviderMethods(ProviderMethodsModule.java:163)

2) Overriding @Provides methods is not allowed.
@Provides method: org.jclouds.compute.config.BaseComputeServiceContextModule.provideTemplateOptions()
overridden by: org.jclouds.aws.ec2.compute.config.AWSEC2ComputeServiceContextModule.provideTemplateOptions()
at com.google.inject.internal.ProviderMethodsModule.getProviderMethods(ProviderMethodsModule.java:163)

3) Overriding @Provides methods is not allowed.
@Provides method: org.jclouds.compute.config.BaseComputeServiceContextModule.provideImageExtension()
overridden by: org.jclouds.aws.ec2.compute.config.AWSEC2ComputeServiceContextModule.provideImageExtension()
at com.google.inject.internal.ProviderMethodsModule.getProviderMethods(ProviderMethodsModule.java:163)

4) Overriding @Provides methods is not allowed.
@Provides method: org.jclouds.compute.config.BaseComputeServiceContextModule.provideSecurityGroupExtension()
overridden by: org.jclouds.aws.ec2.compute.config.AWSEC2ComputeServiceContextModule.provideSecurityGroupExtension()
at com.google.inject.internal.ProviderMethodsModule.getProviderMethods(ProviderMethodsModule.java:163)

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

fritz@fritz-elfert.de (JIRA)

unread,

Nov 4, 2016, 10:20:01 AM11/4/16

to jenkinsc...@googlegroups.com

Fritz Elfert commented on

Which version of the jclouds plugin are you using? (From the plugins page or built from git).

bernd.farka@gmail.com (JIRA)

unread,

Nov 4, 2016, 10:25:02 AM11/4/16

to jenkinsc...@googlegroups.com

Bernd Farka commented on

The Version Form the Plugin page

scm_issue_link@java.net (JIRA)

unread,

Nov 4, 2016, 12:20:01 PM11/4/16

to jenkinsc...@googlegroups.com

SCM/JIRA link daemon commented on

Code changed in jenkins
User: Fritz Elfert
Path:
jclouds-plugin/pom.xml
pom.xml
http://jenkins-ci.org/commit/jclouds-plugin/2af5ce412ccb79e70e26a6eab82e6e8ca0049eb4
Log:
Attempt to fix JENKINS-39505

bernd.farka@gmail.com (JIRA)

unread,

Nov 4, 2016, 12:23:01 PM11/4/16

to jenkinsc...@googlegroups.com

Bernd Farka commented on

If I can hell you verifying something just let me know...

fritz@fritz-elfert.de (JIRA)

unread,

Nov 4, 2016, 12:35:04 PM11/4/16

to jenkinsc...@googlegroups.com

Fritz Elfert commented on

I have just made a commit to https://github.com/jenkinsci/jclouds-plugin which should fix your problem.
Are you able to build this yourself or do you need an installable .hpi?
If you need an installable .hpi, please contact me privately via email.
ATTENTION-ATTENTION-ATTENTION
BEFORE updating, be shure to backup your your <jenkins-homedir>/config.xml and <jenkins-homedir>/credentials.xml !!!!!
Reason: The new plugin uses jenkins credentials framework and auto-migrates any old jclouds-credentials into the new credential-system.
If you want to go downgrade for some reason, you must downgrade the plugin, then sto jenkins and copy the backup over the existing xmls
After that, you can start jenkins again and should have the old configuration.

fritz@fritz-elfert.de (JIRA)

unread,

Nov 4, 2016, 12:36:02 PM11/4/16

to jenkinsc...@googlegroups.com

Fritz Elfert edited a comment on

I have just made a commit to https://github.com/jenkinsci/jclouds-plugin which should fix your problem.
Are you able to build this yourself or do you need an installable .hpi?
If you need an installable .hpi, please contact me privately via email.
ATTENTION-ATTENTION-ATTENTION
BEFORE updating, be shure to backup your your <jenkins-homedir>/config.xml and <jenkins-homedir>/credentials.xml !!!!!
Reason: The new plugin uses jenkins credentials framework and auto-migrates any old jclouds-credentials into the new credential-system.

If you want to go downgrade for some reason, you must downgrade the plugin, then sto stop jenkins and copy the backup over the existing xmls

After that, you can start jenkins again and should have the old configuration.

fritz@fritz-elfert.de (JIRA)

unread,

Nov 4, 2016, 12:45:02 PM11/4/16

to jenkinsc...@googlegroups.com

Fritz Elfert commented on

Correction: The latest build artifact is also available here: https://jenkins.ci.cloudbees.com/job/plugins/job/jclouds-plugin/ws/jclouds-plugin/target/jclouds-jenkins.hpi

dbeck@cloudbees.com (JIRA)

unread,

Nov 5, 2016, 8:07:01 AM11/5/16

to jenkinsc...@googlegroups.com

Daniel Beck commented on

Is this actually related to ~~JENKINS-38736~~ ?

fritz@fritz-elfert.de (JIRA)

unread,

Nov 5, 2016, 9:17:02 AM11/5/16

to jenkinsc...@googlegroups.com

Fritz Elfert commented on

Indirectly yes. The well known classloader isolation problem. The jclouds libs depend on an older guice and some other conflicting libs.
As a developer, I was always struggling with that it is also the reason why a shaded jclouds lib is included in the plugin.
However, I was able to fix this by enabling the pluginFirstClassLoader.

-Fritz

fritz@fritz-elfert.de (JIRA)

unread,

Nov 5, 2016, 9:18:01 AM11/5/16

to jenkinsc...@googlegroups.com

Fritz Elfert edited a comment on

Indirectly yes. The well known classloader isolation problem. The jclouds libs depend on an older guice and some other conflicting libs.

As a developer, I was always struggling with that it . It is also the reason why a shaded jclouds lib is included in the plugin.

However, I was able to fix this by enabling the pluginFirstClassLoader.

-Fritz

fritz@fritz-elfert.de (JIRA)

unread,

Nov 5, 2016, 11:33:01 AM11/5/16

to jenkinsc...@googlegroups.com

Fritz Elfert started work on

Change By:	Fritz Elfert
Status:	Open In Progress

fritz@fritz-elfert.de (JIRA)

unread,

Nov 7, 2016, 7:20:02 AM11/7/16

to jenkinsc...@googlegroups.com

Fritz Elfert commented on

Any further feedback from Reporter or Bernd Farka appreciated!

fritz@fritz-elfert.de (JIRA)

unread,

Nov 7, 2016, 7:21:01 AM11/7/16

to jenkinsc...@googlegroups.com

Fritz Elfert edited a comment on

Any further feedback from Reporter or Bernd Farka appreciated!

Did the provided build help?

bernd.farka@gmail.com (JIRA)

unread,

Nov 7, 2016, 7:39:01 AM11/7/16

to jenkinsc...@googlegroups.com

Bernd Farka commented on

hi

sorry for the late feedback:

i deployed the new version and suddelny our Jenkins started to take more memory and going OOM...

until now i have no furtherer stacktraces for you, but I'll try to repdoruce it and give you new findings...

bernd.farka@gmail.com (JIRA)

unread,

Nov 7, 2016, 7:47:02 AM11/7/16

to jenkinsc...@googlegroups.com

Bernd Farka commented on

some (not very helpfuly) finding

 
                                                                WARNING: Error while serving https://jenkins-one.lab.dynatrace.org/descriptorByName/jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate/fillLocationIdItems
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:335)
        at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:175)
        at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:108)
        at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
        at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
        at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:236)
        at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
        at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
        at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:206)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
        at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
        at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
        at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:80)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
        at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
        at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
        at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:499)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
        at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
        at java.lang.StringBuilder.append(StringBuilder.java:136)
        at java.lang.StringBuilder.append(StringBuilder.java:76)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457)
        at java.lang.StringBuilder.append(StringBuilder.java:166)
        at java.lang.StringBuilder.append(StringBuilder.java:76)
        at java.util.Formatter$FormatSpecifier.print(Formatter.java:2913)
        at java.util.Formatter$FormatSpecifier.printString(Formatter.java:2886)
        at java.util.Formatter$FormatSpecifier.print(Formatter.java:2763)
        at java.util.Formatter.format(Formatter.java:2526)
        at java.util.Formatter.format(Formatter.java:2455)
        at java.lang.String.format(String.java:2940)
        at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:161)
        at java.util.logging.StreamHandler.publish(StreamHandler.java:211)
        at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:116)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:875)
        at jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate$DescriptorImpl.doFillLocationIdItems(JCloudsSlaveTemplate.java:785)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:335)
        at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:175)
        at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:108)
        at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
        at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
 
                                                            

fyi: we have 6GB xmx and using 2 on normal conditions...

bernd.farka@gmail.com (JIRA)

unread,

Nov 7, 2016, 7:48:04 AM11/7/16

to jenkinsc...@googlegroups.com

Bernd Farka edited a comment on

some (not very helpfuly helpful ) finding

{code}

{code}

fyi: we have 6GB xmx and using 2 on normal conditions...

fritz@fritz-elfert.de (JIRA)

unread,

Nov 7, 2016, 7:58:01 AM11/7/16

to jenkinsc...@googlegroups.com

Fritz Elfert commented on

Have you built the plugin yourself or used the provided one? If you downloaded it: when did you fetch it?
Background:
After providing the link, I also upgraded from jclouds-2.9.1 to jclouds-2.9.2 (which might be the memleak cause)
Can you build the plugin yourself? If yes, the checkout 2af5ce412ccb79e70e26a6eab82e6e8ca0049eb4 which has exacly this single fix.

fritz@fritz-elfert.de (JIRA)

unread,

Nov 7, 2016, 8:00:06 AM11/7/16

to jenkinsc...@googlegroups.com

Fritz Elfert edited a comment on

Have you built the plugin yourself or used the provided one? If you downloaded it: *when* did you fetch it?
Background:
After providing the link, I also upgraded from jclouds- 2 1 .9.1 to jclouds- 2 1 .9.2 (which *might* be the memleak cause)

Can you build the plugin yourself? If yes, the checkout 2af5ce412ccb79e70e26a6eab82e6e8ca0049eb4 which has exacly this single fix.

fritz@fritz-elfert.de (JIRA)

unread,

Nov 7, 2016, 8:16:01 AM11/7/16

to jenkinsc...@googlegroups.com

Fritz Elfert commented on

BTW:
The trace ends up in doFillLocationIdItems() where it iterates over the possible locations from your cloud provider and builds a combo-box from them.
This shold happen only, if you actually click on thet specific combobox - oh and it is old code that has not changed compared to 1.8.1.
Indeed probably purely coincidential.

fritz@fritz-elfert.de (JIRA)

unread,

Nov 7, 2016, 8:46:01 AM11/7/16

to jenkinsc...@googlegroups.com

Fritz Elfert commented on

FYI: In the meantime, I have set up a jenkins 2.28 here and was able to reproduce your original issue (using the old plugin), but not the OOM using the latest plugin snapshot. On the other hand I don't have any OpenNebula (using OpenStack juno here)

bernd.farka@gmail.com (JIRA)

unread,

Nov 7, 2016, 8:47:01 AM11/7/16

to jenkinsc...@googlegroups.com

Bernd Farka commented on

I downloaded it about 6 hours ago....

i can build it, but not quite now, we would have to wait a few hours until i have time...

fritz@fritz-elfert.de (JIRA)

unread,

Nov 7, 2016, 8:47:02 AM11/7/16

to jenkinsc...@googlegroups.com

Fritz Elfert edited a comment on

FYI: In the meantime, I have set up a jenkins 2.28 here and was able to reproduce your original issue (using the old plugin), but *not* the OOM using the latest plugin snapshot. On the other hand I don't have any OpenNebula (using OpenStack juno here) In other words: My lates build runs fine here.

fritz@fritz-elfert.de (JIRA)

unread,

Nov 7, 2016, 12:26:02 PM11/7/16

to jenkinsc...@googlegroups.com

Fritz Elfert commented on

If you do, make shure you use maven 3.2.x, not maven-3.3! See README.md

fritz@fritz-elfert.de (JIRA)

unread,

Nov 11, 2016, 5:25:01 AM11/11/16

to jenkinsc...@googlegroups.com

Fritz Elfert commented on

Since I did not get any further feedback, in the meantime, I have released jclouds-plugin v2.9 which runs fine here with 2.28.
Maybe you want to give that one a try.

fritz@fritz-elfert.de (JIRA)

unread,

Nov 23, 2016, 7:53:01 PM11/23/16

to jenkinsc...@googlegroups.com

Fritz Elfert commented on

@Bernd Farka resp. Georg Brunmayr:
Did you had any further chance to test this? (It runs fine here)

georg.brunmayr@dynatrace.com (JIRA)

unread,

Nov 24, 2016, 1:23:03 AM11/24/16

to jenkinsc...@googlegroups.com

Georg Brunmayr commented on

We have installed the latest version and the original issue is gone.
The aws ec2 driver against OpenNebula has the tendency to grab all the memory it can get but that's another topic and most likely related to the endpoint. The nova plugin connects fine even though it does not work to provision instances but that's caused by OpenNebula not supporting SecurityGroup creation via the API.

fritz@fritz-elfert.de (JIRA)

unread,

Nov 24, 2016, 7:31:08 AM11/24/16

to jenkinsc...@googlegroups.com

Fritz Elfert stopped work on

Change By:	Fritz Elfert
Status:	In Progress Open

fritz@fritz-elfert.de (JIRA)

unread,

Nov 24, 2016, 7:42:02 AM11/24/16

to jenkinsc...@googlegroups.com

Fritz Elfert closed an issue as Fixed

Thank you for your response. Closing this issue now.

Regarding the memory problem:
I am going to release v1.11 soon (next week) which contains an important change:

Upgrade to jclouds-2.0.0 (which was released last week)
This also implies that jclouds now supports guice-4.0 which is identical to the
guice-version used by jenkins itself. This solves many problems and might
even solve the memory hog.
So: I'd suggest to upgrade when it's available and simply open another issue, if
the memory problem persists.

Jenkins /

JClouds plugin fails on Jenkins v2.28

Change By:	Fritz Elfert
Status:	Open Closed
Resolution:	Fixed