[JIRA] (JENKINS-57317) Exception when checking 'Validate S3 Bucket configuration'

13 views
Skip to first unread message

bmathus+ossjira@cloudbees.com (JIRA)

unread,
May 3, 2019, 2:49:02 AM5/3/19
to jenkinsc...@googlegroups.com
Baptiste Mathus created an issue
 
Jenkins / Bug JENKINS-57317
Exception when checking 'Validate S3 Bucket configuration'
Issue Type: Bug Bug
Assignee: Unassigned
Components: artifact-manager-s3-plugin
Created: 2019-05-03 06:48
Environment: artifact-manager-s3 1.4 (works on 1.1, fails on 1.2+)
Jenkins 2.164.3-SNAPSHOT
Priority: Minor Minor
Reporter: Baptiste Mathus

(I was about to file it as a blocker, but just realized actually this seems only to be an issue in the validation page, but enabling the plugin still archive artifacts fine, so filing it still because it's misleading to users but with lower priority)

Problem

When opening the /aws page, configuring the plugin and clicking on 'Validate S3 Bucket configuration', we get an error with the following stack trace:

GetBucketLocation failed
com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: D910569B825E3D7C; S3 Extended Request ID: 49Hz3b5JOiRPXGCfP+5fySBgjHmp+iUXSPhqqWDdS2eRAqAo3IrZZlaKKCILTzBCkufWMsK1gpM=), S3 Extended Request ID: 49Hz3b5JOiRPXGCfP+5fySBgjHmp+iUXSPhqqWDdS2eRAqAo3IrZZlaKKCILTzBCkufWMsK1gpM=
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1695)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1350)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1101)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:758)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:732)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:714)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:674)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:656)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:520)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4705)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4652)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4646)
	at com.amazonaws.services.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:989)
	at com.amazonaws.services.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:995)
	at io.jenkins.plugins.artifact_manager_jclouds.s3.S3BlobStoreConfig.checkGetBucketLocation(S3BlobStoreConfig.java:237)
	at io.jenkins.plugins.artifact_manager_jclouds.s3.S3BlobStoreConfig.doValidateS3BucketConfig(S3BlobStoreConfig.java:253)
	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
	at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)
	at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
	at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:537)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:282)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at com.cloudbees.jenkins.support.impl.cloudbees.UnrestrictedApiCallsMonitor$ApiMonitorFilter.doFilter(UnrestrictedApiCallsMonitor.java:120)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:503)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
	at java.lang.Thread.run(Thread.java:748)

To reproduce in short:

  • Set up an IAM Instance Profile allowed to do everything on S3 (or with less permissions, your choice)
  • Create an EC2 instance, use that Instance Profile
  • Instance the plugin and open /aws
  • configure and click Validate.

Easy way to set up everything using CloudFormation

Use Evergreen's AWS flavor: https://github.com/jenkins-infra/evergreen/tree/master/distribution/flavors/aws-ec2-cloud

Once provisioned, just connect to the EC2 instance through SSH, and run the WAR manually like java -jar jenkins.war --httpPort=8081, and copy the config from the Evergreen instance (or just get the bucket name from the AWS console, whatever works)
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

bmathus+ossjira@cloudbees.com (JIRA)

unread,
May 3, 2019, 3:04:03 AM5/3/19
to jenkinsc...@googlegroups.com
Baptiste Mathus updated an issue
Change By: Baptiste Mathus
(I was about to file it as a blocker, but just realized actually this seems only to be an issue in the validation page, but enabling the plugin still archive artifacts fine, so filing it still because it's misleading to users but with lower priority)

h3. Problem

When opening the /aws page, configuring the plugin and clicking on 'Validate S3 Bucket configuration', we get an error with the following stack trace:

{noformat}
{noformat}

To reproduce in short:
* Set up an IAM Instance Profile allowed to do everything on S3 (or with less permissions, your choice)
* Create an EC2 instance, use that Instance Profile
* Instance the plugin and open /aws
* configure and click Validate.

h4. Easy way to set up everything using CloudFormation
Once provisioned, just connect to the EC2 instance through SSH, and run the WAR manually like {{java -jar jenkins.war --httpPort=8081}}, and copy the config from the Evergreen instance (or just get the bucket name from the AWS console, whatever works)
h3. Bisect

{noformat}
git bisect log                        7c69b02
git bisect start
# bad: [7634ca43ec1ea11ac8c3e00fea234c107317c0b0] [maven-release-plugin] prepare release artifact-manager-s3-1.4
git bisect bad 7634ca43ec1ea11ac8c3e00fea234c107317c0b0
# good: [67a7e3a419214a983e34c7fe5c2c9ad4e9b99284] [maven-release-plugin] prepare release artifact-manager-s3-1.1
git bisect good 67a7e3a419214a983e34c7fe5c2c9ad4e9b99284
# bad: [c9d60bf2f88c300d656a287194c25c4b18e852cd] Merge pull request #82 from jenkinsci/ARC-576
git bisect bad c9d60bf2f88c300d656a287194c25c4b18e852cd
# good: [7659f21cfa926eef87f787ace4ed4c52713c1a91] Merge pull request #78 from jenkinsci/metachars-JENKINS-50591-JENKINS-52151
git bisect good 7659f21cfa926eef87f787ace4ed4c52713c1a91
# skip: [f97d65ddb84140ac5e385dbabc5f0579cc68ea18] Merge branch 'master' into GetBucketLocation
git bisect skip f97d65ddb84140ac5e385dbabc5f0579cc68ea18
# good: [f1216a60d6df001e3aedcebe3150406cd929c3d7] Missing imports.
git bisect good f1216a60d6df001e3aedcebe3150406cd929c3d7
# good: [b08502d2b1da462e0994d8d934c898d46e67d14a] Merge pull request #79 from davidcurrie/ARC-480
git bisect good b08502d2b1da462e0994d8d934c898d46e67d14a
# bad: [18ecbe3fe2b5c1466ca16c582385ab8c7c43016e] Check GetBucketLocation on validation
git bisect bad 18ecbe3fe2b5c1466ca16c582385ab8c7c43016e
# good: [7c69b02ba8b097ef32ee0a509407f5280dcb3af9] Re-enable ignored tests
git bisect good 7c69b02ba8b097ef32ee0a509407f5280dcb3af9
# first bad commit: [18ecbe3fe2b5c1466ca16c582385ab8c7c43016e] Check GetBucketLocation on validation
{noformat}

bmathus+ossjira@cloudbees.com (JIRA)

unread,
May 3, 2019, 3:04:03 AM5/3/19
to jenkinsc...@googlegroups.com
Baptiste Mathus updated an issue
(I was about to file it as a blocker, but just realized actually this seems only to be an issue in the validation page, but enabling the plugin still archive artifacts fine, so filing it still because it's misleading to users but with lower priority -- see {{git bisect log}} below )

jglick@cloudbees.com (JIRA)

unread,
May 3, 2019, 9:04:01 AM5/3/19
to jenkinsc...@googlegroups.com
Jesse Glick commented on Bug JENKINS-57317
 
Re: Exception when checking 'Validate S3 Bucket configuration'

If you are sure you granted all permissions to the role, then my first guess offhand would be that Jenkins is failing to pass the right region to the request, which may make it impossible for me to reproduce with the account I use for testing since I suppose you are using a European region. Will try to set up a reproduction environment at some point.

jglick@cloudbees.com (JIRA)

unread,
Jun 6, 2019, 5:40:01 PM6/6/19
to jenkinsc...@googlegroups.com

I tried using the Evergreen instructions but they failed in EC2EvergreenInstance:

The specified instance type can only be used in a VPC. A subnet ID or network interface ID is required to carry out the request. (Service: AmazonEC2; Status Code: 400; Error Code: VPCResourceNotSpecified; Request ID: …)

bmathus+ossjira@cloudbees.com (JIRA)

unread,
Jun 7, 2019, 7:33:06 AM6/7/19
to jenkinsc...@googlegroups.com

Facepalm. This is https://github.com/jenkins-infra/evergreen/pull/396 which I'm going to finally actually revert... Sorry about that

jglick@cloudbees.com (JIRA)

unread,
Jun 18, 2019, 5:37:01 PM6/18/19
to jenkinsc...@googlegroups.com

Still getting the same error. Is there some sort of caching going on?

luke@lukelast.com (JIRA)

unread,
Dec 9, 2019, 10:42:03 PM12/9/19
to jenkinsc...@googlegroups.com

Seems like the "s3:GetBucketLocation" permission is required for validation to work.
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages