Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-57317

Exception when checking 'Validate S3 Bucket configuration'

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Labels:
      None
    • Environment:
      artifact-manager-s3 1.4 (works on 1.1, fails on 1.2+)
      Jenkins 2.164.3-SNAPSHOT
    • Similar Issues:

      Description

      (I was about to file it as a blocker, but just realized actually this seems only to be an issue in the validation page, but enabling the plugin still archive artifacts fine, so filing it still because it's misleading to users but with lower priority – see git bisect log below)

      Problem

      When opening the /aws page, configuring the plugin and clicking on 'Validate S3 Bucket configuration', we get an error with the following stack trace:

      GetBucketLocation failed
      com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: D910569B825E3D7C; S3 Extended Request ID: 49Hz3b5JOiRPXGCfP+5fySBgjHmp+iUXSPhqqWDdS2eRAqAo3IrZZlaKKCILTzBCkufWMsK1gpM=), S3 Extended Request ID: 49Hz3b5JOiRPXGCfP+5fySBgjHmp+iUXSPhqqWDdS2eRAqAo3IrZZlaKKCILTzBCkufWMsK1gpM=
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1695)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1350)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1101)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:758)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:732)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:714)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:674)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:656)
      	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:520)
      	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4705)
      	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4652)
      	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4646)
      	at com.amazonaws.services.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:989)
      	at com.amazonaws.services.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:995)
      	at io.jenkins.plugins.artifact_manager_jclouds.s3.S3BlobStoreConfig.checkGetBucketLocation(S3BlobStoreConfig.java:237)
      	at io.jenkins.plugins.artifact_manager_jclouds.s3.S3BlobStoreConfig.doValidateS3BucketConfig(S3BlobStoreConfig.java:253)
      	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
      	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
      	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
      	at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)
      	at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
      	at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:537)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
      	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:282)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      	at com.cloudbees.jenkins.support.impl.cloudbees.UnrestrictedApiCallsMonitor$ApiMonitorFilter.doFilter(UnrestrictedApiCallsMonitor.java:120)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      	at org.eclipse.jetty.server.Server.handle(Server.java:503)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
      	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
      	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
      	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
      	at java.lang.Thread.run(Thread.java:748)
      

      To reproduce in short:

      • Set up an IAM Instance Profile allowed to do everything on S3 (or with less permissions, your choice)
      • Create an EC2 instance, use that Instance Profile
      • Instance the plugin and open /aws
      • configure and click Validate.

      Easy way to set up everything using CloudFormation

      Use Evergreen's AWS flavor: https://github.com/jenkins-infra/evergreen/tree/master/distribution/flavors/aws-ec2-cloud

      Once provisioned, just connect to the EC2 instance through SSH, and run the WAR manually like java -jar jenkins.war --httpPort=8081, and copy the config from the Evergreen instance (or just get the bucket name from the AWS console, whatever works)

      Bisect

      git bisect log                                                                                                                                            7c69b02
      git bisect start
      # bad: [7634ca43ec1ea11ac8c3e00fea234c107317c0b0] [maven-release-plugin] prepare release artifact-manager-s3-1.4
      git bisect bad 7634ca43ec1ea11ac8c3e00fea234c107317c0b0
      # good: [67a7e3a419214a983e34c7fe5c2c9ad4e9b99284] [maven-release-plugin] prepare release artifact-manager-s3-1.1
      git bisect good 67a7e3a419214a983e34c7fe5c2c9ad4e9b99284
      # bad: [c9d60bf2f88c300d656a287194c25c4b18e852cd] Merge pull request #82 from jenkinsci/ARC-576
      git bisect bad c9d60bf2f88c300d656a287194c25c4b18e852cd
      # good: [7659f21cfa926eef87f787ace4ed4c52713c1a91] Merge pull request #78 from jenkinsci/metachars-JENKINS-50591-JENKINS-52151
      git bisect good 7659f21cfa926eef87f787ace4ed4c52713c1a91
      # skip: [f97d65ddb84140ac5e385dbabc5f0579cc68ea18] Merge branch 'master' into GetBucketLocation
      git bisect skip f97d65ddb84140ac5e385dbabc5f0579cc68ea18
      # good: [f1216a60d6df001e3aedcebe3150406cd929c3d7] Missing imports.
      git bisect good f1216a60d6df001e3aedcebe3150406cd929c3d7
      # good: [b08502d2b1da462e0994d8d934c898d46e67d14a] Merge pull request #79 from davidcurrie/ARC-480
      git bisect good b08502d2b1da462e0994d8d934c898d46e67d14a
      # bad: [18ecbe3fe2b5c1466ca16c582385ab8c7c43016e] Check GetBucketLocation on validation
      git bisect bad 18ecbe3fe2b5c1466ca16c582385ab8c7c43016e
      # good: [7c69b02ba8b097ef32ee0a509407f5280dcb3af9] Re-enable ignored tests
      git bisect good 7c69b02ba8b097ef32ee0a509407f5280dcb3af9
      # first bad commit: [18ecbe3fe2b5c1466ca16c582385ab8c7c43016e] Check GetBucketLocation on validation
      

        Attachments

          Activity

          Hide
          jglick Jesse Glick added a comment -

          If you are sure you granted all permissions to the role, then my first guess offhand would be that Jenkins is failing to pass the right region to the request, which may make it impossible for me to reproduce with the account I use for testing since I suppose you are using a European region. Will try to set up a reproduction environment at some point.

          Show
          jglick Jesse Glick added a comment - If you are sure you granted all permissions to the role, then my first guess offhand would be that Jenkins is failing to pass the right region to the request, which may make it impossible for me to reproduce with the account I use for testing since I suppose you are using a European region. Will try to set up a reproduction environment at some point.
          Hide
          jglick Jesse Glick added a comment -

          I tried using the Evergreen instructions but they failed in EC2EvergreenInstance:

          The specified instance type can only be used in a VPC. A subnet ID or network interface ID is required to carry out the request. (Service: AmazonEC2; Status Code: 400; Error Code: VPCResourceNotSpecified; Request ID: …)

          Show
          jglick Jesse Glick added a comment - I tried using the Evergreen instructions but they failed in EC2EvergreenInstance : The specified instance type can only be used in a VPC. A subnet ID or network interface ID is required to carry out the request. (Service: AmazonEC2; Status Code: 400; Error Code: VPCResourceNotSpecified; Request ID: …)
          Hide
          batmat Baptiste Mathus added a comment -

          Facepalm. This is https://github.com/jenkins-infra/evergreen/pull/396 which I'm going to finally actually revert... Sorry about that

          Show
          batmat Baptiste Mathus added a comment - Facepalm. This is https://github.com/jenkins-infra/evergreen/pull/396 which I'm going to finally actually revert... Sorry about that
          Hide
          jglick Jesse Glick added a comment -

          Still getting the same error. Is there some sort of caching going on?

          Show
          jglick Jesse Glick added a comment - Still getting the same error. Is there some sort of caching going on?
          Hide
          lukelast Luke Last added a comment -

          Seems like the "s3:GetBucketLocation" permission is required for validation to work.

          Show
          lukelast Luke Last added a comment - Seems like the "s3:GetBucketLocation" permission is required for validation to work.

            People

            • Assignee:
              Unassigned
              Reporter:
              batmat Baptiste Mathus
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: