Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23022

Updating EC2 tags sometimes fails with Spot instances, leaking a slave

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • ec2-plugin
    • None
    • Jenkins 1.563
      EC2 Plugin 1.21
      Ubuntu 12.04 LTS

      When using Spot instances with EC2 tags to be applied, sometimes slave creation fails when trying to update Spot Request tag. I guess that's because the Spot request is not yet registered.
      When this occurs, the slave is not created on Jenkins side but the corresponding Spot request is. This cause an instance leak.

      Here's the stack:

      Status Code: 400, AWS Service: AmazonEC2, AWS Request ID: 19812aa6-a6ef-40bd-9d28-2a1b393690e7, AWS Error Code: InvalidSpotInstanceRequestID.NotFound, AWS Error Message: The spot instance request ID 'sir-bf3d3e3b' does not exist

      com.amazonaws.AmazonServiceException: Status Code: 400, AWS Service: AmazonEC2, AWS Request ID: 19812aa6-a6ef-40bd-9d28-2a1b393690e7, AWS Error Code: InvalidSpotInstanceRequestID.NotFound, AWS Error Message: The spot instance request ID 'sir-bf3d3e3b' does not exist
      at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:773)
      at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:417)
      at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:229)
      at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:8504)
      at com.amazonaws.services.ec2.AmazonEC2Client.createTags(AmazonEC2Client.java:4791)
      at hudson.plugins.ec2.SlaveTemplate.updateRemoteTags(SlaveTemplate.java:580)
      at hudson.plugins.ec2.SlaveTemplate.provisionSpot(SlaveTemplate.java:540)
      at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:260)
      at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:243)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
      at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
      at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
      at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
      at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
      at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
      at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631)
      at org.kohsuke.stapler.Stapler.service(Stapler.java:225)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
      at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
      at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:74)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      at org.eclipse.jetty.server.Server.handle(Server.java:370)
      at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
      at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
      at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
      at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
      at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
      at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
      at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
      at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
      at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

            francisu Francis Upton
            mburtin Michaƫl Burtin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: