Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21949

ec2: if spot insantce goes away, ec2 plugin crashes

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • ec2-plugin
    • None
    • 1.551 ubuntu native package, ec2 plugin v 1.20

      I created a slave with a spot instance in ec2 config, however I shut it down over a weekend. Come monday the instance ID isn't recognized by aws, and it causes the ec2 plugin to crash and no longer function (no new slaves, etc).

      Feb 24, 2014 11:55:24 PM hudson.plugins.ec2.EC2Cloud provision
      WARNING: Failed to count the # of live instances on EC2
      Status Code: 400, AWS Service: AmazonEC2, AWS Request ID: 48ef733c-28c1-423e-9900-62de12315457, AWS Error Code: InvalidSpotInstanceRequestID.NotFound, AWS Error Message: The spot instance request ID 'sir-f43fdc4f' does not exist
      at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:614)
      at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:312)
      at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:165)
      at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:6047)
      at com.amazonaws.services.ec2.AmazonEC2Client.describeSpotInstanceRequests(AmazonEC2Client.java:3068)
      at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:331)
      at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281)
      at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51)
      at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:366)
      at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      1)
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:724)

      And now I get this over and over:
      Feb 24, 2014 11:55:53 PM hudson.node_monitors.ResponseTimeMonitor$1 monitor
      WARNING: Making 541693b6-21dc-4c5d-b98b-717d03349d20 offline because it’s not responding

      however it will not launch any new slaves, even if I have the executor limit on the ec2 plugin set to 20.

      I'm not sure how to clear the history of all executors, so I am stuck without any slaves now.

            francisu Francis Upton
            rvanderwerf Ryan Vanderwerf
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: