Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31474

After unsuccessful terminate, ec2-plugin reaches a state where no new slaves are being initialized

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Minor Minor
    • ec2-plugin
    • None
    • Jenkins 1.631, ec2-plugin 1.29

      We use ec2-plugin to start on-demand slaves and terminate them after 30m of idle time.

      This was working fine, until last week when we noticed a full build queue and an empty list of build executors. On EC2, there actually was a slave running (i-1e42a2a7). Below is the relevant fragment from the log:

      WARNING: Failed to fetch EC2 instance: i-1e42a2a7
com.amazonaws.AmazonServiceException: The service is unavailable. Please try again shortly. (Service: AmazonEC2; Status Code: 503; Error Code: Unavailable; Re-rw-r---quest ID: ff9e539b-f87f-4143-80b6-e11fbf61378e)
        at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1160)
        at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:748)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302)
        at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11768)
        at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5899)
        at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:232)
        at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:378)
        at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:350)
        at hudson.plugins.ec2.EC2OndemandSlave.terminate(EC2OndemandSlave.java:63)
        at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:297)
        at hudson.plugins.ec2.EC2RetentionStrategy._check(EC2RetentionStrategy.java:99)
        at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:71)
        at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:40)
        at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:70)
        at hudson.model.Queue._withLock(Queue.java:1286)
        at hudson.model.Queue.withLock(Queue.java:1169)
        at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:61)
        at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Oct 30, 2015 12:50:32 PM hudson.plugins.ec2.EC2OndemandSlave terminate
INFO: EC2 instance already terminated: i-1e42a2a7
Oct 30, 2015 12:50:32 PM hudson.plugins.ec2.EC2OndemandSlave terminate
INFO: Removed EC2 instance from jenkins master: i-1e42a2a7

            francisu Francis Upton
            jeroen_s Jeroen Smolenaers
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: