Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46505

Jenkins slaves are being stopped multiple times with ec2 plugin

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: ec2-plugin
    • Environment:
      Jenkins Version: 2.60.3
      Amazon EC2 plugins: 1.36
      Idle termination time: -5
    • Similar Issues:

      Description

      I am seeing cases where Jenkins slaves are being stopped two or three times when using the ec2 plugin.  I was able to go to the AWS console and confirm the slave was successfully stopped after the first request. 

      Update: We changed our local branch to only go through the logic if the agent isOnline() which seemed to resolve the issue.

      From /var/log/jenkins/jenkins.log:

      Aug 26, 2017 8:02:05 AM hudson.plugins.ec2.EC2RetentionStrategy internalCheck
      INFO: Idle timeout of jenkins-slave-name (i-xxxxxxxxxxxxxxxx) after 24 idle minutes, with 3 minutes remaining in billing period
      
      Aug 26, 2017 8:02:05 AM hudson.plugins.ec2.EC2AbstractSlave idleTimeout
      INFO: EC2 instance idle time expired: i-xxxxxxxxxxxxxxxx
      
      Aug 26, 2017 8:02:06 AM hudson.plugins.ec2.EC2AbstractSlave stop
      INFO: EC2 instance stop request sent for i-xxxxxxxxxxxxxxxx
      
      
      Aug 26, 2017 8:03:59 AM hudson.plugins.ec2.EC2RetentionStrategy internalCheck
      INFO: Idle timeout of jenkins-slave-name (i-xxxxxxxxxxxxxxxx) after 26 idle minutes, with 2 minutes remaining in billing period
      
      Aug 26, 2017 8:03:59 AM hudson.plugins.ec2.EC2AbstractSlave idleTimeout
      INFO: EC2 instance idle time expired: i-xxxxxxxxxxxxxxxx
      
      Aug 26, 2017 8:04:00 AM hudson.plugins.ec2.EC2AbstractSlave stop
      INFO: EC2 instance stop request sent for i-xxxxxxxxxxxxxxxx

       

      From AWS console:

      State transition reason:
      User initiated (2017-08-26 08:02:05 GMT)
          
      State transition reason message:
      Client.UserInitiatedShutdown: User initiated shutdown

        Attachments

          Activity

          Hide
          gcimpoies George Cimpoies added a comment -

          I'm having the same issue but in my case any build taking over 30 min is broken because the node is shut down no matter what... Any ideas?

          Show
          gcimpoies George Cimpoies added a comment - I'm having the same issue but in my case any build taking over 30 min is broken because the node is shut down no matter what... Any ideas?

            People

            • Assignee:
              francisu Francis Upton
              Reporter:
              mkozell Mike Kozell
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: