Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-57795

Orphaned EC2 instances after Jenkins restart

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: ec2-plugin
    • Labels:
      None
    • Environment:
      Jenkins ver. 2.176.1
      ec2 plugin 1.43, 1.44, 1.45
    • Similar Issues:

      Description

      Sometimes after a Jenkins restart the plugin won't be able to spawn more agents.

      The plugin will just loop on this:

      SlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker'}. Attempting to provision slave needed by excess workload of 1 units
      May 31, 2019 2:23:53 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlave
      SlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker'}. Cannot provision - no capacity for instances: 0
      May 31, 2019 2:23:53 PM WARNING hudson.plugins.ec2.EC2Cloud provision
      Can't raise nodes for SlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker'}
      

      If I go to the EC2 console and terminate the instance manually the plugin will spawn a new one and use it.

      It seems like there is some mismatch in the plugin logic. The part responsible for calculating the number of instances and checking the cap sees the EC2 instance. However the part responsible for picking up running EC2 instances doesn't seem to be able to find it.

      We use a single subnet, security group and vpc (I've seen some reports about this causing problems).

      We use instanceCap = 1 setting as we are testing the plugin, this might make this problem more visible than with a higher cap.

        Attachments

          Activity

          Hide
          thoulen FABRIZIO MANFREDI added a comment -

          I believe I found the problem, I trying to put in the 1.44.2 that should be release in a couple of days.

          One more question what do you mean with orphaned, stop state or no longer in the jenkins interface ? 

          Did you apply the all the IAM role requested specify in the ec2 plugin page ? 

          Show
          thoulen FABRIZIO MANFREDI added a comment - I believe I found the problem, I trying to put in the 1.44.2 that should be release in a couple of days. One more question what do you mean with orphaned, stop state or no longer in the jenkins interface ?  Did you apply the all the IAM role requested specify in the ec2 plugin page ? 
          Hide
          jbochenski Jakub Bochenski added a comment - - edited

          > One more question what do you mean with orphaned, stop state or no longer in the jenkins interface ?

          It's not available as an agent in jenkins. It's still in running state when I check the status in AWS console

          > Did you apply the all the IAM role requested specify in the ec2 plugin page ?

          I believe I did, since this is a random error it doesn't happen every time. E.g. the instances get terminated after idle timeout

          Show
          jbochenski Jakub Bochenski added a comment - - edited > One more question what do you mean with orphaned, stop state or no longer in the jenkins interface ? It's not available as an agent in jenkins. It's still in running state when I check the status in AWS console > Did you apply the all the IAM role requested specify in the ec2 plugin page ? I believe I did, since this is a random error it doesn't happen every time. E.g. the instances get terminated after idle timeout
          Hide
          jbochenski Jakub Bochenski added a comment -

          > I believe I found the problem, I trying to put in the 1.44.2 that should be release in a couple of days.

          FABRIZIO MANFREDI it's been a month now and I can't see any new releases after 1.44.1. Any updates?

          Show
          jbochenski Jakub Bochenski added a comment - > I believe I found the problem, I trying to put in the 1.44.2 that should be release in a couple of days. FABRIZIO MANFREDI it's been a month now and I can't see any new releases after 1.44.1. Any updates?
          Hide
          thoulen FABRIZIO MANFREDI added a comment -

          Can you test the 1.45 ? 

          Show
          thoulen FABRIZIO MANFREDI added a comment - Can you test the 1.45 ? 
          Hide
          jbochenski Jakub Bochenski added a comment -

          FABRIZIO MANFREDI the same problem is happening with 1.45

          Show
          jbochenski Jakub Bochenski added a comment - FABRIZIO MANFREDI the same problem is happening with 1.45

            People

            • Assignee:
              thoulen FABRIZIO MANFREDI
              Reporter:
              jbochenski Jakub Bochenski
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: