Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56871

EC2 Plugin fails to terminate spot instances after agent deleted

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: ec2-plugin
    • Labels:
      None
    • Environment:
      CJE Managed Master 2.138.1.2-rolling
      EC2Plugin 1.42
    • Similar Issues:

      Description

      We had a case where idle instances that were supposedly deleted still running on EC2.
      To reproduce:
      Launch EC2 template with spot instance enabled
      Immediately delete that instance.
      On the Spot request console, the request will be "cancelled"
      But the associated EC2 instance is still running, and orphaned.

      We had a case where we had so many of these orphaned instances that we reached the instance cap of our account.

        Attachments

          Activity

          Hide
          ravibagri2 Ravi Bagri added a comment - - edited

          I recently faced the same issue. later after investing into my cloud-trail logs i found that jenkins role had the permission to cancel the spot request but the permission to terminate instance was missing. So that's why the spot request was getting cancelled but the instance was still persisting. 

          Below is the log snippet.

          "eventName": "TerminateInstances",
          "awsRegion": "us-west-2",
          "userAgent": "aws-sdk-java/1.11.457 Linux/4.9.0-6-amd64 OpenJDK_64-Bit_Server_VM/25.212-b01 java/1.8.0_212 groovy/2.4.12",
          "errorCode": "Client.UnauthorizedOperation",
          "errorMessage": "You are not authorized to perform this operation. Encoded authorization failure message:

          After adding the appropriate permission it seems to have resolved. 

          Show
          ravibagri2 Ravi Bagri added a comment - - edited I recently faced the same issue. later after investing into my cloud-trail logs i found that jenkins role had the permission to cancel the spot request but the permission to terminate instance was missing. So that's why the spot request was getting cancelled but the instance was still persisting.  Below is the log snippet. "eventName": "TerminateInstances", "awsRegion": "us-west-2", "userAgent": "aws-sdk-java/1.11.457 Linux/4.9.0-6-amd64 OpenJDK_64-Bit_Server_VM/25.212-b01 java/1.8.0_212 groovy/2.4.12", "errorCode": "Client.UnauthorizedOperation", "errorMessage": "You are not authorized to perform this operation. Encoded authorization failure message: After adding the appropriate permission it seems to have resolved. 
          Hide
          thoulen FABRIZIO MANFREDI added a comment -

          thanks for the investigation, did you use the iam roles suggested in the wiki page ? 

          Show
          thoulen FABRIZIO MANFREDI added a comment - thanks for the investigation, did you use the iam roles suggested in the wiki page ? 

            People

            • Assignee:
              thoulen FABRIZIO MANFREDI
              Reporter:
              gordonshieh Gordon Shieh
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: