Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49705

EC2 plugin fails to launch agents after some time.

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: ec2-plugin
    • Labels:
    • Environment:
      Jenkins 2.89.4
      Ec2 Plugin 1.38
    • Similar Issues:

      Description

      This issue seems similar to JENKINS-36516.  Jenkins is able to launch EC2 build agents and after some time will stop launching new agents.  We see the below exception in the system log.

      Poking at the cloud settings in "Manage Jenkins" seems to do something that clears this error.  I suspect it is refreshing cached or stale credentials/tokens or something similar.

      Jenkins version 2.89.4
      Ec2 Plugin 1.38
      Jenkins is running in a docker container behind an nginx reverse proxy.

      Is there a way to troubleshoot or debug this issue further?  I would be very interested in assisting with investigation/debug/fix as I have an environment where this happens frequently.   

      {{Exception during provisioning com.amazonaws.services.ec2.model.AmazonEC2Exception: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 0916d4bd-d804-4ade-af52-0786f9a4aaa0) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:15654) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:15630) at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeInstances(AmazonEC2Client.java:7621) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:7597) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:7633) at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:363) at hudson.plugins.ec2.EC2Cloud.getPossibleNewSlavesCount(EC2Cloud.java:502) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:522) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:551) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) }}

        Attachments

          Activity

          Hide
          jreslock Jason Reslock added a comment -

          I was able to work around/fix this by setting up an appropriate instance profile to allow the Jenkins master to launch build agents using the instance profile instead of using credentials.  If anyone else experiences similar behavior, try the instance profile method to see if it resolves the problem.

          Show
          jreslock Jason Reslock added a comment - I was able to work around/fix this by setting up an appropriate instance profile to allow the Jenkins master to launch build agents using the instance profile instead of using credentials.  If anyone else experiences similar behavior, try the instance profile method to see if it resolves the problem.
          Hide
          francisu Francis Upton added a comment -

          Another instance of this:

          Hello Francis,

          I am testing the viability of using the EC2 plugin and I am running to an issue where the plugin works for a little while and then stops. I have it configured to terminate after 30 minutes of inactivity. I’ve seen it work twice after it terminates, but after that I see the message below in the Jenkins log. Any idea what could be causing this error? I am using Jenkins 7.3.3 with the previous version of the plugin. The current version, 1.37, was not removing the instances after the timeout.

          I appreciate any help you can give me.

          Nov 30, 2017 6:47:15 PM hudson.plugins.ec2.EC2Cloud provision

          INFO: Attempting to provision slave from template hudson.plugins.ec2.SlaveTemplate@78984608 needed by excess workload of 1 units of label 'ec2-tag'

          Nov 30, 2017 6:47:15 PM hudson.plugins.ec2.EC2Cloud provision

          WARNING: Exception during provisioning

          com.amazonaws.services.ec2.model.AmazonEC2Exception: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: c92c94d1-481e-4257-93d4-cd1d067da2d4)

          at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)

          at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)

          at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)

          at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)

          at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)

          at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)

          at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)

          at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)

          at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)

          at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:14388)

          at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:14364)

          at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeInstances(AmazonEC2Client.java:7020)

          at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:6996)

          at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:7032)

          at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:363)

          at hudson.plugins.ec2.EC2Cloud.getPossibleNewSlavesCount(EC2Cloud.java:498)

          at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:518)

          at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:547)

          at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715)

          at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320)

          at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61)

          at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)

          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)

          at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)

          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

          at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

          at java.lang.Thread.run(Thread.java:748)

          Show
          francisu Francis Upton added a comment - Another instance of this: Hello Francis, I am testing the viability of using the EC2 plugin and I am running to an issue where the plugin works for a little while and then stops. I have it configured to terminate after 30 minutes of inactivity. I’ve seen it work twice after it terminates, but after that I see the message below in the Jenkins log. Any idea what could be causing this error? I am using Jenkins 7.3.3 with the previous version of the plugin. The current version, 1.37, was not removing the instances after the timeout. I appreciate any help you can give me. Nov 30, 2017 6:47:15 PM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting to provision slave from template hudson.plugins.ec2.SlaveTemplate@78984608 needed by excess workload of 1 units of label 'ec2-tag' Nov 30, 2017 6:47:15 PM hudson.plugins.ec2.EC2Cloud provision WARNING: Exception during provisioning com.amazonaws.services.ec2.model.AmazonEC2Exception: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: c92c94d1-481e-4257-93d4-cd1d067da2d4) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:14388) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:14364) at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeInstances(AmazonEC2Client.java:7020) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:6996) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:7032) at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:363) at hudson.plugins.ec2.EC2Cloud.getPossibleNewSlavesCount(EC2Cloud.java:498) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:518) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:547) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
          Hide
          bithead Ben Copeland added a comment - - edited

          I have exactly the same issue here as well. I have confirmed my clocks are correct and synchronized with ntp. The jenkins master just gives up trying to contact the slave, the slave as a result is orphaned, in running state and reachable via SSH from the jenkins master. I am using a VPC peer in AWS. The Jenkins master is in an AWS "infrastructure" account. 

          com.amazonaws.services.ec2.model.AmazonEC2Exception: Request has expired.

           

          • Jenkins ver. 2.150.2 (running in a docker container)
          • EC2 Plugin v. 1.39
          • AWS SDK Plugin v. 1.11.457

          this is working with a Jenkins master running similar network setup

          • Jenkins ver. 2.10 (not in a container)
          • EC2 Plugin v. 1.35
          • AWS SDK plugin 1.10.16

           Note: This issue is blocking us from upgrading Jenkins. 

           

          Show
          bithead Ben Copeland added a comment - - edited I have exactly the same issue here as well. I have confirmed my clocks are correct and synchronized with ntp. The jenkins master just gives up trying to contact the slave, the slave as a result is orphaned, in running state and reachable via SSH from the jenkins master. I am using a VPC peer in AWS. The Jenkins master is in an AWS "infrastructure" account.  com.amazonaws.services.ec2.model.AmazonEC2Exception: Request has expired.   Jenkins ver. 2.150.2  (running in a docker container) EC2 Plugin v. 1.39 AWS SDK Plugin v. 1.11.457 this is working with a Jenkins master running similar network setup Jenkins ver. 2.10 (not in a container) EC2 Plugin v. 1.35 AWS SDK plugin 1.10.16  Note: This issue is blocking us from upgrading Jenkins.   
          Hide
          bithead Ben Copeland added a comment - - edited

          Was this not fixed in JENKINS-26854 ? I still see problems with EC2 Plugin 1.42 and I have confirmed with AWS support the session is in fact expiring and not being refreshed. 

          Show
          bithead Ben Copeland added a comment - - edited Was this not fixed in JENKINS-26854  ? I still see problems with EC2 Plugin 1.42 and I have confirmed with AWS support the session is in fact expiring and not being refreshed. 

            People

            • Assignee:
              francisu Francis Upton
              Reporter:
              jreslock Jason Reslock
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: