Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36468

Tasks fail to run on ECS after upgrade to 1.4

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • amazon-ecs-plugin
    • None
    • Jenkins 2.17
      ECS-plugin 1.4

      With the latest plugin version (1.4) my tasks fail to run on the ecs slaves. They sit waiting for executors forever. I made sure I had two servers with plenty of resources connected to ECS.
      It looks like the task is assigned to a node but then immediately disconnects.

      After downgrading to version 1.3 it all started working perfectly again.

      I tested it with the same number of servers, same tasks, same jenkins versions, etc.

      See the logs below for both versions.

      Version 1.4 failures:

      Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources
      INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/ceb4eece-2abe-4f19-aad3-8d3cbd18a7ee has 3,767mb of free memory. 3,000mb are required
      Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources
      INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/ceb4eece-2abe-4f19-aad3-8d3cbd18a7ee has 1,024 units of free cpu. 512 units are required
      Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
      INFO: Created Slave: JenkinsSlavesECS-16e94238750e85
      Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService runEcsTask
      WARNING: Slave JenkinsSlavesECS-16e94238750e85 - Failure to run task with definition arn:aws:ecs:eu-west-1:AWS_ACC:task-definition/jenkins-slave:78 on ECS cluster arn:aws:ecs:eu-west-1:AWS_ACC:cluster/JenkinsSlaves
      Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService runEcsTask
      WARNING: Slave JenkinsSlavesECS-16e94238750e85 - Failure reason=RESOURCE:MEMORY, arn=arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601
      Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$2 run
      WARNING: Provisioned agent ECS Slave BackendTests failed to launch
      java.lang.IllegalStateException: ECS Slave JenkinsSlavesECS-16e9422efc9bc3 (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/b6621ca4-b397-46ad-9659-9c6fe1633b32) not connected since 7/6/16 8:37 AM seconds
      at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:278)
      at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:219)
      at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

      Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$2 run
      WARNING: Provisioned agent ECS Slave BackendTests failed to launch
      hudson.AbortException: Failed to run slave container JenkinsSlavesECS-16e94238750e85
      at com.cloudbees.jenkins.plugins.amazonecs.ECSService.runEcsTask(ECSService.java:157)
      at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:250)
      at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:219)
      at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

      Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
      INFO: Started provisioning ECS Slave BackendTests from JenkinsSlavesECS with 1 executors. Remaining excess workload: 0.816
      Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
      INFO: Started provisioning ECS Slave BackendTests from JenkinsSlavesECS with 1 executors. Remaining excess workload: -0.184
      Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources

      ..........

      Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources
      INFO: Resources found in instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601: [

      Unknown macro: {Name}

      ,

      Unknown macro: {Name}

      ,

      Unknown macro: {Name}

      ,

      Unknown macro: {Name}

      ]
      Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources
      INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601 has 3,767mb of free memory. 3,000mb are required
      Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources
      INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601 has 1,024 units of free cpu. 512 units are required
      Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
      INFO: Created Slave: JenkinsSlavesECS-16e9448c0da1df
      Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
      INFO: Slave JenkinsSlavesECS-16e9448c0da1df - Slave Task Started : arn:aws:ecs:eu-west-1:AWS_ACC:task/d8ad05da-902d-4c10-8b58-c55f612bb0da
      Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
      WARNING: ECS Slave JenkinsSlavesECS-16e9448c0da1df (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/d8ad05da-902d-4c10-8b58-c55f612bb0da) not connected since 7/6/16 8:37 AM seconds
      Jul 06, 2016 8:37:43 AM jenkins.model.Jenkins$21 onAttained
      INFO: Started termination
      Jul 06, 2016 8:37:43 AM jenkins.model.Jenkins$21 onAttained
      INFO: Completed termination
      Jul 06, 2016 8:37:43 AM jenkins.model.Jenkins _cleanUpDisconnectComputers
      INFO: Starting node disconnection

      After downgrade to version 1.3, working well:

      Jul 06, 2016 8:40:06 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
      INFO: Started provisioning ECS Slave BackendTests from JenkinsSlavesECS with 1 executors. Remaining excess workload: 0.303
      Jul 06, 2016 8:40:06 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
      INFO: Created Slave: JenkinsSlavesECS-16e965bf8c0ffe
      Jul 06, 2016 8:40:07 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
      INFO: Slave JenkinsSlavesECS-16e965bf8c0ffe - Slave Task Started : arn:aws:ecs:eu-west-1:AWS_ACC:task/efa27aa4-599c-4da8-89fb-a8e21740882c
      Jul 06, 2016 8:40:11 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
      INFO: Accepted connection #1 from /XX.XX.XX.XX:YY
      Jul 06, 2016 8:40:15 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
      INFO: ECS Slave JenkinsSlavesECS-16e965bf8c0ffe (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/efa27aa4-599c-4da8-89fb-a8e21740882c) connected
      Jul 06, 2016 8:40:16 AM hudson.slaves.NodeProvisioner$2 run
      INFO: ECS Slave BackendTests provisioning successfully completed. We have now 3 computer(s)

            luissilva Luis Silva
            luissilva Luis Silva
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: