-
Bug
-
Resolution: Fixed
-
Blocker
-
None
-
Jenkins 2.17
ECS-plugin 1.4
With the latest plugin version (1.4) my tasks fail to run on the ecs slaves. They sit waiting for executors forever. I made sure I had two servers with plenty of resources connected to ECS.
It looks like the task is assigned to a node but then immediately disconnects.
After downgrading to version 1.3 it all started working perfectly again.
I tested it with the same number of servers, same tasks, same jenkins versions, etc.
See the logs below for both versions.
Version 1.4 failures:
Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources
INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/ceb4eece-2abe-4f19-aad3-8d3cbd18a7ee has 3,767mb of free memory. 3,000mb are required
Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources
INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/ceb4eece-2abe-4f19-aad3-8d3cbd18a7ee has 1,024 units of free cpu. 512 units are required
Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
INFO: Created Slave: JenkinsSlavesECS-16e94238750e85
Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService runEcsTask
WARNING: Slave JenkinsSlavesECS-16e94238750e85 - Failure to run task with definition arn:aws:ecs:eu-west-1:AWS_ACC:task-definition/jenkins-slave:78 on ECS cluster arn:aws:ecs:eu-west-1:AWS_ACC:cluster/JenkinsSlaves
Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService runEcsTask
WARNING: Slave JenkinsSlavesECS-16e94238750e85 - Failure reason=RESOURCE:MEMORY, arn=arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601
Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$2 run
WARNING: Provisioned agent ECS Slave BackendTests failed to launch
java.lang.IllegalStateException: ECS Slave JenkinsSlavesECS-16e9422efc9bc3 (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/b6621ca4-b397-46ad-9659-9c6fe1633b32) not connected since 7/6/16 8:37 AM seconds
at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:278)
at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:219)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$2 run
WARNING: Provisioned agent ECS Slave BackendTests failed to launch
hudson.AbortException: Failed to run slave container JenkinsSlavesECS-16e94238750e85
at com.cloudbees.jenkins.plugins.amazonecs.ECSService.runEcsTask(ECSService.java:157)
at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:250)
at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:219)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning ECS Slave BackendTests from JenkinsSlavesECS with 1 executors. Remaining excess workload: 0.816
Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning ECS Slave BackendTests from JenkinsSlavesECS with 1 executors. Remaining excess workload: -0.184
Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources..........
Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources
INFO: Resources found in instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601: [Unknown macro: {Name},
Unknown macro: {Name},
Unknown macro: {Name},
Unknown macro: {Name}]
Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources
INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601 has 3,767mb of free memory. 3,000mb are required
Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources
INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601 has 1,024 units of free cpu. 512 units are required
Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
INFO: Created Slave: JenkinsSlavesECS-16e9448c0da1df
Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
INFO: Slave JenkinsSlavesECS-16e9448c0da1df - Slave Task Started : arn:aws:ecs:eu-west-1:AWS_ACC:task/d8ad05da-902d-4c10-8b58-c55f612bb0da
Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
WARNING: ECS Slave JenkinsSlavesECS-16e9448c0da1df (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/d8ad05da-902d-4c10-8b58-c55f612bb0da) not connected since 7/6/16 8:37 AM seconds
Jul 06, 2016 8:37:43 AM jenkins.model.Jenkins$21 onAttained
INFO: Started termination
Jul 06, 2016 8:37:43 AM jenkins.model.Jenkins$21 onAttained
INFO: Completed termination
Jul 06, 2016 8:37:43 AM jenkins.model.Jenkins _cleanUpDisconnectComputers
INFO: Starting node disconnection
After downgrade to version 1.3, working well:
Jul 06, 2016 8:40:06 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning ECS Slave BackendTests from JenkinsSlavesECS with 1 executors. Remaining excess workload: 0.303
Jul 06, 2016 8:40:06 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
INFO: Created Slave: JenkinsSlavesECS-16e965bf8c0ffe
Jul 06, 2016 8:40:07 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
INFO: Slave JenkinsSlavesECS-16e965bf8c0ffe - Slave Task Started : arn:aws:ecs:eu-west-1:AWS_ACC:task/efa27aa4-599c-4da8-89fb-a8e21740882c
Jul 06, 2016 8:40:11 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted connection #1 from /XX.XX.XX.XX:YY
Jul 06, 2016 8:40:15 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
INFO: ECS Slave JenkinsSlavesECS-16e965bf8c0ffe (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/efa27aa4-599c-4da8-89fb-a8e21740882c) connected
Jul 06, 2016 8:40:16 AM hudson.slaves.NodeProvisioner$2 run
INFO: ECS Slave BackendTests provisioning successfully completed. We have now 3 computer(s)