-
Improvement
-
Resolution: Fixed
-
Minor
-
None
EC2 Jenkins slaves sometimes fails to launch because AWS throttle the API calls. While this would normally not be a problem, this results in a bunch of Jenkins offline slaves which don't go away and can clutter up the UI.
When too many requests hit the EC2 API, Amazon responds with a 503 error like this one:
ERROR: Request limit exceeded. (Service: AmazonEC2; Status Code: 503; Error Code: RequestLimitExceeded; Request ID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX)
com.amazonaws.AmazonServiceException: Request limit exceeded. (Service: AmazonEC2; Status Code: 503; Error Code: RequestLimitExceeded; Request ID:xxxxx)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:934)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:506)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:273)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:8567)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4143)
at hudson.plugins.ec2.EC2Computer._describeInstanceOnce(EC2Computer.java:158)
at hudson.plugins.ec2.EC2Computer._describeInstance(EC2Computer.java:141)
at hudson.plugins.ec2.EC2Computer.getState(EC2Computer.java:117)
at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:60)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:232)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
The EC2 Jenkins plugin could better handle this kind of AWS exceptions. If this error is thrown, maybe it could attempt again after a few minutes.
Enhancing this behavior requires to wrap all API call to handle this error. Any advice on a way to do it properly ?
