-
Bug
-
Resolution: Cannot Reproduce
-
Minor
-
None
-
Jenkins 2.60.3
ec2-plugin 1.63
Similar to the previous issues about lost instances (for both spot and demand), the problem occurs for us roughly once per week. As I investigated a bit, I want to share the details here - for EC2 Spot Instances in our case.
Sep 14, 2017 7:26:14 AM WARNING hudson.plugins.ec2.EC2AbstractSlave getInstance Failed to fetch EC2 instance: ESC (sir-xxxxxxx) com.amazonaws.services.ec2.model.AmazonEC2Exception: Invalid id: "<AMI Description> (sir-xxxxxx)" (Service: AmazonEC2; Status Code: 400; Error Code: InvalidInstanceID.Malformed; Request ID: e0563b6f-ce0d-44ae-ba0f-c654934487be) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:13551) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:13527) at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeInstances(AmazonEC2Client.java:6530) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:6506) at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:278) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:429) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:397) at hudson.plugins.ec2.EC2SpotSlave.isAlive(EC2SpotSlave.java:52) at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:43) at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:101) at java.lang.Thread.run(Thread.java:748)
Investigating CloudTrail logs confirms that it makes DescribeInstances requests to an instanceID "<AMI Description> (sir-xxxxxx)" (which AWS doesn't know, of course):
"eventSource": "ec2.amazonaws.com",
"eventName": "DescribeInstances",
"awsRegion": "eu-west-1",
"sourceIPAddress": "....",
"userAgent": "aws-sdk-java/1.11.119 Linux/4.9.27-14.31.amzn1.x86_64 OpenJDK_64-Bit_Server_VM/25.141-b15/1.8.0_141",
"errorCode": "Client.InvalidInstanceID.Malformed",
"errorMessage": "Invalid id: \"<AMI Description> (sir-xxxxxx)\"",
"requestParameters": {
"instancesSet": {
"items": [
{
"instanceId": "<AMI Description> (sir-xxxxx)"
}
]
},
Actually, it should look like this
"requestParameters": {
"instancesSet": {
"items": [
{
"instanceId": "i-067209....."
}
]
},
- is related to
-
-
- Closed
-
-
JENKINS-43774 Stop expensive runaway EC2 slaves by using the --valid-until option for spot instances
-
- Open
-
