-
Type:
New Feature
-
Status: Open (View Workflow)
-
Priority:
Blocker
-
Resolution: Unresolved
-
Component/s: ec2-plugin
-
Labels:
-
Similar Issues:
When it is time to request a worker node, more than one can be requested.
It requests one, then rapidly checks for if its up (node needs a minute or so to come up when using spot instances), and fires another one, then another – until one of spots is finally there on time. This one becomes Jenkins slave, rest linger there until Spot Marketplace kills it.
Seems to be 1.42 specific: I reverted to version 1.41 and it seems to work fine.
This is how it looks in Jenkins log file:
(notice really short delays, only sir-bd78adpq gets into Jenkins slave list)
Jan 15, 2019 2:17:40 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
INFO: Spot instance id in provision: sir-dvfib1pn
Jan 15, 2019 2:17:40 PM hudson.plugins.ec2.EC2Cloud provision
INFO: SlaveTemplate
Jan 15, 2019 2:17:40 PM hudson.plugins.ec2.EC2Cloud provision
INFO: We have now 1 computers, waiting for 1 more
Jan 15, 2019 2:17:40 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning current_ami (ami-XXXXX) from ec2-ec2 cloud with 1 executors. Remaining excess workload: 0
Jan 15, 2019 2:17:40 PM hudson.plugins.ec2.EC2Cloud$1 call
WARNING: SlaveTemplate{ami='ami-XXXXX', labels=''}
. Node terminated is neither pending, neither running, its
{2}. Terminate provisioningJan 15, 2019 2:17:48 PM hudson.plugins.ec2.EC2Cloud provision
INFO: SlaveTemplate{ami='ami-XXXXX', labels=''}. Attempting to provision slave needed by excess workload of 1 units
Jan 15, 2019 2:17:49 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
INFO: Launching ami-XXXXX for template current_ami
Jan 15, 2019 2:17:50 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
INFO: Spot instance id in provision: sir-bieg943m
Jan 15, 2019 2:17:50 PM hudson.plugins.ec2.EC2Cloud provision
INFO: SlaveTemplate{ami='ami-XXXXX', labels=''}. Attempting provision finished, excess workload: 0
Jan 15, 2019 2:17:50 PM hudson.plugins.ec2.EC2Cloud provision
INFO: We have now 1 computers, waiting for 1 more
Jan 15, 2019 2:17:50 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning current_ami (ami-XXXXX) from ec2-ec2 cloud with 1 executors. Remaining excess workload: 0
Jan 15, 2019 2:17:50 PM hudson.plugins.ec2.EC2Cloud$1 call
WARNING: SlaveTemplate{ami='ami-XXXXX', labels=''}. Node terminated is neither pending, neither running, its {2}
. Terminate provisioning
Jan 15, 2019 2:17:58 PM hudson.plugins.ec2.EC2Cloud provision
INFO: SlaveTemplate
Jan 15, 2019 2:17:59 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
INFO: Launching ami-XXXXX for template current_ami
Jan 15, 2019 2:18:00 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
INFO: Spot instance id in provision: sir-5mtr9erp
Jan 15, 2019 2:18:00 PM hudson.plugins.ec2.EC2Cloud provision
INFO: SlaveTemplate{ami='ami-XXXXX', labels=''}
. Attempting provision finished, excess workload: 0
Jan 15, 2019 2:18:00 PM hudson.plugins.ec2.EC2Cloud provision
INFO: We have now 1 computers, waiting for 1 more
Jan 15, 2019 2:18:00 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning current_ami (ami-XXXXX) from ec2-ec2 cloud with 1 executors. Remaining excess workload: 0
Jan 15, 2019 2:18:00 PM hudson.plugins.ec2.EC2Cloud$1 call
WARNING: SlaveTemplate
Jan 15, 2019 2:18:08 PM hudson.plugins.ec2.EC2Cloud provision
INFO: SlaveTemplate{ami='ami-XXXXX', labels=''}
. Attempting to provision slave needed by excess workload of 1 units
Jan 15, 2019 2:18:09 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
INFO: Launching ami-XXXXX for template current_ami
Jan 15, 2019 2:18:10 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
INFO: Spot instance id in provision: sir-bd78adpq
Jan 15, 2019 2:18:10 PM hudson.plugins.ec2.EC2Cloud provision
INFO: SlaveTemplate
. Attempting provision finished, excess workload: 0
Jan 15, 2019 2:18:10 PM hudson.plugins.ec2.EC2Cloud provision
INFO: We have now 1 computers, waiting for 1 more
Jan 15, 2019 2:18:10 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning current_ami (ami-XXXXX) from ec2-ec2 cloud with 1 executors. Remaining excess workload: 0
Jan 15, 2019 2:18:21 PM hudson.plugins.ec2.EC2Cloud$1 call
> how long takes your node to be ready for use ?
Up to a minute with Spot instances. It used to be much worse few years ago – it could take up to 5 minutes back then.
Most of this time is spent in that marketplace – on-demand instance comes up 2-3x times faster from same AMI.