Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23787

EC2-plugin not spooling up stopped nodes - "still in the queue ... all nodes of label ... are offline"

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      The Jenkins EC2 plugin no longer launches stopped nodes. Unfortunately I'm not sure exactly when it stopped working - I wasn't sure that was the issue until later, due to unrelated issues caused by too many nodes spawning and having to be killed.

      If I use Manage Jenkins -> Manage Nodes to start a stopped EC2 node that a build is waiting on manually, the build proceeds.

      Builds succeed when the EC2 plugin spawns a new node for the first time. It's only a problem if the node is stopped for idleness - the plugin doesn't seem to restart it.

      Builds get stuck with output like:

      Triggering bdr_linux ? x64,debian7
      Triggering bdr_linux ? x86,amazonlinux201209
      Triggering bdr_linux ? x86,debian6
      Triggering bdr_linux ? x64,amazonlinux201209
      Configuration bdr_linux ? x86,amazonlinux201209 is still in the queue: Amazon Linux 2012.09 EBS 32-bit  (i-b848fbfa) is offline
      Configuration bdr_linux ? x86,amazonlinux201209 is still in the queue: All nodes of label ?amazonlinux201209&&x86? are offline
      

      where there's at least one node with that label stopped, ready to start and use.

      There's no sign that any attempt is made to start the node.

        Attachments

          Issue Links

            Activity

            Hide
            fengxx Ted added a comment - - edited

            I think the commit https://github.com/jenkinsci/ec2-plugin/commit/c9d69a5da8c9be094701d4c191ba7b1d06c200c9 breaks plugin, user can not launch multiple instances for same AMI since DescribeInstancesRequest returns all instances instead of stopped instances, and no instances can be launched if there is any running

            below line was removed
            diFilters.add(new Filter("instance-state-name").withValues(InstanceStateName.Stopped.toString(), InstanceStateName.Stopping.toString()));

            Show
            fengxx Ted added a comment - - edited I think the commit https://github.com/jenkinsci/ec2-plugin/commit/c9d69a5da8c9be094701d4c191ba7b1d06c200c9 breaks plugin, user can not launch multiple instances for same AMI since DescribeInstancesRequest returns all instances instead of stopped instances, and no instances can be launched if there is any running below line was removed diFilters.add(new Filter("instance-state-name").withValues(InstanceStateName.Stopped.toString(), InstanceStateName.Stopping.toString()));
            Hide
            francisu Francis Upton added a comment -

            @Ted, I don't see how that line broke it. Note that I changed EC2Cloud to look up any running (or pending) instances and check that they are known. Only if they are known will they count against the existing instances. I will test it again though, to try multiple instances with the same AMI. The reason the filter was removed is that we want to see if there is a running instance that's not known, and if so, use that one.

            Show
            francisu Francis Upton added a comment - @Ted, I don't see how that line broke it. Note that I changed EC2Cloud to look up any running (or pending) instances and check that they are known. Only if they are known will they count against the existing instances. I will test it again though, to try multiple instances with the same AMI. The reason the filter was removed is that we want to see if there is a running instance that's not known, and if so, use that one.
            Hide
            francisu Francis Upton added a comment -

            I believe I have now fixed this.

            Show
            francisu Francis Upton added a comment - I believe I have now fixed this.
            Hide
            francisu Francis Upton added a comment -

            Should be released in 1.30

            Show
            francisu Francis Upton added a comment - Should be released in 1.30
            Hide
            francisu Francis Upton added a comment -

            Fixed a problem with this of it not respecting the instance caps for on-demand nodes.

            Show
            francisu Francis Upton added a comment - Fixed a problem with this of it not respecting the instance caps for on-demand nodes.

              People

              • Assignee:
                francisu Francis Upton
                Reporter:
                ringerc Craig Ringer
              • Votes:
                2 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: