Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27565

Nodes can be removed as idle before the assigned tasks have started

    Details

    • Similar Issues:

      Description

      Quite a number of different manifestations of this observed by a number of our customers using different cloud providers. In common is the use of a "single-shot" style retention strategy, though the root cause is observable with great care when using any retention strategy other than Always.

      The basic issue is that you cannot determine if a node is idle unless you hold the Queue lock as that is the only way to ensure that the Queue is not in the process of assigning work to the node you are removing.

      Symptoms include:

      • Build logs that claim the job was executed on "master" even though the job is tied to a specific label that master does not have. The build log will have been "unable to be determined"
      • Build logs where the node is gone just as soon as the job starts
        2015-03-05 13:27:55.101 Started by upstream project "____" build number ___ 
        2015-03-05 13:27:55.102 originally caused by: 
        2015-03-05 13:27:55.103 Started by user ____ 
        2015-03-05 13:27:55.437 FATAL: no longer a configured node for ____ 
        2015-03-05 13:27:55.440 java.lang.IllegalStateException: no longer a configured node for ____ 
        2015-03-05 13:27:55.440 at hudson.model.AbstractBuild$AbstractBuildExecution.getCurrentNode(AbstractBuild.java:452) 
        2015-03-05 13:27:55.440 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:484) 
        2015-03-05 13:27:55.441 at hudson.model.Run.execute(Run.java:1745) 
        2015-03-05 13:27:55.441 at hudson.model.Build.run(Build.java:113) 
        2015-03-05 13:27:55.441 at hudson.model.ResourceController.execute(ResourceController.java:89) 
        2015-03-05 13:27:55.441 at hudson.model.Executor.run(Executor.java:240)
        

        Attachments

          Issue Links

            Activity

            stephenconnolly Stephen Connolly created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 1596 (Web Link)" [ 12179 ]
            jglick Jesse Glick made changes -
            Labels queue slave threads
            jglick Jesse Glick made changes -
            Link This issue is blocking JENKINS-20046 [ JENKINS-20046 ]
            pablaasmo Per Arnold Blaasmo made changes -
            Link This issue is blocking JENKINS-27476 [ JENKINS-27476 ]
            stephenconnolly Stephen Connolly made changes -
            Status In Progress [ 3 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            jglick Jesse Glick made changes -
            Link This issue depends on JENKINS-27700 [ JENKINS-27700 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-27708 [ JENKINS-27708 ]
            szubster Tomasz Szuba made changes -
            Link This issue is related to JENKINS-28690 [ JENKINS-28690 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-20967 [ JENKINS-20967 ]
            jglick Jesse Glick made changes -
            Link This issue depends on JENKINS-32517 [ JENKINS-32517 ]
            stephenconnolly Stephen Connolly made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 161775 ] JNJira + In-Review [ 208569 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-56403 [ JENKINS-56403 ]

              People

              • Assignee:
                stephenconnolly Stephen Connolly
                Reporter:
                stephenconnolly Stephen Connolly
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: