Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-8882

Slave-Squatter + Heavy-Job plugin causes many executor threads to die

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Environment:
      Jenkins 1.397, heavy-job 1.0, slave-squatter trunk head (1.2-SNAPSHOT 10/07/2010)
      RHEL 6 (java-1.6.0-openjdk 1.6.0_17)
    • Similar Issues:
      Show 5 results

      Description

      We have been running the slave-squatter and heavy-jobs plugins since they were released under Hudson 1.377 without issue. Recently, we upgraded to Jenkins 1.397 and have started to run into situations where large numbers of executor threads die due to an exception thrown within hudson.model.queue.MappingWorksheet.

      This behavior has been observed before in JENKINS-7667.

      I believe the simple cause of the exception is that the load predictor routine in MappingWorksheet can come up with a maximum load that exceeds the number of available executors. When the code attempts to reduce the apparent executor pool (here), the underlying list throws the exception.

      The simple solution is to clamp minIdle to 0 if it is less than 0.

      That solution will prevent the exception that is killing executors (and is probably a good idea to implement if for no other reason than the load predictor is an extension point, and we cannot guarantee that implementers will always return <= the number of executors!). However, this does not address the underlying logical issue in the current load predictors (i.e., why is the predicted load greater than the number of executors???).

        Attachments

          Issue Links

            Activity

            Hide
            jsiirola jsiirola added a comment -

            Behavior reported here was observed in 7667.

            Show
            jsiirola jsiirola added a comment - Behavior reported here was observed in 7667.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: jsiirola
            Path:
            core/src/main/java/hudson/model/queue/MappingWorksheet.java
            http://jenkins-ci.org/commit/jenkins/da4ddde7564a15f6e8d9ee83f8499b7369dbe24b
            Log:
            [FIXED JENKINS-8882].

            Clamp the lower bound of the predicted minimum number of idle executors
            at 0 to prevent an IndexOutOfBoundsException.

            Compare: https://github.com/jenkinsci/jenkins/compare/a3354b1...da4ddde

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: jsiirola Path: core/src/main/java/hudson/model/queue/MappingWorksheet.java http://jenkins-ci.org/commit/jenkins/da4ddde7564a15f6e8d9ee83f8499b7369dbe24b Log: [FIXED JENKINS-8882] . Clamp the lower bound of the predicted minimum number of idle executors at 0 to prevent an IndexOutOfBoundsException. Compare: https://github.com/jenkinsci/jenkins/compare/a3354b1...da4ddde
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #887
            [FIXED JENKINS-8882].

            Kohsuke Kawaguchi : da4ddde7564a15f6e8d9ee83f8499b7369dbe24b
            Files :

            • core/src/main/java/hudson/model/queue/MappingWorksheet.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #887 [FIXED JENKINS-8882] . Kohsuke Kawaguchi : da4ddde7564a15f6e8d9ee83f8499b7369dbe24b Files : core/src/main/java/hudson/model/queue/MappingWorksheet.java
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: jsiirola
            Path:
            core/src/main/java/hudson/model/queue/MappingWorksheet.java
            http://jenkins-ci.org/commit/jenkins/8a2209fe73c5b5eef88bf2e1cffc5d3141060032
            Log:
            [FIXED JENKINS-8882].

            Clamp the lower bound of the predicted minimum number of idle executors
            at 0 to prevent an IndexOutOfBoundsException.
            (cherry picked from commit da4ddde7564a15f6e8d9ee83f8499b7369dbe24b)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: jsiirola Path: core/src/main/java/hudson/model/queue/MappingWorksheet.java http://jenkins-ci.org/commit/jenkins/8a2209fe73c5b5eef88bf2e1cffc5d3141060032 Log: [FIXED JENKINS-8882] . Clamp the lower bound of the predicted minimum number of idle executors at 0 to prevent an IndexOutOfBoundsException. (cherry picked from commit da4ddde7564a15f6e8d9ee83f8499b7369dbe24b)
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: jsiirola
            Path:
            core/src/main/java/hudson/model/queue/MappingWorksheet.java
            http://jenkins-ci.org/commit/jenkins/da4ddde7564a15f6e8d9ee83f8499b7369dbe24b
            Log:
            [FIXED JENKINS-8882].

            Clamp the lower bound of the predicted minimum number of idle executors
            at 0 to prevent an IndexOutOfBoundsException.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: jsiirola Path: core/src/main/java/hudson/model/queue/MappingWorksheet.java http://jenkins-ci.org/commit/jenkins/da4ddde7564a15f6e8d9ee83f8499b7369dbe24b Log: [FIXED JENKINS-8882] . Clamp the lower bound of the predicted minimum number of idle executors at 0 to prevent an IndexOutOfBoundsException.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: jsiirola
            Path:
            core/src/main/java/hudson/model/queue/MappingWorksheet.java
            http://jenkins-ci.org/commit/jenkins/da4ddde7564a15f6e8d9ee83f8499b7369dbe24b
            Log:
            [FIXED JENKINS-8882].

            Clamp the lower bound of the predicted minimum number of idle executors
            at 0 to prevent an IndexOutOfBoundsException.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: jsiirola Path: core/src/main/java/hudson/model/queue/MappingWorksheet.java http://jenkins-ci.org/commit/jenkins/da4ddde7564a15f6e8d9ee83f8499b7369dbe24b Log: [FIXED JENKINS-8882] . Clamp the lower bound of the predicted minimum number of idle executors at 0 to prevent an IndexOutOfBoundsException.

              People

              • Assignee:
                Unassigned
                Reporter:
                jsiirola jsiirola
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: