Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-8882

Slave-Squatter + Heavy-Job plugin causes many executor threads to die

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Environment:
      Jenkins 1.397, heavy-job 1.0, slave-squatter trunk head (1.2-SNAPSHOT 10/07/2010)
      RHEL 6 (java-1.6.0-openjdk 1.6.0_17)

      Description

      We have been running the slave-squatter and heavy-jobs plugins since they were released under Hudson 1.377 without issue. Recently, we upgraded to Jenkins 1.397 and have started to run into situations where large numbers of executor threads die due to an exception thrown within hudson.model.queue.MappingWorksheet.

      This behavior has been observed before in JENKINS-7667.

      I believe the simple cause of the exception is that the load predictor routine in MappingWorksheet can come up with a maximum load that exceeds the number of available executors. When the code attempts to reduce the apparent executor pool (here), the underlying list throws the exception.

      The simple solution is to clamp minIdle to 0 if it is less than 0.

      That solution will prevent the exception that is killing executors (and is probably a good idea to implement if for no other reason than the load predictor is an extension point, and we cannot guarantee that implementers will always return <= the number of executors!). However, this does not address the underlying logical issue in the current load predictors (i.e., why is the predicted load greater than the number of executors???).

        Issue Links

          Activity

          jsiirola jsiirola created issue -
          Hide
          jsiirola jsiirola added a comment -

          Behavior reported here was observed in 7667.

          Show
          jsiirola jsiirola added a comment - Behavior reported here was observed in 7667.
          jsiirola jsiirola made changes -
          Field Original Value New Value
          Link This issue is related to JENKINS-7667 [ JENKINS-7667 ]
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: jsiirola
          Path:
          core/src/main/java/hudson/model/queue/MappingWorksheet.java
          http://jenkins-ci.org/commit/jenkins/da4ddde7564a15f6e8d9ee83f8499b7369dbe24b
          Log:
          [FIXED JENKINS-8882].

          Clamp the lower bound of the predicted minimum number of idle executors
          at 0 to prevent an IndexOutOfBoundsException.

          Compare: https://github.com/jenkinsci/jenkins/compare/a3354b1...da4ddde

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: jsiirola Path: core/src/main/java/hudson/model/queue/MappingWorksheet.java http://jenkins-ci.org/commit/jenkins/da4ddde7564a15f6e8d9ee83f8499b7369dbe24b Log: [FIXED JENKINS-8882] . Clamp the lower bound of the predicted minimum number of idle executors at 0 to prevent an IndexOutOfBoundsException. Compare: https://github.com/jenkinsci/jenkins/compare/a3354b1...da4ddde
          scm_issue_link SCM/JIRA link daemon made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          dogfood dogfood added a comment -

          Integrated in jenkins_main_trunk #887
          [FIXED JENKINS-8882].

          Kohsuke Kawaguchi : da4ddde7564a15f6e8d9ee83f8499b7369dbe24b
          Files :

          • core/src/main/java/hudson/model/queue/MappingWorksheet.java
          Show
          dogfood dogfood added a comment - Integrated in jenkins_main_trunk #887 [FIXED JENKINS-8882] . Kohsuke Kawaguchi : da4ddde7564a15f6e8d9ee83f8499b7369dbe24b Files : core/src/main/java/hudson/model/queue/MappingWorksheet.java
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: jsiirola
          Path:
          core/src/main/java/hudson/model/queue/MappingWorksheet.java
          http://jenkins-ci.org/commit/jenkins/8a2209fe73c5b5eef88bf2e1cffc5d3141060032
          Log:
          [FIXED JENKINS-8882].

          Clamp the lower bound of the predicted minimum number of idle executors
          at 0 to prevent an IndexOutOfBoundsException.
          (cherry picked from commit da4ddde7564a15f6e8d9ee83f8499b7369dbe24b)

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: jsiirola Path: core/src/main/java/hudson/model/queue/MappingWorksheet.java http://jenkins-ci.org/commit/jenkins/8a2209fe73c5b5eef88bf2e1cffc5d3141060032 Log: [FIXED JENKINS-8882] . Clamp the lower bound of the predicted minimum number of idle executors at 0 to prevent an IndexOutOfBoundsException. (cherry picked from commit da4ddde7564a15f6e8d9ee83f8499b7369dbe24b)
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: jsiirola
          Path:
          core/src/main/java/hudson/model/queue/MappingWorksheet.java
          http://jenkins-ci.org/commit/jenkins/da4ddde7564a15f6e8d9ee83f8499b7369dbe24b
          Log:
          [FIXED JENKINS-8882].

          Clamp the lower bound of the predicted minimum number of idle executors
          at 0 to prevent an IndexOutOfBoundsException.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: jsiirola Path: core/src/main/java/hudson/model/queue/MappingWorksheet.java http://jenkins-ci.org/commit/jenkins/da4ddde7564a15f6e8d9ee83f8499b7369dbe24b Log: [FIXED JENKINS-8882] . Clamp the lower bound of the predicted minimum number of idle executors at 0 to prevent an IndexOutOfBoundsException.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: jsiirola
          Path:
          core/src/main/java/hudson/model/queue/MappingWorksheet.java
          http://jenkins-ci.org/commit/jenkins/da4ddde7564a15f6e8d9ee83f8499b7369dbe24b
          Log:
          [FIXED JENKINS-8882].

          Clamp the lower bound of the predicted minimum number of idle executors
          at 0 to prevent an IndexOutOfBoundsException.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: jsiirola Path: core/src/main/java/hudson/model/queue/MappingWorksheet.java http://jenkins-ci.org/commit/jenkins/da4ddde7564a15f6e8d9ee83f8499b7369dbe24b Log: [FIXED JENKINS-8882] . Clamp the lower bound of the predicted minimum number of idle executors at 0 to prevent an IndexOutOfBoundsException.

            People

            • Assignee:
              Unassigned
              Reporter:
              jsiirola jsiirola
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: