Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12994

Quiet period is blocking other jobs in queue

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Environment:
      Platform: All, OS: All

      Description

      Starting from version 1.452, a job in queue waiting for the quiet period to pass is blocking all other jobs that have been queued after it.

      Example:

      1. Job A has quiet period set to 10 seconds
      2. Job B has quiet period set to 300 seconds
      3. Job C has quiet period set to 100 seconds
      4. Jobs get queued in A-B-C order.
      5. Job A starts executing after 10 seconds
      6. After 100 seconds, status of job C changes to "pending - Waiting for next available executor" even if free executors are available
      7. After 300 seconds both jobs B and C start executing

      This problem is hardly visible when short quiet periods are used but becomes problematic with longer quiet periods or plugins like Naginator that will increase the quiet period to hours if the builds fail enough.

      Version 1.451 is the last release where this problem isn't visible. From public releases, at least 1.452 and 1.454 are affected.

        Issue Links

          Activity

          Hide
          mwebber mwebber added a comment -

          I'm seeing this, or something like it, as well.

          I have also recently installed the "Throttle Concurrent Builds Plugin" and the "Exclusion-Plugin", so initially I though it might be related to that, but this ticket suggests it's a core problem.

          Show
          mwebber mwebber added a comment - I'm seeing this, or something like it, as well. I have also recently installed the "Throttle Concurrent Builds Plugin" and the "Exclusion-Plugin", so initially I though it might be related to that, but this ticket suggests it's a core problem.
          Hide
          allenservedio allenservedio added a comment -

          I am seeing the same thing in 1.454. Even worse, I am starting to see deadlocks around this. I had to go in and kill a three jobs where two of them were stuck in an infinite loop of quiet periods and the third (a scheduled job) was stuck waiting for an executor. As I was not watching the builds, they were stuck in that state for over an hour before I caught them (the quiet period for the other two jobs was 190 seconds, it just kept cycling through quiet periods).

          Show
          allenservedio allenservedio added a comment - I am seeing the same thing in 1.454. Even worse, I am starting to see deadlocks around this. I had to go in and kill a three jobs where two of them were stuck in an infinite loop of quiet periods and the third (a scheduled job) was stuck waiting for an executor. As I was not watching the builds, they were stuck in that state for over an hour before I caught them (the quiet period for the other two jobs was 190 seconds, it just kept cycling through quiet periods).
          Hide
          bruce Dirk Kuypers added a comment -

          I am seeing this, too, since 1.452 I think. Maybe this is a hint (or just coincidence?): I never had to kill the pending jobs, I went into "Configure Jenkins->Configure System" and just pressed back, without changing/saving anything. After that the queues got filled again. Maybe others could try this, too, to confirm if this helps.

          Show
          bruce Dirk Kuypers added a comment - I am seeing this, too, since 1.452 I think. Maybe this is a hint (or just coincidence?): I never had to kill the pending jobs, I went into "Configure Jenkins->Configure System" and just pressed back, without changing/saving anything. After that the queues got filled again. Maybe others could try this, too, to confirm if this helps.
          Hide
          sti Sami Tikka added a comment -

          I upgraded from 1.451 to 1.453 and all my builds started staying in the queue forever. Exactly same symptom: Build says it is waiting for available executor even though all slaves were online with free executors. Restarting Jenkins did not help. Downgraded back to 1.451 and the problem went away.

          Show
          sti Sami Tikka added a comment - I upgraded from 1.451 to 1.453 and all my builds started staying in the queue forever. Exactly same symptom: Build says it is waiting for available executor even though all slaves were online with free executors. Restarting Jenkins did not help. Downgraded back to 1.451 and the problem went away.
          Hide
          vlatombe Vincent Latombe added a comment - - edited

          I've dug in the history between 1.451 and 1.452, and it seems the issue is coming from the following commit : https://github.com/jenkinsci/jenkins/commit/1ac4872ecdc500c63c0dbfce5e4beb5ce78369e5

          You can see at line 920-921 that the method returns if the first element of the waiting list is not ready to build, so it doesn't consider other elements in the waiting list.
          waitingList is a TreeSet<WaitingItem> and it looks like it is not properly sorted by timestamp, meaning that the compareTo method is not properly implemented.

          Show
          vlatombe Vincent Latombe added a comment - - edited I've dug in the history between 1.451 and 1.452, and it seems the issue is coming from the following commit : https://github.com/jenkinsci/jenkins/commit/1ac4872ecdc500c63c0dbfce5e4beb5ce78369e5 You can see at line 920-921 that the method returns if the first element of the waiting list is not ready to build, so it doesn't consider other elements in the waiting list. waitingList is a TreeSet<WaitingItem> and it looks like it is not properly sorted by timestamp, meaning that the compareTo method is not properly implemented.
          Hide
          cnogradi Chris Nogradi added a comment -

          I tried upgrading to 1.452 and 1.455 and both behave as described above with all jobs remaining in 'Waiting for next executor' state with all nodes free. Downgrading to 1.451 restores functionality.

          Show
          cnogradi Chris Nogradi added a comment - I tried upgrading to 1.452 and 1.455 and both behave as described above with all jobs remaining in 'Waiting for next executor' state with all nodes free. Downgrading to 1.451 restores functionality.
          Hide
          allenservedio allenservedio added a comment -

          To work around this, I have removed the quiet period from all of my jobs (running on 1.455 now). This will mean we will get out of order builds sometimes as it will not necessarily see changes from dependent repos - but we can live with that for a bit until this is fixed.

          Show
          allenservedio allenservedio added a comment - To work around this, I have removed the quiet period from all of my jobs (running on 1.455 now). This will mean we will get out of order builds sometimes as it will not necessarily see changes from dependent repos - but we can live with that for a bit until this is fixed.
          Hide
          dtka Daniel Tkatch added a comment -
          Show
          dtka Daniel Tkatch added a comment - I might have the same problem I described here: http://groups.google.com/group/jenkinsci-users/browse_thread/thread/df72335ffe030639
          Hide
          wbauer wbauer added a comment -

          I am seeing the same issue with "Waiting for next available executor" when several machines are actually completely free. Issues happens since 1.452.

          Show
          wbauer wbauer added a comment - I am seeing the same issue with "Waiting for next available executor" when several machines are actually completely free. Issues happens since 1.452.
          Hide
          devtools DevTools Active added a comment -

          This started for us with our upgrade to 1.455 and continues with 1.456

          Show
          devtools DevTools Active added a comment - This started for us with our upgrade to 1.455 and continues with 1.456
          Hide
          busybyte Shawn Garner added a comment -

          I've noted this issue on 1.452.
          We have job a with a quiet period and three idle executors.
          Then job b comes with no wait period.
          Job b waits until job a is done.
          I've also noticed that job b even when it is on the front of the queue that it seems to wait even though all three executors are free.
          It's almost as if it thinks job b has a quiet period too.
          They eventually all go through but just not quite the throughput you'd expect with three idle executors.
          I've noticed as many as 4 or 5 jobs backed up with all three executors idle.

          Show
          busybyte Shawn Garner added a comment - I've noted this issue on 1.452. We have job a with a quiet period and three idle executors. Then job b comes with no wait period. Job b waits until job a is done. I've also noticed that job b even when it is on the front of the queue that it seems to wait even though all three executors are free. It's almost as if it thinks job b has a quiet period too. They eventually all go through but just not quite the throughput you'd expect with three idle executors. I've noticed as many as 4 or 5 jobs backed up with all three executors idle.
          Hide
          raspy Krzysztof Malinowski added a comment -

          Same issue here. All builds are blocked until all quiet periods end. This is a serious disadvantage and I would like to vote for fixing this in the first place.

          Show
          raspy Krzysztof Malinowski added a comment - Same issue here. All builds are blocked until all quiet periods end. This is a serious disadvantage and I would like to vote for fixing this in the first place.
          Hide
          weakcamel Waldek M added a comment -

          Same here. In case of many scheduled builds, this may cause them never to happen.

          Show
          weakcamel Waldek M added a comment - Same here. In case of many scheduled builds, this may cause them never to happen.
          Hide
          mwebber mwebber added a comment -

          Bumping priority from Major to Critical, as this problem is adversely affecting a lot of people.

          Show
          mwebber mwebber added a comment - Bumping priority from Major to Critical, as this problem is adversely affecting a lot of people.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          changelog.html
          core/src/main/java/hudson/model/Queue.java
          http://jenkins-ci.org/commit/jenkins/e4abef20fbdb71971085e7507d7fff4b92676368
          Log:
          Merge pull request #419 from Vlatombe/jenkins-12994

          [FIX JENKINS-12994] Quiet period is blocking other jobs in queue

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: changelog.html core/src/main/java/hudson/model/Queue.java http://jenkins-ci.org/commit/jenkins/e4abef20fbdb71971085e7507d7fff4b92676368 Log: Merge pull request #419 from Vlatombe/jenkins-12994 [FIX JENKINS-12994] Quiet period is blocking other jobs in queue
          Hide
          dogfood dogfood added a comment -

          Integrated in jenkins_main_trunk #1623
          [FIX JENKINS-12994] Quiet period is blocking other jobs in queue (Revision 394e9d6c0488fae6834d97a158a018abb31f3179)
          Update changelog for JENKINS-12994 (Revision 07b3f2cccb077df85617f2748f9b329528bc263b)

          Result = SUCCESS
          Vincent Latombe : 394e9d6c0488fae6834d97a158a018abb31f3179
          Files :

          • core/src/main/java/hudson/model/Queue.java

          Vincent Latombe : 07b3f2cccb077df85617f2748f9b329528bc263b
          Files :

          • changelog.html
          Show
          dogfood dogfood added a comment - Integrated in jenkins_main_trunk #1623 [FIX JENKINS-12994] Quiet period is blocking other jobs in queue (Revision 394e9d6c0488fae6834d97a158a018abb31f3179) Update changelog for JENKINS-12994 (Revision 07b3f2cccb077df85617f2748f9b329528bc263b) Result = SUCCESS Vincent Latombe : 394e9d6c0488fae6834d97a158a018abb31f3179 Files : core/src/main/java/hudson/model/Queue.java Vincent Latombe : 07b3f2cccb077df85617f2748f9b329528bc263b Files : changelog.html
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Vincent Latombe
          Path:
          core/src/main/java/hudson/model/Queue.java
          http://jenkins-ci.org/commit/jenkins/394e9d6c0488fae6834d97a158a018abb31f3179
          Log:
          [FIX JENKINS-12994] Quiet period is blocking other jobs in queue
          Queue#maintain() was returning too soon

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vincent Latombe Path: core/src/main/java/hudson/model/Queue.java http://jenkins-ci.org/commit/jenkins/394e9d6c0488fae6834d97a158a018abb31f3179 Log: [FIX JENKINS-12994] Quiet period is blocking other jobs in queue Queue#maintain() was returning too soon
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Vincent Latombe
          Path:
          changelog.html
          http://jenkins-ci.org/commit/jenkins/07b3f2cccb077df85617f2748f9b329528bc263b
          Log:
          Update changelog for JENKINS-12994

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vincent Latombe Path: changelog.html http://jenkins-ci.org/commit/jenkins/07b3f2cccb077df85617f2748f9b329528bc263b Log: Update changelog for JENKINS-12994
          Hide
          dogfood dogfood added a comment -

          Integrated in jenkins_ui-changes_branch #21
          [FIX JENKINS-12994] Quiet period is blocking other jobs in queue (Revision 394e9d6c0488fae6834d97a158a018abb31f3179)
          Update changelog for JENKINS-12994 (Revision 07b3f2cccb077df85617f2748f9b329528bc263b)

          Result = SUCCESS
          Vincent Latombe : 394e9d6c0488fae6834d97a158a018abb31f3179
          Files :

          • core/src/main/java/hudson/model/Queue.java

          Vincent Latombe : 07b3f2cccb077df85617f2748f9b329528bc263b
          Files :

          • changelog.html
          Show
          dogfood dogfood added a comment - Integrated in jenkins_ui-changes_branch #21 [FIX JENKINS-12994] Quiet period is blocking other jobs in queue (Revision 394e9d6c0488fae6834d97a158a018abb31f3179) Update changelog for JENKINS-12994 (Revision 07b3f2cccb077df85617f2748f9b329528bc263b) Result = SUCCESS Vincent Latombe : 394e9d6c0488fae6834d97a158a018abb31f3179 Files : core/src/main/java/hudson/model/Queue.java Vincent Latombe : 07b3f2cccb077df85617f2748f9b329528bc263b Files : changelog.html
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Vincent Latombe
          Path:
          core/src/main/java/hudson/model/Queue.java
          http://jenkins-ci.org/commit/jenkins/210e50fc8547c99d66b033787371dae115f03b7e
          Log:
          [FIX JENKINS-12994] Quiet period is blocking other jobs in queue
          Queue#maintain() was returning too soon(cherry picked from commit 394e9d6c0488fae6834d97a158a018abb31f3179)

          Conflicts:

          core/src/main/java/hudson/model/Queue.java

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vincent Latombe Path: core/src/main/java/hudson/model/Queue.java http://jenkins-ci.org/commit/jenkins/210e50fc8547c99d66b033787371dae115f03b7e Log: [FIX JENKINS-12994] Quiet period is blocking other jobs in queue Queue#maintain() was returning too soon(cherry picked from commit 394e9d6c0488fae6834d97a158a018abb31f3179) Conflicts: core/src/main/java/hudson/model/Queue.java
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Vincent Latombe
          Path:
          changelog.html
          http://jenkins-ci.org/commit/jenkins/a86887043605c6070c81894bf38e4c6084e1f229
          Log:
          Update changelog for JENKINS-12994(cherry picked from commit 07b3f2cccb077df85617f2748f9b329528bc263b)

          Conflicts:

          changelog.html

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vincent Latombe Path: changelog.html http://jenkins-ci.org/commit/jenkins/a86887043605c6070c81894bf38e4c6084e1f229 Log: Update changelog for JENKINS-12994 (cherry picked from commit 07b3f2cccb077df85617f2748f9b329528bc263b) Conflicts: changelog.html
          Hide
          bqe_antonio_rojas Antonio Rojas added a comment -

          Was it fixed and how to apply the fix if that is the case ? thanks for your help

          Show
          bqe_antonio_rojas Antonio Rojas added a comment - Was it fixed and how to apply the fix if that is the case ? thanks for your help
          Hide
          danielbeck Daniel Beck added a comment -

          Antonio Rojas: This was fixed years ago. It's really time to update your Jenkins if you're still affected.

          However if you think this happens in a recent Jenkins version, it's almost guaranteed an entirely different issue (e.g. JENKINS-27871) and has nothing to do with this one.

          Show
          danielbeck Daniel Beck added a comment - Antonio Rojas : This was fixed years ago. It's really time to update your Jenkins if you're still affected. However if you think this happens in a recent Jenkins version, it's almost guaranteed an entirely different issue (e.g. JENKINS-27871 ) and has nothing to do with this one.

            People

            • Assignee:
              Unassigned
              Reporter:
              teetoivo Teemu Toivola
            • Votes:
              32 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: