Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12994

Quiet period is blocking other jobs in queue

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Environment:
      Platform: All, OS: All
    • Similar Issues:

      Description

      Starting from version 1.452, a job in queue waiting for the quiet period to pass is blocking all other jobs that have been queued after it.

      Example:

      1. Job A has quiet period set to 10 seconds
      2. Job B has quiet period set to 300 seconds
      3. Job C has quiet period set to 100 seconds
      4. Jobs get queued in A-B-C order.
      5. Job A starts executing after 10 seconds
      6. After 100 seconds, status of job C changes to "pending - Waiting for next available executor" even if free executors are available
      7. After 300 seconds both jobs B and C start executing

      This problem is hardly visible when short quiet periods are used but becomes problematic with longer quiet periods or plugins like Naginator that will increase the quiet period to hours if the builds fail enough.

      Version 1.451 is the last release where this problem isn't visible. From public releases, at least 1.452 and 1.454 are affected.

        Attachments

          Issue Links

            Activity

            Hide
            mwebber mwebber added a comment -

            I'm seeing this, or something like it, as well.

            I have also recently installed the "Throttle Concurrent Builds Plugin" and the "Exclusion-Plugin", so initially I though it might be related to that, but this ticket suggests it's a core problem.

            Show
            mwebber mwebber added a comment - I'm seeing this, or something like it, as well. I have also recently installed the "Throttle Concurrent Builds Plugin" and the "Exclusion-Plugin", so initially I though it might be related to that, but this ticket suggests it's a core problem.
            Hide
            allenservedio allenservedio added a comment -

            I am seeing the same thing in 1.454. Even worse, I am starting to see deadlocks around this. I had to go in and kill a three jobs where two of them were stuck in an infinite loop of quiet periods and the third (a scheduled job) was stuck waiting for an executor. As I was not watching the builds, they were stuck in that state for over an hour before I caught them (the quiet period for the other two jobs was 190 seconds, it just kept cycling through quiet periods).

            Show
            allenservedio allenservedio added a comment - I am seeing the same thing in 1.454. Even worse, I am starting to see deadlocks around this. I had to go in and kill a three jobs where two of them were stuck in an infinite loop of quiet periods and the third (a scheduled job) was stuck waiting for an executor. As I was not watching the builds, they were stuck in that state for over an hour before I caught them (the quiet period for the other two jobs was 190 seconds, it just kept cycling through quiet periods).
            Hide
            bruce Dirk Kuypers added a comment -

            I am seeing this, too, since 1.452 I think. Maybe this is a hint (or just coincidence?): I never had to kill the pending jobs, I went into "Configure Jenkins->Configure System" and just pressed back, without changing/saving anything. After that the queues got filled again. Maybe others could try this, too, to confirm if this helps.

            Show
            bruce Dirk Kuypers added a comment - I am seeing this, too, since 1.452 I think. Maybe this is a hint (or just coincidence?): I never had to kill the pending jobs, I went into "Configure Jenkins->Configure System" and just pressed back, without changing/saving anything. After that the queues got filled again. Maybe others could try this, too, to confirm if this helps.
            Hide
            sti Sami Tikka added a comment -

            I upgraded from 1.451 to 1.453 and all my builds started staying in the queue forever. Exactly same symptom: Build says it is waiting for available executor even though all slaves were online with free executors. Restarting Jenkins did not help. Downgraded back to 1.451 and the problem went away.

            Show
            sti Sami Tikka added a comment - I upgraded from 1.451 to 1.453 and all my builds started staying in the queue forever. Exactly same symptom: Build says it is waiting for available executor even though all slaves were online with free executors. Restarting Jenkins did not help. Downgraded back to 1.451 and the problem went away.
            Hide
            vlatombe Vincent Latombe added a comment - - edited

            I've dug in the history between 1.451 and 1.452, and it seems the issue is coming from the following commit : https://github.com/jenkinsci/jenkins/commit/1ac4872ecdc500c63c0dbfce5e4beb5ce78369e5

            You can see at line 920-921 that the method returns if the first element of the waiting list is not ready to build, so it doesn't consider other elements in the waiting list.
            waitingList is a TreeSet<WaitingItem> and it looks like it is not properly sorted by timestamp, meaning that the compareTo method is not properly implemented.

            Show
            vlatombe Vincent Latombe added a comment - - edited I've dug in the history between 1.451 and 1.452, and it seems the issue is coming from the following commit : https://github.com/jenkinsci/jenkins/commit/1ac4872ecdc500c63c0dbfce5e4beb5ce78369e5 You can see at line 920-921 that the method returns if the first element of the waiting list is not ready to build, so it doesn't consider other elements in the waiting list. waitingList is a TreeSet<WaitingItem> and it looks like it is not properly sorted by timestamp, meaning that the compareTo method is not properly implemented.
            Hide
            cnogradi Chris Nogradi added a comment -

            I tried upgrading to 1.452 and 1.455 and both behave as described above with all jobs remaining in 'Waiting for next executor' state with all nodes free. Downgrading to 1.451 restores functionality.

            Show
            cnogradi Chris Nogradi added a comment - I tried upgrading to 1.452 and 1.455 and both behave as described above with all jobs remaining in 'Waiting for next executor' state with all nodes free. Downgrading to 1.451 restores functionality.
            Hide
            allenservedio allenservedio added a comment -

            To work around this, I have removed the quiet period from all of my jobs (running on 1.455 now). This will mean we will get out of order builds sometimes as it will not necessarily see changes from dependent repos - but we can live with that for a bit until this is fixed.

            Show
            allenservedio allenservedio added a comment - To work around this, I have removed the quiet period from all of my jobs (running on 1.455 now). This will mean we will get out of order builds sometimes as it will not necessarily see changes from dependent repos - but we can live with that for a bit until this is fixed.
            Hide
            dtka Daniel Tkatch added a comment -
            Show
            dtka Daniel Tkatch added a comment - I might have the same problem I described here: http://groups.google.com/group/jenkinsci-users/browse_thread/thread/df72335ffe030639
            Hide
            wbauer wbauer added a comment -

            I am seeing the same issue with "Waiting for next available executor" when several machines are actually completely free. Issues happens since 1.452.

            Show
            wbauer wbauer added a comment - I am seeing the same issue with "Waiting for next available executor" when several machines are actually completely free. Issues happens since 1.452.
            Hide
            devtools DevTools Active added a comment -

            This started for us with our upgrade to 1.455 and continues with 1.456

            Show
            devtools DevTools Active added a comment - This started for us with our upgrade to 1.455 and continues with 1.456
            Hide
            busybyte Shawn Garner added a comment -

            I've noted this issue on 1.452.
            We have job a with a quiet period and three idle executors.
            Then job b comes with no wait period.
            Job b waits until job a is done.
            I've also noticed that job b even when it is on the front of the queue that it seems to wait even though all three executors are free.
            It's almost as if it thinks job b has a quiet period too.
            They eventually all go through but just not quite the throughput you'd expect with three idle executors.
            I've noticed as many as 4 or 5 jobs backed up with all three executors idle.

            Show
            busybyte Shawn Garner added a comment - I've noted this issue on 1.452. We have job a with a quiet period and three idle executors. Then job b comes with no wait period. Job b waits until job a is done. I've also noticed that job b even when it is on the front of the queue that it seems to wait even though all three executors are free. It's almost as if it thinks job b has a quiet period too. They eventually all go through but just not quite the throughput you'd expect with three idle executors. I've noticed as many as 4 or 5 jobs backed up with all three executors idle.
            Hide
            raspy Krzysztof Malinowski added a comment -

            Same issue here. All builds are blocked until all quiet periods end. This is a serious disadvantage and I would like to vote for fixing this in the first place.

            Show
            raspy Krzysztof Malinowski added a comment - Same issue here. All builds are blocked until all quiet periods end. This is a serious disadvantage and I would like to vote for fixing this in the first place.
            Hide
            weakcamel Waldek M added a comment -

            Same here. In case of many scheduled builds, this may cause them never to happen.

            Show
            weakcamel Waldek M added a comment - Same here. In case of many scheduled builds, this may cause them never to happen.
            Hide
            mwebber mwebber added a comment -

            Bumping priority from Major to Critical, as this problem is adversely affecting a lot of people.

            Show
            mwebber mwebber added a comment - Bumping priority from Major to Critical, as this problem is adversely affecting a lot of people.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            changelog.html
            core/src/main/java/hudson/model/Queue.java
            http://jenkins-ci.org/commit/jenkins/e4abef20fbdb71971085e7507d7fff4b92676368
            Log:
            Merge pull request #419 from Vlatombe/jenkins-12994

            [FIX JENKINS-12994] Quiet period is blocking other jobs in queue

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: changelog.html core/src/main/java/hudson/model/Queue.java http://jenkins-ci.org/commit/jenkins/e4abef20fbdb71971085e7507d7fff4b92676368 Log: Merge pull request #419 from Vlatombe/jenkins-12994 [FIX JENKINS-12994] Quiet period is blocking other jobs in queue
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #1623
            [FIX JENKINS-12994] Quiet period is blocking other jobs in queue (Revision 394e9d6c0488fae6834d97a158a018abb31f3179)
            Update changelog for JENKINS-12994 (Revision 07b3f2cccb077df85617f2748f9b329528bc263b)

            Result = SUCCESS
            Vincent Latombe : 394e9d6c0488fae6834d97a158a018abb31f3179
            Files :

            • core/src/main/java/hudson/model/Queue.java

            Vincent Latombe : 07b3f2cccb077df85617f2748f9b329528bc263b
            Files :

            • changelog.html
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #1623 [FIX JENKINS-12994] Quiet period is blocking other jobs in queue (Revision 394e9d6c0488fae6834d97a158a018abb31f3179) Update changelog for JENKINS-12994 (Revision 07b3f2cccb077df85617f2748f9b329528bc263b) Result = SUCCESS Vincent Latombe : 394e9d6c0488fae6834d97a158a018abb31f3179 Files : core/src/main/java/hudson/model/Queue.java Vincent Latombe : 07b3f2cccb077df85617f2748f9b329528bc263b Files : changelog.html
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Vincent Latombe
            Path:
            core/src/main/java/hudson/model/Queue.java
            http://jenkins-ci.org/commit/jenkins/394e9d6c0488fae6834d97a158a018abb31f3179
            Log:
            [FIX JENKINS-12994] Quiet period is blocking other jobs in queue
            Queue#maintain() was returning too soon

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vincent Latombe Path: core/src/main/java/hudson/model/Queue.java http://jenkins-ci.org/commit/jenkins/394e9d6c0488fae6834d97a158a018abb31f3179 Log: [FIX JENKINS-12994] Quiet period is blocking other jobs in queue Queue#maintain() was returning too soon
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Vincent Latombe
            Path:
            changelog.html
            http://jenkins-ci.org/commit/jenkins/07b3f2cccb077df85617f2748f9b329528bc263b
            Log:
            Update changelog for JENKINS-12994

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vincent Latombe Path: changelog.html http://jenkins-ci.org/commit/jenkins/07b3f2cccb077df85617f2748f9b329528bc263b Log: Update changelog for JENKINS-12994
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_ui-changes_branch #21
            [FIX JENKINS-12994] Quiet period is blocking other jobs in queue (Revision 394e9d6c0488fae6834d97a158a018abb31f3179)
            Update changelog for JENKINS-12994 (Revision 07b3f2cccb077df85617f2748f9b329528bc263b)

            Result = SUCCESS
            Vincent Latombe : 394e9d6c0488fae6834d97a158a018abb31f3179
            Files :

            • core/src/main/java/hudson/model/Queue.java

            Vincent Latombe : 07b3f2cccb077df85617f2748f9b329528bc263b
            Files :

            • changelog.html
            Show
            dogfood dogfood added a comment - Integrated in jenkins_ui-changes_branch #21 [FIX JENKINS-12994] Quiet period is blocking other jobs in queue (Revision 394e9d6c0488fae6834d97a158a018abb31f3179) Update changelog for JENKINS-12994 (Revision 07b3f2cccb077df85617f2748f9b329528bc263b) Result = SUCCESS Vincent Latombe : 394e9d6c0488fae6834d97a158a018abb31f3179 Files : core/src/main/java/hudson/model/Queue.java Vincent Latombe : 07b3f2cccb077df85617f2748f9b329528bc263b Files : changelog.html
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Vincent Latombe
            Path:
            core/src/main/java/hudson/model/Queue.java
            http://jenkins-ci.org/commit/jenkins/210e50fc8547c99d66b033787371dae115f03b7e
            Log:
            [FIX JENKINS-12994] Quiet period is blocking other jobs in queue
            Queue#maintain() was returning too soon(cherry picked from commit 394e9d6c0488fae6834d97a158a018abb31f3179)

            Conflicts:

            core/src/main/java/hudson/model/Queue.java

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vincent Latombe Path: core/src/main/java/hudson/model/Queue.java http://jenkins-ci.org/commit/jenkins/210e50fc8547c99d66b033787371dae115f03b7e Log: [FIX JENKINS-12994] Quiet period is blocking other jobs in queue Queue#maintain() was returning too soon(cherry picked from commit 394e9d6c0488fae6834d97a158a018abb31f3179) Conflicts: core/src/main/java/hudson/model/Queue.java
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Vincent Latombe
            Path:
            changelog.html
            http://jenkins-ci.org/commit/jenkins/a86887043605c6070c81894bf38e4c6084e1f229
            Log:
            Update changelog for JENKINS-12994(cherry picked from commit 07b3f2cccb077df85617f2748f9b329528bc263b)

            Conflicts:

            changelog.html

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vincent Latombe Path: changelog.html http://jenkins-ci.org/commit/jenkins/a86887043605c6070c81894bf38e4c6084e1f229 Log: Update changelog for JENKINS-12994 (cherry picked from commit 07b3f2cccb077df85617f2748f9b329528bc263b) Conflicts: changelog.html
            Hide
            bqe_antonio_rojas Antonio Rojas added a comment -

            Was it fixed and how to apply the fix if that is the case ? thanks for your help

            Show
            bqe_antonio_rojas Antonio Rojas added a comment - Was it fixed and how to apply the fix if that is the case ? thanks for your help
            Hide
            danielbeck Daniel Beck added a comment -

            Antonio Rojas: This was fixed years ago. It's really time to update your Jenkins if you're still affected.

            However if you think this happens in a recent Jenkins version, it's almost guaranteed an entirely different issue (e.g. JENKINS-27871) and has nothing to do with this one.

            Show
            danielbeck Daniel Beck added a comment - Antonio Rojas : This was fixed years ago. It's really time to update your Jenkins if you're still affected. However if you think this happens in a recent Jenkins version, it's almost guaranteed an entirely different issue (e.g. JENKINS-27871 ) and has nothing to do with this one.

              People

              • Assignee:
                Unassigned
                Reporter:
                teetoivo Teemu Toivola
              • Votes:
                32 Vote for this issue
                Watchers:
                25 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: