Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44747

Queue locked indefinitely in ThrottleQueueTaskDispatcher.categoriesForPipeline

    Details

    • Similar Issues:

      Description

      See this thread dump. This call can block (for example while a Pipeline build is getting ready to resume), whereas this caller ought to return quickly (milliseconds, if at all possible).

      Observed in a Jenkins server after restarting with some running builds. Since all Queue calls hanged, the system seemed to be frozen—no branch indexing, etc. Even taking a fresh support bundle hanged for the same reason (probably support-core ought to interrupt component threads which do not finish within some fixed timeout). Cannot cancel running builds for the same reason. The only way to recover was to disable the TCB plugin.

      Have also heard a field report of a server running TCB 2.0 exhibiting similar symptoms, but have not yet confirmed duplicate, pending existence of a thread dump.

        Attachments

          Issue Links

            Activity

            jglick Jesse Glick created issue -
            Hide
            abayer Andrew Bayer added a comment -

            Is there any way to determine whether the call for the FlowNode could block ahead of time?

            Show
            abayer Andrew Bayer added a comment - Is there any way to determine whether the call for the FlowNode could block ahead of time?
            Hide
            jglick Jesse Glick added a comment -

            Filed PR 50. I would consider it more of a hot-patch to avoid the deadlock than a real fix—presumably the throttling would be ineffective if the timeout were in fact hit, which could cause other problems, though surely less severe.

            Show
            jglick Jesse Glick added a comment - Filed PR 50. I would consider it more of a hot-patch to avoid the deadlock than a real fix—presumably the throttling would be ineffective if the timeout were in fact hit, which could cause other problems, though surely less severe.
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Remote Link This issue links to "PR 50 (Web Link)" [ 16966 ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Assignee Andrew Bayer [ abayer ] Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/a4cb745f7f616d5ffe3a212bb45f5c8612388157
            Log:
            JENKINS-44747 Limit the amount of time categoriesForPipeline can run.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/a4cb745f7f616d5ffe3a212bb45f5c8612388157 Log: JENKINS-44747 Limit the amount of time categoriesForPipeline can run.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: README.md http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/c22a7e781ac0e393691f9b03b2221b209ab9abfe Log: Adding disclaimer for JENKINS-44747 change. Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/a4cb745f7f61 ^...c22a7e781ac0
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-44756 [ JENKINS-44756 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            pom.xml
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/d105bb791364f3d8bfe42dcad448b14f9b0ed8b7
            Log:
            Merge pull request #50 from jglick/Queue-JENKINS-44747

            JENKINS-44747 Limit the amount of time categoriesForPipeline can run

            Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/7c61f6119c67...d105bb791364

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pom.xml src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/d105bb791364f3d8bfe42dcad448b14f9b0ed8b7 Log: Merge pull request #50 from jglick/Queue- JENKINS-44747 JENKINS-44747 Limit the amount of time categoriesForPipeline can run Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/7c61f6119c67...d105bb791364
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            README.md
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/fc9f6db53d21ebf50447e20a6304323ce62f6a46
            Log:
            Merge pull request #51 from jenkinsci/jenkins-44747-disclaimer

            Disclaimer for JENKINS-44747 change.

            Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/d105bb791364...fc9f6db53d21

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: README.md http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/fc9f6db53d21ebf50447e20a6304323ce62f6a46 Log: Merge pull request #51 from jenkinsci/jenkins-44747-disclaimer Disclaimer for JENKINS-44747 change. Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/d105bb791364...fc9f6db53d21
            Hide
            abayer Andrew Bayer added a comment -

            Releasing as 2.0.1 as we speak.

            Show
            abayer Andrew Bayer added a comment - Releasing as 2.0.1 as we speak.
            abayer Andrew Bayer made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            CHANGELOG.md
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/080f4768bfd31d55c717f904c223bfb6119a89fe
            Log:
            [JENKINS-44756, JENKINS-44747] - 2.0.1 changelog + Errata notice

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: CHANGELOG.md http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/080f4768bfd31d55c717f904c223bfb6119a89fe Log: [JENKINS-44756, JENKINS-44747] - 2.0.1 changelog + Errata notice
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            CHANGELOG.md
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/d58cb2873a58ed59323b033aecd3c2536c569ad3
            Log:
            Merge pull request #52 from jenkinsci/changelog/2.0.1

            [JENKINS-44756, JENKINS-44747] - 2.0.1 changelog + Errata notice

            Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/cfc775b523fe...d58cb2873a58

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: CHANGELOG.md http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/d58cb2873a58ed59323b033aecd3c2536c569ad3 Log: Merge pull request #52 from jenkinsci/changelog/2.0.1 [JENKINS-44756, JENKINS-44747] - 2.0.1 changelog + Errata notice Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/cfc775b523fe...d58cb2873a58
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal OSS-2306 (Web Link)" [ 18350 ]

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                jglick Jesse Glick
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: