Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21044

Throttle Concurrent Builds blocking Jenkins queue

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Jenkins had stopped responding to browser requests for Jenkins pages and I think it may be caused by the recent upgrade to Throttle Concurrent Builds 1.8.1

      Requests were getting blocked waiting on 0x00000004181e4520

      "Handling GET /jenkins/ : RequestHandlerThread[#171]" daemon prio=10 tid=0x00000000168ee800 nid=0x193b waiting for monitor entry [0x000000004335b000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at hudson.model.Queue.getItems(Queue.java:687)
      	- waiting to lock <0x00000004181e4520> (a hudson.model.Queue)
      	at hudson.model.Queue$CachedItemList.get(Queue.java:216)
      	at hudson.model.Queue.getApproximateItemsQuickly(Queue.java:717)
      	at hudson.model.View.getApproximateQueueItemsQuickly(View.java:483)
      	at sun.reflect.GeneratedMethodAccessor355.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      

      This seems to be waiting on Throttle Concurrent Builds code.
      Further dumps after 10 minutes, 20 minutes and 30 minutes showed this same stack trace.

      "Thread-126" daemon prio=10 tid=0x00002aaae0529800 nid=0x1785 runnable [0x0000000046590000]
         java.lang.Thread.State: RUNNABLE
      	at java.util.WeakHashMap$HashIterator.hasNext(WeakHashMap.java:875)
      	at java.util.AbstractCollection.toArray(AbstractCollection.java:139)
      	at java.util.ArrayList.<init>(ArrayList.java:164)
      	at hudson.plugins.throttleconcurrents.ThrottleJobProperty.getCategoryProjects(ThrottleJobProperty.java:141)
      	- locked <0x000000041a79b778> (a java.util.HashMap)
      	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:118)
      	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:90)
      	at hudson.model.Queue.isBuildBlocked(Queue.java:937)
      	at hudson.model.Queue.maintain(Queue.java:1006)
      	- locked <0x00000004181e4520> (a hudson.model.Queue)
      	at hudson.model.Queue$1.call(Queue.java:303)
      	at hudson.model.Queue$1.call(Queue.java:300)
      	at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:69)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:104)
      	at java.lang.Thread.run(Thread.java:724)
      
         Locked ownable synchronizers:
      	- None
      

      CPU usage was at ~100% for this thread for the 30 minutes that I was watching it before I restarted Jenkins.
      (6021 = 0x1785)

        PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                   
       6021 rcbuild_  35  10 18.0g 6.2g  32m R 99.7 19.7  30:45.97 java     
      

      I have rolled back to Throttle Concurrent Builds 1.8 for now.

      Still learning how to investigate thread dumps but please let me know if there is anything I can do to help.

        Attachments

          Issue Links

            Activity

            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            The new version works for me as well.

            @abayer, do you confirm the merge?

            Show
            oleg_nenashev Oleg Nenashev added a comment - The new version works for me as well. @abayer, do you confirm the merge?
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/9b7562d4b08e0a4202130d43307082553142df82
            Log:
            [FIXED JENKINS-21044] - Throttling blocks the Jenkins queue

            Seems the issue was in improper usage of WeakHashMap (see analysis from @centic).
            I've managed to reproduce the behavior in the following case:

            • There is a big number of jobs/configurations with throttling
            • The builds queue is not empty
              // Seems that ThrottleQueueTaskDispatcher tries to access the data before the complete loading of the plugin's configuration.

            This fix provides an explicit locking of any load operations + manual cleanup of erroneous cache data, which goes to persistence in 1.8.1
            Resolves https://issues.jenkins-ci.org/browse/JENKINS-21044

            Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/9b7562d4b08e0a4202130d43307082553142df82 Log: [FIXED JENKINS-21044] - Throttling blocks the Jenkins queue Seems the issue was in improper usage of WeakHashMap (see analysis from @centic). I've managed to reproduce the behavior in the following case: There is a big number of jobs/configurations with throttling The builds queue is not empty // Seems that ThrottleQueueTaskDispatcher tries to access the data before the complete loading of the plugin's configuration. This fix provides an explicit locking of any load operations + manual cleanup of erroneous cache data, which goes to persistence in 1.8.1 Resolves https://issues.jenkins-ci.org/browse/JENKINS-21044 Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/c453516716079248d74ce588efc0293669e6e1a7
            Log:
            JENKINS-21044 - Don't create a new HashMap after the load operation

            Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/c453516716079248d74ce588efc0293669e6e1a7 Log: JENKINS-21044 - Don't create a new HashMap after the load operation Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/70107b4222502935a9e46beffa31daae2e99e50b
            Log:
            Merge pull request #13 from synopsys-arc-oss/JENKINS_21044_fix

            [FIXED JENKINS-21044] - Throttling blocks the Jenkins queue

            Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/dc16282a90b7...70107b422250

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/70107b4222502935a9e46beffa31daae2e99e50b Log: Merge pull request #13 from synopsys-arc-oss/JENKINS_21044_fix [FIXED JENKINS-21044] - Throttling blocks the Jenkins queue Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/dc16282a90b7...70107b422250
            Hide
            abayer Andrew Bayer added a comment -

            Thanks, Oleg!

            Show
            abayer Andrew Bayer added a comment - Thanks, Oleg!

              People

              • Assignee:
                oleg_nenashev Oleg Nenashev
                Reporter:
                gcummings Geoff Cummings
              • Votes:
                9 Vote for this issue
                Watchers:
                18 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: