Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21044

Throttle Concurrent Builds blocking Jenkins queue

    Details

    • Similar Issues:

      Description

      Jenkins had stopped responding to browser requests for Jenkins pages and I think it may be caused by the recent upgrade to Throttle Concurrent Builds 1.8.1

      Requests were getting blocked waiting on 0x00000004181e4520

      "Handling GET /jenkins/ : RequestHandlerThread[#171]" daemon prio=10 tid=0x00000000168ee800 nid=0x193b waiting for monitor entry [0x000000004335b000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at hudson.model.Queue.getItems(Queue.java:687)
      	- waiting to lock <0x00000004181e4520> (a hudson.model.Queue)
      	at hudson.model.Queue$CachedItemList.get(Queue.java:216)
      	at hudson.model.Queue.getApproximateItemsQuickly(Queue.java:717)
      	at hudson.model.View.getApproximateQueueItemsQuickly(View.java:483)
      	at sun.reflect.GeneratedMethodAccessor355.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      

      This seems to be waiting on Throttle Concurrent Builds code.
      Further dumps after 10 minutes, 20 minutes and 30 minutes showed this same stack trace.

      "Thread-126" daemon prio=10 tid=0x00002aaae0529800 nid=0x1785 runnable [0x0000000046590000]
         java.lang.Thread.State: RUNNABLE
      	at java.util.WeakHashMap$HashIterator.hasNext(WeakHashMap.java:875)
      	at java.util.AbstractCollection.toArray(AbstractCollection.java:139)
      	at java.util.ArrayList.<init>(ArrayList.java:164)
      	at hudson.plugins.throttleconcurrents.ThrottleJobProperty.getCategoryProjects(ThrottleJobProperty.java:141)
      	- locked <0x000000041a79b778> (a java.util.HashMap)
      	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:118)
      	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:90)
      	at hudson.model.Queue.isBuildBlocked(Queue.java:937)
      	at hudson.model.Queue.maintain(Queue.java:1006)
      	- locked <0x00000004181e4520> (a hudson.model.Queue)
      	at hudson.model.Queue$1.call(Queue.java:303)
      	at hudson.model.Queue$1.call(Queue.java:300)
      	at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:69)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:104)
      	at java.lang.Thread.run(Thread.java:724)
      
         Locked ownable synchronizers:
      	- None
      

      CPU usage was at ~100% for this thread for the 30 minutes that I was watching it before I restarted Jenkins.
      (6021 = 0x1785)

        PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                   
       6021 rcbuild_  35  10 18.0g 6.2g  32m R 99.7 19.7  30:45.97 java     
      

      I have rolled back to Throttle Concurrent Builds 1.8 for now.

      Still learning how to investigate thread dumps but please let me know if there is anything I can do to help.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                oleg_nenashev Oleg Nenashev
                Reporter:
                gcummings Geoff Cummings
              • Votes:
                9 Vote for this issue
                Watchers:
                18 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: