Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21044

Throttle Concurrent Builds blocking Jenkins queue

XMLWordPrintable

      Jenkins had stopped responding to browser requests for Jenkins pages and I think it may be caused by the recent upgrade to Throttle Concurrent Builds 1.8.1

      Requests were getting blocked waiting on 0x00000004181e4520

      "Handling GET /jenkins/ : RequestHandlerThread[#171]" daemon prio=10 tid=0x00000000168ee800 nid=0x193b waiting for monitor entry [0x000000004335b000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at hudson.model.Queue.getItems(Queue.java:687)
	- waiting to lock <0x00000004181e4520> (a hudson.model.Queue)
	at hudson.model.Queue$CachedItemList.get(Queue.java:216)
	at hudson.model.Queue.getApproximateItemsQuickly(Queue.java:717)
	at hudson.model.View.getApproximateQueueItemsQuickly(View.java:483)
	at sun.reflect.GeneratedMethodAccessor355.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

      This seems to be waiting on Throttle Concurrent Builds code.
      Further dumps after 10 minutes, 20 minutes and 30 minutes showed this same stack trace.

      "Thread-126" daemon prio=10 tid=0x00002aaae0529800 nid=0x1785 runnable [0x0000000046590000]
   java.lang.Thread.State: RUNNABLE
	at java.util.WeakHashMap$HashIterator.hasNext(WeakHashMap.java:875)
	at java.util.AbstractCollection.toArray(AbstractCollection.java:139)
	at java.util.ArrayList.<init>(ArrayList.java:164)
	at hudson.plugins.throttleconcurrents.ThrottleJobProperty.getCategoryProjects(ThrottleJobProperty.java:141)
	- locked <0x000000041a79b778> (a java.util.HashMap)
	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:118)
	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:90)
	at hudson.model.Queue.isBuildBlocked(Queue.java:937)
	at hudson.model.Queue.maintain(Queue.java:1006)
	- locked <0x00000004181e4520> (a hudson.model.Queue)
	at hudson.model.Queue$1.call(Queue.java:303)
	at hudson.model.Queue$1.call(Queue.java:300)
	at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:69)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:104)
	at java.lang.Thread.run(Thread.java:724)

   Locked ownable synchronizers:
	- None

      CPU usage was at ~100% for this thread for the 30 minutes that I was watching it before I restarted Jenkins.
      (6021 = 0x1785)

        PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                   
 6021 rcbuild_  35  10 18.0g 6.2g  32m R 99.7 19.7  30:45.97 java

      I have rolled back to Throttle Concurrent Builds 1.8 for now.

      Still learning how to investigate thread dumps but please let me know if there is anything I can do to help.

        1. throttle-concurrents.hpi
          26 kB

            oleg_nenashev Oleg Nenashev
            gcummings Geoff Cummings
            Votes:
            9 Vote for this issue
            Watchers:
            18 Start watching this issue

              Created:
              Updated:
              Resolved: