Since version 1.4x we've had a problem where threads continue to keep climbing without releasing. We upped the default file handlers to 8192 on the machine but we reach that limit as well. I've provided a JavaMelody graph as well as a current thread list.

It appears that we keep spawning Thread-pools but I'm not sure how to debug /why/ we keep spawning thread pools and why they don't clean up. Nearly all of the spawned threads look as such:

"pool-1451-thread-1" prio=5 WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:679)".

It's very little to go on. It appears they aren't doing anything (see attached for the full list). Could use some help with triage/debugging this problem. It's really starting to kill us with having to run monitors on thre thread pool and restart jenkins before they overrun.