-
Bug
-
Resolution: Unresolved
-
Major
-
None
Given the following build flow job:
// main_job
parallel(
build_list.collect { a_build_name -> return {
guard{
run_build = build(a_build_name)
} rescue {
// rescue builds
}//rescue
}// return
}// collect
)//parallel
Aborting "main_job" will abort all running/spawned children and "cleanup" queued jobs from the UI which is good. Problem that happens is that the queued jobs also had threads created for them and added to a pool. The threads
were spawned asynchronously using a future object which calls get() waiting for the computation to complete. In this case it never does, so the threads wait indefinitely
Notes:
1. To view the threads use .../jenkins/monitoring url
2. Queue occurs if the number of machines available to run the jobs is less than the number of jobs spawned in parallel
3. Name of the threads are also cryptic, ex. "pool-X-thread-Y"
4. The only way I can clear up these threads is to either restart Jenkins or run the following script:
Thread.getAllStackTraces().keySet().each(){ item ->
if(item.getName().contains("pool-")){
println "Interrupting thread " + item.getId() + " " + item.getName();
item.interrupt()
}
}
- Note due to issue #3 this script is useless as I may end up deleting threads spawned by a different build entirely which may not be a desired effect
5. Due to large number of concurrent build flows in our setup, these threads pile up taking up resources until eventually Jenkins is not accessible
It seems that when a build flow job is aborted, future objects for "scheduled/queued jobs" wait indefinitely taking up resources depending on Jenkins setup
Sample thread dump:
"pool-16-thread-1" prio=5 WAITING java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:73) hudson.model.queue.FutureImpl.waitForStart(FutureImpl.java:68) hudson.model.queue.QueueTaskFuture$waitForStart.call(Unknown Source) com.cloudbees.plugins.flow.JobInvocation.waitForStart(JobInvocation.groovy:234) com.cloudbees.plugins.flow.JobInvocation$waitForStart$0.call(Unknown Source) com.cloudbees.plugins.flow.FlowDelegate.build(FlowDSL.groovy:217) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079) groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903) groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:723) com.cloudbees.plugins.flow.FlowDelegate.invokeMethod(FlowDSL.groovy) hudson.util.spring.ClosureScript.invokeMethod(ClosureScript.java:83) org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:407) org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:346) groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903) groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:723) groovy.lang.GroovyObjectSupport.invokeMethod(GroovyObjectSupport.java:44) org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:407) org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:346) groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903) org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46) org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145) Script1$_run_closure2_closure5.doCall(Script1.groovy:139) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272) groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903) org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141) Script1$_run_closure2_closure5.doCall(Script1.groovy) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272) groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903) org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39) org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112) com.cloudbees.plugins.flow.FlowDelegate$_parallel_closure5_closure7.doCall(FlowDSL.groovy:427) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272) groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903) org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46) org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141) com.cloudbees.plugins.flow.FlowDelegate$_parallel_closure5_closure7.doCall(FlowDSL.groovy) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272) groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903) groovy.lang.Closure.call(Closure.java:415) groovy.lang.Closure.call(Closure.java:409) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:744)
