Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25952

Build flow does not release threads for scheduled or queued jobs after an abort

    Details

    • Similar Issues:

      Description

      Given the following build flow job:

      // main_job

      parallel(
        build_list.collect { a_build_name -> return {
           guard{
              run_build = build(a_build_name) 
           } rescue {
              // rescue builds
           }//rescue
         }// return
        }// collect
      )//parallel
      

      Aborting "main_job" will abort all running/spawned children and "cleanup" queued jobs from the UI which is good. Problem that happens is that the queued jobs also had threads created for them and added to a pool. The threads
      were spawned asynchronously using a future object which calls get() waiting for the computation to complete. In this case it never does, so the threads wait indefinitely

      Notes:
      1. To view the threads use .../jenkins/monitoring url
      2. Queue occurs if the number of machines available to run the jobs is less than the number of jobs spawned in parallel
      3. Name of the threads are also cryptic, ex. "pool-X-thread-Y"
      4. The only way I can clear up these threads is to either restart Jenkins or run the following script:

      Thread.getAllStackTraces().keySet().each(){ item ->
        if(item.getName().contains("pool-")){ 
          println "Interrupting thread " + item.getId() + " " + item.getName(); 
          item.interrupt() 
        }
      }
      
      • Note due to issue #3 this script is useless as I may end up deleting threads spawned by a different build entirely which may not be a desired effect

      5. Due to large number of concurrent build flows in our setup, these threads pile up taking up resources until eventually Jenkins is not accessible

      It seems that when a build flow job is aborted, future objects for "scheduled/queued jobs" wait indefinitely taking up resources depending on Jenkins setup

      Sample thread dump:

      "pool-16-thread-1" prio=5 WAITING
      	java.lang.Object.wait(Native Method)
      	java.lang.Object.wait(Object.java:503)
      	hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:73)
      	hudson.model.queue.FutureImpl.waitForStart(FutureImpl.java:68)
      	hudson.model.queue.QueueTaskFuture$waitForStart.call(Unknown Source)
      	com.cloudbees.plugins.flow.JobInvocation.waitForStart(JobInvocation.groovy:234)
      	com.cloudbees.plugins.flow.JobInvocation$waitForStart$0.call(Unknown Source)
      	com.cloudbees.plugins.flow.FlowDelegate.build(FlowDSL.groovy:217)
      	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	java.lang.reflect.Method.invoke(Method.java:606)
      	org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      	groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      	groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
      	groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      	groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:723)
      	com.cloudbees.plugins.flow.FlowDelegate.invokeMethod(FlowDSL.groovy)
      	hudson.util.spring.ClosureScript.invokeMethod(ClosureScript.java:83)
      	org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:407)
      	org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:346)
      	groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      	groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:723)
      	groovy.lang.GroovyObjectSupport.invokeMethod(GroovyObjectSupport.java:44)
      	org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:407)
      	org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:346)
      	groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      	org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
      	org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
      	org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
      	org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
      	Script1$_run_closure2_closure5.doCall(Script1.groovy:139)
      	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	java.lang.reflect.Method.invoke(Method.java:606)
      	org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      	groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      	org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
      	groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      	org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
      	org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
      	Script1$_run_closure2_closure5.doCall(Script1.groovy)
      	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	java.lang.reflect.Method.invoke(Method.java:606)
      	org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      	groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      	org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
      	groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      	org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39)
      	org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
      	com.cloudbees.plugins.flow.FlowDelegate$_parallel_closure5_closure7.doCall(FlowDSL.groovy:427)
      	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	java.lang.reflect.Method.invoke(Method.java:606)
      	org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      	groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      	org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
      	groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      	org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
      	org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
      	org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
      	org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
      	com.cloudbees.plugins.flow.FlowDelegate$_parallel_closure5_closure7.doCall(FlowDSL.groovy)
      	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	java.lang.reflect.Method.invoke(Method.java:606)
      	org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      	groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      	org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
      	groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      	groovy.lang.Closure.call(Closure.java:415)
      	groovy.lang.Closure.call(Closure.java:409)
      	java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	java.lang.Thread.run(Thread.java:744)
      

        Attachments

          Activity

          Hide
          thomaste stefan thomasson added a comment -

          Could you check if this pull request ? It could be duplicate for JENKINS-20102

          https://github.com/jenkinsci/build-flow-plugin/pull/61

          Show
          thomaste stefan thomasson added a comment - Could you check if this pull request ? It could be duplicate for JENKINS-20102 https://github.com/jenkinsci/build-flow-plugin/pull/61
          Hide
          eric_stein_de Eric Stein added a comment -

          I think I am affected by this. I have zero builds currently running, have cancelled some build flow jobs today (due to them hanging for reasons I believe are related to JENKINS-23607) and I see this in my Java Melody:

          pool-10-thread-1 yes 5 WAITINGWAITING sun.misc.Unsafe.park(Native Method) 1,985 1,490 Kill the thread pool-10-thread-1
          pool-204-thread-1 no 5 WAITINGWAITING java.lang.Object.wait(Native Method) 4 0 Kill the thread pool-204-thread-1
          pool-204-thread-2 no 5 WAITINGWAITING java.lang.Object.wait(Native Method) 3 0 Kill the thread pool-204-thread-2
          pool-334-thread-3 no 5 WAITINGWAITING sun.misc.Unsafe.park(Native Method) 4 0 Kill the thread pool-334-thread-3
          pool-338-thread-1 no 5 WAITINGWAITING java.lang.Object.wait(Native Method) 3 0 Kill the thread pool-338-thread-1
          pool-338-thread-3 no 5 WAITINGWAITING java.lang.Object.wait(Native Method) 3 0 Kill the thread pool-338-thread-3
          pool-592-thread-3 no 5 WAITINGWAITING sun.misc.Unsafe.park(Native Method) 16 10 Kill the thread pool-592-thread-3
          pool-597-thread-1 no 5 WAITINGWAITING java.lang.Object.wait(Native Method) 2 0 Kill the thread pool-597-thread-1
          pool-597-thread-3 no 5 WAITINGWAITING java.lang.Object.wait(Native Method) 2 0 Kill the thread pool-597-thread-3
          pool-9-thread-1

          Show
          eric_stein_de Eric Stein added a comment - I think I am affected by this. I have zero builds currently running, have cancelled some build flow jobs today (due to them hanging for reasons I believe are related to JENKINS-23607 ) and I see this in my Java Melody: pool-10-thread-1 yes 5 WAITINGWAITING sun.misc.Unsafe.park(Native Method) 1,985 1,490 Kill the thread pool-10-thread-1 pool-204-thread-1 no 5 WAITINGWAITING java.lang.Object.wait(Native Method) 4 0 Kill the thread pool-204-thread-1 pool-204-thread-2 no 5 WAITINGWAITING java.lang.Object.wait(Native Method) 3 0 Kill the thread pool-204-thread-2 pool-334-thread-3 no 5 WAITINGWAITING sun.misc.Unsafe.park(Native Method) 4 0 Kill the thread pool-334-thread-3 pool-338-thread-1 no 5 WAITINGWAITING java.lang.Object.wait(Native Method) 3 0 Kill the thread pool-338-thread-1 pool-338-thread-3 no 5 WAITINGWAITING java.lang.Object.wait(Native Method) 3 0 Kill the thread pool-338-thread-3 pool-592-thread-3 no 5 WAITINGWAITING sun.misc.Unsafe.park(Native Method) 16 10 Kill the thread pool-592-thread-3 pool-597-thread-1 no 5 WAITINGWAITING java.lang.Object.wait(Native Method) 2 0 Kill the thread pool-597-thread-1 pool-597-thread-3 no 5 WAITINGWAITING java.lang.Object.wait(Native Method) 2 0 Kill the thread pool-597-thread-3 pool-9-thread-1

            People

            • Assignee:
              Unassigned
              Reporter:
              mschmidt2014 Mable Schmidt
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: