Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-30271

Jenkins job hangs and cannot be killed after aborting DSL job

    Details

    • Type: Bug
    • Status: Fixed but Unreleased (View Workflow)
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Component/s: job-dsl-plugin
    • Labels:
      None
    • Similar Issues:

      Description

      Running job-dsl-plugin off the master branch (78ae0c67c3d0b3ce630cca0987c5fba869c8dfb1) plus 3 of my outstanding pull requests folded in. I don't think that this issue is related to my changes (which are about adding new DSL configuration options for plugins)

      Steps to reproduce:

      1. Run a job with a Process DSL Script task (ideally long-running)
      2. Abort job while it's executing. Job will not stop, but will finish normally.
      3. Run the job again. Job will hang and cannot be aborted.

      Interestingly, Thread.interrupt() from the script console will not kill it. Using Monitoring plugin's kill feature does kill the job, but the next run will still hang. Only a restart of the Jenkins master fixes the problem.

      UPDATE: also happens without aborting jobs, making this more than minor. BTW, I am running this on two masters and it happened only once on one and a lot on the other.

      Stacktrace for the stuck job:

      Executor #1 for master : executing DSL Job Builder #108
      sun.misc.Unsafe.park(Native Method)
      java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
      org.codehaus.groovy.util.LockableObject.lock(LockableObject.java:34)
      org.codehaus.groovy.reflection.ClassInfo.lock(ClassInfo.java:268)
      org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:193)
      org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:231)
      org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:747)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.createPojoSite(CallSiteArray.java:109)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.createCallSite(CallSiteArray.java:150)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
      javaposse.jobdsl.dsl.helpers.BuildParametersContext.checkParameterName(BuildParametersContext.groovy:270)
      javaposse.jobdsl.dsl.helpers.BuildParametersContext.simpleParam(BuildParametersContext.groovy:187)
      javaposse.jobdsl.dsl.helpers.BuildParametersContext.this$3$simpleParam(BuildParametersContext.groovy)
      javaposse.jobdsl.dsl.helpers.BuildParametersContext$this$3$simpleParam.callCurrent(Unknown Source)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
      javaposse.jobdsl.dsl.helpers.BuildParametersContext$this$3$simpleParam.callCurrent(Unknown Source)
      javaposse.jobdsl.dsl.helpers.BuildParametersContext.stringParam(BuildParametersContext.groovy:179)
      javaposse.jobdsl.dsl.helpers.BuildParametersContext.stringParam(BuildParametersContext.groovy)
      sun.reflect.GeneratedMethodAccessor459.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
      PipelineJobsBuilder$_run_closure1_closure9_closure10_closure11_closure20.doCall(PipelineJobsBuilder.groovy:634)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      groovy.lang.Closure.call(Closure.java:415)
      groovy.lang.Closure.call(Closure.java:428)
      org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1379)
      org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1351)
      org.codehaus.groovy.runtime.dgm$170.invoke(Unknown Source)
      org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
      org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.callSafe(AbstractCallSite.java:82)
      PipelineJobsBuilder$_run_closure1_closure9_closure10_closure11.doCall(PipelineJobsBuilder.groovy:633)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
      PipelineJobsBuilder$_run_closure1_closure9_closure10_closure11.doCall(PipelineJobsBuilder.groovy)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
      org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:54)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
      javaposse.jobdsl.dsl.ContextHelper.executeInContext(ContextHelper.groovy:14)
      javaposse.jobdsl.dsl.ContextHelper$executeInContext.call(Unknown Source)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
      javaposse.jobdsl.dsl.ContextHelper$executeInContext.call(Unknown Source)
      javaposse.jobdsl.dsl.Job.parameters(Job.groovy:468)
      sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
      PipelineJobsBuilder$_run_closure1_closure9_closure10.doCall(PipelineJobsBuilder.groovy:632)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      groovy.lang.Closure.call(Closure.java:415)
      groovy.lang.Closure.call(Closure.java:428)
      org.codehaus.groovy.runtime.DefaultGroovyMethods.with(DefaultGroovyMethods.java:196)
      org.codehaus.groovy.runtime.dgm$926.invoke(Unknown Source)
      org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoMetaMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:313)
      org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:64)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
      org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:69)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
      javaposse.jobdsl.dsl.JobParent.processJob(JobParent.groovy:108)
      sun.reflect.GeneratedMethodAccessor559.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:272)
      org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:52)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
      org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:57)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149)
      javaposse.jobdsl.dsl.JobParent.freeStyleJob(JobParent.groovy:42)
      sun.reflect.GeneratedMethodAccessor637.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361)
      org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnCurrentN(ScriptBytecodeAdapter.java:78)
      PipelineJobsBuilder$_run_closure1_closure9.doCall(PipelineJobsBuilder.groovy:615)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      groovy.lang.Closure.call(Closure.java:415)
      groovy.lang.Closure.call(Closure.java:428)
      org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1379)
      org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1351)
      org.codehaus.groovy.runtime.dgm$170.doMethodInvoke(Unknown Source)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
      org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
      PipelineJobsBuilder$_run_closure1.doCall(PipelineJobsBuilder.groovy:589)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      groovy.lang.Closure.call(Closure.java:415)
      groovy.lang.Closure.call(Closure.java:428)
      org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1379)
      org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1351)
      org.codehaus.groovy.runtime.dgm$170.doMethodInvoke(Unknown Source)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
      org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
      PipelineJobsBuilder.run(PipelineJobsBuilder.groovy:586)
      javaposse.jobdsl.dsl.DslScriptLoader.runDslEngineForParent(DslScriptLoader.java:80)
      javaposse.jobdsl.dsl.DslScriptLoader.runDslEngine(DslScriptLoader.java:123)
      javaposse.jobdsl.plugin.ExecuteDslScripts.perform(ExecuteDslScripts.java:216)
      hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
      hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:761)
      hudson.model.Build$BuildExecution.build(Build.java:203)
      hudson.model.Build$BuildExecution.doRun(Build.java:160)
      hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
      hudson.model.Run.execute(Run.java:1741)
      hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      hudson.model.ResourceController.execute(ResourceController.java:98)
      hudson.model.Executor.run(Executor.java:374)

        Attachments

        1. config.xml
          5 kB
        2. job-dsl.hpi
          2.33 MB
        3. Stuck_seedjob_stacktrace.txt
          20 kB
        4. Thread Dump.txt
          51 kB

          Issue Links

            Activity

            Hide
            daspilker Daniel Spilker added a comment -

            Sebastian Hoß that is a different problem with a different stack trace.

            Can someone reproduce the problem on Jenkins 2.0? If not, I'm going to close this ticket.

            Show
            daspilker Daniel Spilker added a comment - Sebastian Hoß that is a different problem with a different stack trace. Can someone reproduce the problem on Jenkins 2.0? If not, I'm going to close this ticket.
            Hide
            pianoroy Roy Tinker added a comment -

            I'm having the exact problem and stack trace Sebastian pasted. Freestyle builds are hanging. This is Jenkins 2.53 on JDK7.

            Show
            pianoroy Roy Tinker added a comment - I'm having the exact problem and stack trace Sebastian pasted. Freestyle builds are hanging. This is Jenkins 2.53 on JDK7.
            Hide
            akom Alexander Komarov added a comment -

            Daniel Spilker I have not reproduced the problem in 2.x but mostly because I'm too scared to try to kill a DSL run, I even have a red notice against that in the description.  It's quite possible that it's fixed because I haven't had it happen without aborting anymore.

            Show
            akom Alexander Komarov added a comment - Daniel Spilker I have not reproduced the problem in 2.x but mostly because I'm too scared to try to kill a DSL run, I even have a red notice against that in the description.  It's quite possible that it's fixed because I haven't had it happen without aborting anymore.
            Hide
            daspilker Daniel Spilker added a comment -

            Groovy has been updated from 2.4.8 to 2.4.11 in Jenkins 2.61. The issue should be fixed, see https://issues.apache.org/jira/browse/GROOVY-8067. Can anyone reproduce the problem in 2.61 or later? If not, I'm going to close this ticket.

            Show
            daspilker Daniel Spilker added a comment - Groovy has been updated from 2.4.8 to 2.4.11 in Jenkins 2.61. The issue should be fixed, see https://issues.apache.org/jira/browse/GROOVY-8067. Can anyone reproduce the problem in 2.61 or later? If not, I'm going to close this ticket.
            Hide
            daspilker Daniel Spilker added a comment -

            I'm closing the issue because the problem does not occur with newer versions of Jenkins and Job DSL. Please re-open if you can reproduce the problem.

            Show
            daspilker Daniel Spilker added a comment - I'm closing the issue because the problem does not occur with newer versions of Jenkins and Job DSL. Please re-open if you can reproduce the problem.

              People

              • Assignee:
                daspilker Daniel Spilker
                Reporter:
                akom Alexander Komarov
              • Votes:
                6 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: