Details

    • Similar Issues:

      Description

      When batch task is included in a Pipeline job it will hang on completion of the batch task. I can see in task manager that the job startup it logs data to jenkins-log.txt. The batch completes and I see in task manager that the batch is nolonger running. But Jenkins is still waiting for the task to complete. I do not see jenkins-result.txt writen to the workspace tmp durabletask directory. If I create the file manually or run the workflow-wrap.bat manually the task completes. This is an itermitent bug. Task might work 3 time then fail 5 times then work 8 times. No change to the system during this time. I am setting the job to run every min to see what the stats look like for longer run.

      job:

      node { 
       bat 'ping 127.0.0.1 -n 10'
       echo 'batch completed'
      }
      

      Could be any command you want ping is just an easy one to have it take a little bit of time. And require nothing else installed on machine.

      I see many other task like this I have tested on serveral differnt machines using a base install of Jenkins.

        Attachments

        1. durable-task.hpi
          41 kB
        2. durable-task.hpi
          41 kB
        3. How.png
          How.png
          321 kB
        4. jenkins2-pipeline-batch-hang.png
          jenkins2-pipeline-batch-hang.png
          109 kB
        5. jenkins2-pipeline-batch-nohang.png
          jenkins2-pipeline-batch-nohang.png
          98 kB

          Issue Links

            Activity

            dpd_30 Daniel Daugherty created issue -
            dpd_30 Daniel Daugherty made changes -
            Field Original Value New Value
            Epic Link JENKINS-31152 [ 165808 ]
            rtyler R. Tyler Croy made changes -
            Description When batch task is included in a Pipeline job it will hang on completion of the batch task. I can see in task manager that the job startup it logs data to jenkins-log.txt. The batch completes and I see in task manager that the batch is nolonger running. But Jenkins is still waiting for the task to complete. I do not see jenkins-result.txt writen to the workspace tmp durabletask directory. If I create the file manually or run the workflow-wrap.bat manually the task completes. This is an itermitent bug. Task might work 3 time then fail 5 times then work 8 times. No change to the system during this time. I am setting the job to run every min to see what the stats look like for longer run.

            job:
            node {
             bat 'ping 127.0.0.1 -n 10'
             echo 'batch completed'
            }

            Could be any command you want ping is just an easy one to have it take a little bit of time. And require nothing else installed on machine.

            I see many other task like this I have tested on serveral differnt machines using a base install of Jenkins.
            When batch task is included in a Pipeline job it will hang on completion of the batch task. I can see in task manager that the job startup it logs data to jenkins-log.txt. The batch completes and I see in task manager that the batch is nolonger running. But Jenkins is still waiting for the task to complete. I do not see jenkins-result.txt writen to the workspace tmp durabletask directory. If I create the file manually or run the workflow-wrap.bat manually the task completes. This is an itermitent bug. Task might work 3 time then fail 5 times then work 8 times. No change to the system during this time. I am setting the job to run every min to see what the stats look like for longer run.

            job:
            {code}
            node {
             bat 'ping 127.0.0.1 -n 10'
             echo 'batch completed'
            }
            {code}

            Could be any command you want ping is just an easy one to have it take a little bit of time. And require nothing else installed on machine.

            I see many other task like this I have tested on serveral differnt machines using a base install of Jenkins.
            rtyler R. Tyler Croy made changes -
            Labels workflow 2.0 workflow
            Hide
            rtyler R. Tyler Croy added a comment -

            I have just tried this on a Windows Server 2012R2 instance on EC2 with the Jenkins 2.0RC1 installer.

            Using the Pipeline script:

            node {
              bat 'ping 127.0.0.1 -n 10'
              echo 'pinging done!'
            }
            

            I was able to successfully run the Pipeline a few times. I added a sleep 30 and was also able to abort the Pipeline successfully.

            This is with:

            • Durable Task 1.9
            • Pipeline 2.0
            • Pipeline Job, Pipeline Groovy 2.1
            Show
            rtyler R. Tyler Croy added a comment - I have just tried this on a Windows Server 2012R2 instance on EC2 with the Jenkins 2.0RC1 installer. Using the Pipeline script: node { bat 'ping 127.0.0.1 -n 10' echo 'pinging done!' } I was able to successfully run the Pipeline a few times. I added a sleep 30 and was also able to abort the Pipeline successfully. This is with: Durable Task 1.9 Pipeline 2.0 Pipeline Job, Pipeline Groovy 2.1
            rtyler R. Tyler Croy made changes -
            Attachment jenkins2-pipeline-batch-nohang.png [ 32426 ]
            rtyler R. Tyler Croy made changes -
            Labels 2.0 workflow 2.0 testfest workflow
            Hide
            rtyler R. Tyler Croy added a comment -

            Daniel Daugherty I'm going to mark this as "Cannot Reproduce" but please re-open if you have additional reproduction steps that I could use to help verify.

            Show
            rtyler R. Tyler Croy added a comment - Daniel Daugherty I'm going to mark this as "Cannot Reproduce" but please re-open if you have additional reproduction steps that I could use to help verify.
            rtyler R. Tyler Croy made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Assignee Jesse Glick [ jglick ] R. Tyler Croy [ rtyler ]
            Resolution Cannot Reproduce [ 5 ]
            Hide
            rtyler R. Tyler Croy added a comment -

            I think I actually found the reproduction case for this!

            If you click the "Build Now" button a lot to where pipelines are running in parallel the pipelines will all hang.

            Show
            rtyler R. Tyler Croy added a comment - I think I actually found the reproduction case for this! If you click the "Build Now" button a lot to where pipelines are running in parallel the pipelines will all hang.
            rtyler R. Tyler Croy made changes -
            Resolution Cannot Reproduce [ 5 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            rtyler R. Tyler Croy made changes -
            Attachment jenkins2-pipeline-batch-hang.png [ 32433 ]
            Hide
            rtyler R. Tyler Croy added a comment -

            Thread dump from a Windows 2012R2 server with two hanging Pipeline executions

            Thread Dump
            
            Computer.threadPoolForRemoting [#6]
            "Computer.threadPoolForRemoting [#6]" Id=197 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@d13463
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.SynchronousQueue$TransferStack@d13463
            	at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            	at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            Computer.threadPoolForRemoting [#7]
            "Computer.threadPoolForRemoting [#7]" Id=198 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@d13463
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.SynchronousQueue$TransferStack@d13463
            	at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            	at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            DestroyJavaVM
            "DestroyJavaVM" Id=16 Group=main RUNNABLE
            
            
            
            FilePath.localPool [#5]
            "FilePath.localPool [#5]" Id=208 Group=main RUNNABLE
            	at sun.management.ThreadImpl.dumpThreads0(Native Method)
            	at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
            	at hudson.Functions.getThreadInfos(Functions.java:1195)
            	at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:98)
            	at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:95)
            	at hudson.remoting.LocalChannel$1.call(LocalChannel.java:52)
            	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
            	at java.util.concurrent.FutureTask.run(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            	Number of locked synchronizers = 1
            	- java.util.concurrent.ThreadPoolExecutor$Worker@1f9039d
            
            
            
            Handling GET /threadDump from 127.0.0.1 : RequestHandlerThread[#10] Jenkins/threadDump.jelly
            "Handling GET /threadDump from 127.0.0.1 : RequestHandlerThread[#10] Jenkins/threadDump.jelly" Id=150 Group=main TIMED_WAITING on java.util.concurrent.FutureTask@295f13
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.FutureTask@295f13
            	at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            	at java.util.concurrent.FutureTask.awaitDone(Unknown Source)
            	at java.util.concurrent.FutureTask.get(Unknown Source)
            	at hudson.remoting.LocalChannel$2.get(LocalChannel.java:81)
            	at jenkins.model.Jenkins.getAllThreadDumps(Jenkins.java:3522)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
            	at java.lang.reflect.Method.invoke(Unknown Source)
            	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
            	at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
            	at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
            	at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
            	at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
            	at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
            	at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
            	at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
            	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
            	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
            	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
            	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
            	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
            	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
            	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
            	at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
            	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
            	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
            	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
            	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
            	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
            	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
            	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
            	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
            	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
            	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
            	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
            	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
            	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
            	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
            	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
            	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
            	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
            	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
            	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
            	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
            	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
            	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
            	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
            	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
            	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
            	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
            	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
            	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
            	at org.eclipse.jetty.server.Server.handle(Server.java:499)
            	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
            	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
            	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
            	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            	Number of locked synchronizers = 1
            	- java.util.concurrent.ThreadPoolExecutor$Worker@fa28b
            
            
            
            Jenkins cron thread
            "Jenkins cron thread" Id=17 Group=main WAITING on java.util.TaskQueue@1679501
            	at java.lang.Object.wait(Native Method)
            	-  waiting on java.util.TaskQueue@1679501
            	at java.lang.Object.wait(Unknown Source)
            	at java.util.TimerThread.mainLoop(Unknown Source)
            	at java.util.TimerThread.run(Unknown Source)
            
            
            
            Jenkins UDP 33848 monitoring thread
            "Jenkins UDP 33848 monitoring thread" Id=45 Group=main RUNNABLE (in native)
            	at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method)
            	-  locked java.net.TwoStacksPlainDatagramSocketImpl@16ef542
            	at java.net.TwoStacksPlainDatagramSocketImpl.receive(Unknown Source)
            	-  locked java.net.TwoStacksPlainDatagramSocketImpl@16ef542
            	at java.net.DatagramSocket.receive(Unknown Source)
            	-  locked java.net.DatagramPacket@f55b4d
            	-  locked java.net.MulticastSocket@1bb58c1
            	at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:83)
            
            
            
            jenkins.util.Timer [#10]
            "jenkins.util.Timer [#10]" Id=41 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
            	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            jenkins.util.Timer [#1]
            "jenkins.util.Timer [#1]" Id=18 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
            	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            jenkins.util.Timer [#2]
            "jenkins.util.Timer [#2]" Id=30 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            jenkins.util.Timer [#3]
            "jenkins.util.Timer [#3]" Id=31 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
            	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            jenkins.util.Timer [#4]
            "jenkins.util.Timer [#4]" Id=32 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
            	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            jenkins.util.Timer [#5]
            "jenkins.util.Timer [#5]" Id=33 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
            	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            jenkins.util.Timer [#6]
            "jenkins.util.Timer [#6]" Id=34 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
            	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            jenkins.util.Timer [#7]
            "jenkins.util.Timer [#7]" Id=35 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
            	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            jenkins.util.Timer [#8]
            "jenkins.util.Timer [#8]" Id=36 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
            	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            jenkins.util.Timer [#9]
            "jenkins.util.Timer [#9]" Id=37 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7
            	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
            	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            JmDNS(WIN-N2IVJVRIP82.local.).State.Timer
            "JmDNS(WIN-N2IVJVRIP82.local.).State.Timer" Id=51 Group=main TIMED_WAITING on java.util.TaskQueue@12bccb2
            	at java.lang.Object.wait(Native Method)
            	-  waiting on java.util.TaskQueue@12bccb2
            	at java.util.TimerThread.mainLoop(Unknown Source)
            	at java.util.TimerThread.run(Unknown Source)
            
            
            
            JmDNS(WIN-N2IVJVRIP82.local.).Timer
            "JmDNS(WIN-N2IVJVRIP82.local.).Timer" Id=50 Group=main TIMED_WAITING on java.util.TaskQueue@b48798
            	at java.lang.Object.wait(Native Method)
            	-  waiting on java.util.TaskQueue@b48798
            	at java.util.TimerThread.mainLoop(Unknown Source)
            	at java.util.TimerThread.run(Unknown Source)
            
            
            
            LauncherControlThread[ControlPort=-1]
            "LauncherControlThread[ControlPort=-1]" Id=14 Group=main TIMED_WAITING
            	at java.lang.Thread.sleep(Native Method)
            	at winstone.Launcher.run(Launcher.java:243)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            NioSocketAcceptor-1
            "NioSocketAcceptor-1" Id=43 Group=main RUNNABLE (in native)
            	at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
            	at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
            	at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
            	at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
            	at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
            	-  locked sun.nio.ch.Util$2@150341e
            	-  locked java.util.Collections$UnmodifiableSet@1372a4a
            	-  locked sun.nio.ch.WindowsSelectorImpl@350cfd
            	at sun.nio.ch.SelectorImpl.select(Unknown Source)
            	at sun.nio.ch.SelectorImpl.select(Unknown Source)
            	at org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:238)
            	at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:432)
            	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            	Number of locked synchronizers = 1
            	- java.util.concurrent.ThreadPoolExecutor$Worker@15a9d55
            
            
            
            org.eclipse.jetty.server.session.HashSessionManager@1ea3742Timer
            "org.eclipse.jetty.server.session.HashSessionManager@1ea3742Timer" Id=9 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@da471
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@da471
            	at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            RequestHandlerThread[#13]
            "RequestHandlerThread[#13]" Id=194 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@16c689
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.SynchronousQueue$TransferStack@16c689
            	at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            	at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            RequestHandlerThread[#1]-selector-ServerConnectorManager@15e4a26/0
            "RequestHandlerThread[#1]-selector-ServerConnectorManager@15e4a26/0" Id=12 Group=main RUNNABLE (in native)
            	at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
            	at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
            	at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
            	at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
            	at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
            	-  locked sun.nio.ch.Util$2@1f243b0
            	-  locked java.util.Collections$UnmodifiableSet@552324
            	-  locked sun.nio.ch.WindowsSelectorImpl@16f6b72
            	at sun.nio.ch.SelectorImpl.select(Unknown Source)
            	at sun.nio.ch.SelectorImpl.select(Unknown Source)
            	at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
            	at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
            	at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
            	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            	Number of locked synchronizers = 1
            	- java.util.concurrent.ThreadPoolExecutor$Worker@12646b
            
            
            
            RequestHandlerThread[#2]-acceptor-0@14888b2-ServerConnector@24a83d{HTTP/1.1}{0.0.0.0:8080}
            "RequestHandlerThread[#2]-acceptor-0@14888b2-ServerConnector@24a83d{HTTP/1.1}{0.0.0.0:8080}" Id=13 Group=main RUNNABLE (in native)
            	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
            	at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
            	at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
            	-  locked java.lang.Object@1bb3a34
            	at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
            	at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
            	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            	Number of locked synchronizers = 1
            	- java.util.concurrent.ThreadPoolExecutor$Worker@136aa0c
            
            
            
            RequestHandlerThread[#3]
            "RequestHandlerThread[#3]" Id=24 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@16c689
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.SynchronousQueue$TransferStack@16c689
            	at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            	at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            RequestHandlerThread[#9]
            "RequestHandlerThread[#9]" Id=149 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@16c689
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.SynchronousQueue$TransferStack@16c689
            	at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            	at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            Scheduler-5482987
            "Scheduler-5482987" Id=23 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e28f0d
            	at sun.misc.Unsafe.park(Native Method)
            	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e28f0d
            	at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            SocketListener(WIN-N2IVJVRIP82.local.)
            "SocketListener(WIN-N2IVJVRIP82.local.)" Id=49 Group=main RUNNABLE (in native)
            	at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method)
            	-  locked java.net.TwoStacksPlainDatagramSocketImpl@ca2349
            	at java.net.TwoStacksPlainDatagramSocketImpl.receive(Unknown Source)
            	-  locked java.net.TwoStacksPlainDatagramSocketImpl@ca2349
            	at java.net.DatagramSocket.receive(Unknown Source)
            	-  locked java.net.DatagramPacket@bf36c4
            	-  locked java.net.MulticastSocket@10433fd
            	at javax.jmdns.impl.SocketListener.run(SocketListener.java:41)
            
            
            
            TCP agent listener port=0
            "TCP agent listener port=0" Id=44 Group=main RUNNABLE (in native)
            	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
            	at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
            	at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
            	-  locked java.lang.Object@14a0c50
            	at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:110)
            
            
            
            Attach Listener
            "Attach Listener" Id=4 Group=system RUNNABLE
            
            
            
            Finalizer
            "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@13ef3d7
            	at java.lang.Object.wait(Native Method)
            	-  waiting on java.lang.ref.ReferenceQueue$Lock@13ef3d7
            	at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            	at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            	at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
            
            
            
            Java2D Disposer
            "Java2D Disposer" Id=10 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@19b9fb1
            	at java.lang.Object.wait(Native Method)
            	-  waiting on java.lang.ref.ReferenceQueue$Lock@19b9fb1
            	at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            	at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            	at sun.java2d.Disposer.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            
            
            
            Reference Handler
            "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@1522b39
            	at java.lang.Object.wait(Native Method)
            	-  waiting on java.lang.ref.Reference$Lock@1522b39
            	at java.lang.Object.wait(Unknown Source)
            	at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
            
            
            
            Show
            rtyler R. Tyler Croy added a comment - Thread dump from a Windows 2012R2 server with two hanging Pipeline executions Thread Dump Computer.threadPoolForRemoting [#6] "Computer.threadPoolForRemoting [#6]" Id=197 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@d13463 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@d13463 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) Computer.threadPoolForRemoting [#7] "Computer.threadPoolForRemoting [#7]" Id=198 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@d13463 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@d13463 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) DestroyJavaVM "DestroyJavaVM" Id=16 Group=main RUNNABLE FilePath.localPool [#5] "FilePath.localPool [#5]" Id=208 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(Unknown Source) at hudson.Functions.getThreadInfos(Functions.java:1195) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:98) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:95) at hudson.remoting.LocalChannel$1.call(LocalChannel.java:52) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@1f9039d Handling GET /threadDump from 127.0.0.1 : RequestHandlerThread[#10] Jenkins/threadDump.jelly "Handling GET /threadDump from 127.0.0.1 : RequestHandlerThread[#10] Jenkins/threadDump.jelly" Id=150 Group=main TIMED_WAITING on java.util.concurrent.FutureTask@295f13 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.FutureTask@295f13 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.FutureTask.awaitDone(Unknown Source) at java.util.concurrent.FutureTask.get(Unknown Source) at hudson.remoting.LocalChannel$2.get(LocalChannel.java:81) at jenkins.model.Jenkins.getAllThreadDumps(Jenkins.java:3522) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258) at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74) at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@fa28b Jenkins cron thread "Jenkins cron thread" Id=17 Group=main WAITING on java.util.TaskQueue@1679501 at java.lang. Object .wait(Native Method) - waiting on java.util.TaskQueue@1679501 at java.lang. Object .wait(Unknown Source) at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) Jenkins UDP 33848 monitoring thread "Jenkins UDP 33848 monitoring thread" Id=45 Group=main RUNNABLE (in native ) at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method) - locked java.net.TwoStacksPlainDatagramSocketImpl@16ef542 at java.net.TwoStacksPlainDatagramSocketImpl.receive(Unknown Source) - locked java.net.TwoStacksPlainDatagramSocketImpl@16ef542 at java.net.DatagramSocket.receive(Unknown Source) - locked java.net.DatagramPacket@f55b4d - locked java.net.MulticastSocket@1bb58c1 at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:83) jenkins.util.Timer [#10] "jenkins.util.Timer [#10]" Id=41 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) jenkins.util.Timer [#1] "jenkins.util.Timer [#1]" Id=18 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) jenkins.util.Timer [#2] "jenkins.util.Timer [#2]" Id=30 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) jenkins.util.Timer [#3] "jenkins.util.Timer [#3]" Id=31 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) jenkins.util.Timer [#4] "jenkins.util.Timer [#4]" Id=32 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) jenkins.util.Timer [#5] "jenkins.util.Timer [#5]" Id=33 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) jenkins.util.Timer [#6] "jenkins.util.Timer [#6]" Id=34 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) jenkins.util.Timer [#7] "jenkins.util.Timer [#7]" Id=35 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) jenkins.util.Timer [#8] "jenkins.util.Timer [#8]" Id=36 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) jenkins.util.Timer [#9] "jenkins.util.Timer [#9]" Id=37 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@182ace7 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) JmDNS(WIN-N2IVJVRIP82.local.).State.Timer "JmDNS(WIN-N2IVJVRIP82.local.).State.Timer" Id=51 Group=main TIMED_WAITING on java.util.TaskQueue@12bccb2 at java.lang. Object .wait(Native Method) - waiting on java.util.TaskQueue@12bccb2 at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) JmDNS(WIN-N2IVJVRIP82.local.).Timer "JmDNS(WIN-N2IVJVRIP82.local.).Timer" Id=50 Group=main TIMED_WAITING on java.util.TaskQueue@b48798 at java.lang. Object .wait(Native Method) - waiting on java.util.TaskQueue@b48798 at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) LauncherControlThread[ControlPort=-1] "LauncherControlThread[ControlPort=-1]" Id=14 Group=main TIMED_WAITING at java.lang. Thread .sleep(Native Method) at winstone.Launcher.run(Launcher.java:243) at java.lang. Thread .run(Unknown Source) NioSocketAcceptor-1 "NioSocketAcceptor-1" Id=43 Group=main RUNNABLE (in native ) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source) at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source) at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) - locked sun.nio.ch.Util$2@150341e - locked java.util.Collections$UnmodifiableSet@1372a4a - locked sun.nio.ch.WindowsSelectorImpl@350cfd at sun.nio.ch.SelectorImpl.select(Unknown Source) at sun.nio.ch.SelectorImpl.select(Unknown Source) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:238) at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:432) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@15a9d55 org.eclipse.jetty.server.session.HashSessionManager@1ea3742Timer "org.eclipse.jetty.server.session.HashSessionManager@1ea3742Timer" Id=9 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@da471 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@da471 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) RequestHandlerThread[#13] "RequestHandlerThread[#13]" Id=194 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@16c689 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@16c689 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) RequestHandlerThread[#1]-selector-ServerConnectorManager@15e4a26/0 "RequestHandlerThread[#1]-selector-ServerConnectorManager@15e4a26/0" Id=12 Group=main RUNNABLE (in native ) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source) at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source) at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) - locked sun.nio.ch.Util$2@1f243b0 - locked java.util.Collections$UnmodifiableSet@552324 - locked sun.nio.ch.WindowsSelectorImpl@16f6b72 at sun.nio.ch.SelectorImpl.select(Unknown Source) at sun.nio.ch.SelectorImpl.select(Unknown Source) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@12646b RequestHandlerThread[#2]-acceptor-0@14888b2-ServerConnector@24a83d{HTTP/1.1}{0.0.0.0:8080} "RequestHandlerThread[#2]-acceptor-0@14888b2-ServerConnector@24a83d{HTTP/1.1}{0.0.0.0:8080}" Id=13 Group=main RUNNABLE (in native ) at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source) at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source) - locked java.lang. Object @1bb3a34 at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@136aa0c RequestHandlerThread[#3] "RequestHandlerThread[#3]" Id=24 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@16c689 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@16c689 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) RequestHandlerThread[#9] "RequestHandlerThread[#9]" Id=149 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@16c689 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@16c689 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) Scheduler-5482987 "Scheduler-5482987" Id=23 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e28f0d at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e28f0d at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) SocketListener(WIN-N2IVJVRIP82.local.) "SocketListener(WIN-N2IVJVRIP82.local.)" Id=49 Group=main RUNNABLE (in native ) at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method) - locked java.net.TwoStacksPlainDatagramSocketImpl@ca2349 at java.net.TwoStacksPlainDatagramSocketImpl.receive(Unknown Source) - locked java.net.TwoStacksPlainDatagramSocketImpl@ca2349 at java.net.DatagramSocket.receive(Unknown Source) - locked java.net.DatagramPacket@bf36c4 - locked java.net.MulticastSocket@10433fd at javax.jmdns.impl.SocketListener.run(SocketListener.java:41) TCP agent listener port=0 "TCP agent listener port=0" Id=44 Group=main RUNNABLE (in native ) at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source) at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source) - locked java.lang. Object @14a0c50 at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:110) Attach Listener "Attach Listener" Id=4 Group=system RUNNABLE Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@13ef3d7 at java.lang. Object .wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@13ef3d7 at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Java2D Disposer "Java2D Disposer" Id=10 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@19b9fb1 at java.lang. Object .wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@19b9fb1 at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at sun.java2d.Disposer.run(Unknown Source) at java.lang. Thread .run(Unknown Source) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@1522b39 at java.lang. Object .wait(Native Method) - waiting on java.lang.ref.Reference$Lock@1522b39 at java.lang. Object .wait(Unknown Source) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
            Hide
            rtyler R. Tyler Croy added a comment -

            A thread dump from the Pipeline execution itself:

            Thread #2
            	at DSL.bat(Native Method)
            	at WorkflowScript.run(WorkflowScript:2)
            	at DSL.node(Native Method)
            	at WorkflowScript.run(WorkflowScript:1)
            
            Show
            rtyler R. Tyler Croy added a comment - A thread dump from the Pipeline execution itself: Thread #2 at DSL.bat(Native Method) at WorkflowScript.run(WorkflowScript:2) at DSL.node(Native Method) at WorkflowScript.run(WorkflowScript:1)
            rtyler R. Tyler Croy made changes -
            Assignee R. Tyler Croy [ rtyler ] Antonio Muñiz [ amuniz ]
            Hide
            amuniz Antonio Muñiz added a comment -

            Something to fix in durable-task, nothing to do with Pipeline I believe.

            Show
            amuniz Antonio Muñiz added a comment - Something to fix in durable-task, nothing to do with Pipeline I believe.
            Hide
            rufer7 Marc Rufer added a comment - - edited

            I had the same issue with the following Jenkinsfile

            node {
              checkout scm
              bat 'Powershell.exe -executionpolicy remotesigned -File Build.ps1'
            }
            

            Batch hangs, if there are multiple executions running in parallel. Sometimes the problem occurs as well if there is only one job running

            Show
            rufer7 Marc Rufer added a comment - - edited I had the same issue with the following Jenkinsfile node { checkout scm bat 'Powershell.exe -executionpolicy remotesigned -File Build.ps1' } Batch hangs, if there are multiple executions running in parallel. Sometimes the problem occurs as well if there is only one job running
            Hide
            dpd_30 Daniel Daugherty added a comment -

            I agree with Antonio that this is a durable task issue. While trouble shooting this I reverted a logging change in durable task and the issue could not longer be replicated. I suspected a locking issue with the results file where the durable task was locking the results file causing the batch to not be able to create it. But the current 1.9 version of the durable task plugin does not capture the output of the wraper batch file. When I enable the logging of that information the issue no longer shows up. True heisenbug. The act of looking for the cause of the problem causes it to nolonger exist. I submitted pull request to the durable task plugin with the 3 line change that I made. Reverting to older logging code in that project. https://github.com/jenkinsci/durable-task-plugin/pull/20

            While this resolves the issue I do not feel that it is a fix. Logging or not logging should not cause batch to work or not work. It is masking the underlying issue where the result file does not get created. But with out the logging you can't see why the result file is not created.

            Multiple executions is one of the ways that I have triggered it also.

            Show
            dpd_30 Daniel Daugherty added a comment - I agree with Antonio that this is a durable task issue. While trouble shooting this I reverted a logging change in durable task and the issue could not longer be replicated. I suspected a locking issue with the results file where the durable task was locking the results file causing the batch to not be able to create it. But the current 1.9 version of the durable task plugin does not capture the output of the wraper batch file. When I enable the logging of that information the issue no longer shows up. True heisenbug. The act of looking for the cause of the problem causes it to nolonger exist. I submitted pull request to the durable task plugin with the 3 line change that I made. Reverting to older logging code in that project. https://github.com/jenkinsci/durable-task-plugin/pull/20 While this resolves the issue I do not feel that it is a fix. Logging or not logging should not cause batch to work or not work. It is masking the underlying issue where the result file does not get created. But with out the logging you can't see why the result file is not created. Multiple executions is one of the ways that I have triggered it also.
            Hide
            nitram Martin Karing added a comment -

            It is a bug in the durable task plugin.
            I created a test to reproduce it and submitted a fix to resolve the issue.

            Both neatly bundled as a pull request: https://github.com/jenkinsci/durable-task-plugin/pull/21

            Show
            nitram Martin Karing added a comment - It is a bug in the durable task plugin. I created a test to reproduce it and submitted a fix to resolve the issue. Both neatly bundled as a pull request: https://github.com/jenkinsci/durable-task-plugin/pull/21
            Hide
            amuniz Antonio Muñiz added a comment -

            After some hours trying combinations (and installing Windows VMs), I've not been able to reproduce it. Tried on:

            1. Jenkins 2.0 RC1 (running on Ubuntu, Durable Task 1.9) + Agent on Windows Server 2008R2 (JDK8)
            2. Jenkins 2.0 RC1 (running on OSX, Durable Task 1.9) + Agent on Windows Server 2012R2 (JDK8)

            My test script is:

            node ('windows') {
                bat 'ping 192.168.1.108 -n 10' // this is the Jenkins master IP in my local network
                echo 'batch completed'
            }
            

            I executed up to 10 concurrent builds. All of them finished successfully.

            Show
            amuniz Antonio Muñiz added a comment - After some hours trying combinations (and installing Windows VMs), I've not been able to reproduce it. Tried on: Jenkins 2.0 RC1 (running on Ubuntu, Durable Task 1.9) + Agent on Windows Server 2008R2 (JDK8) Jenkins 2.0 RC1 (running on OSX, Durable Task 1.9) + Agent on Windows Server 2012R2 (JDK8) My test script is: node ( 'windows' ) { bat 'ping 192.168.1.108 -n 10' // this is the Jenkins master IP in my local network echo 'batch completed' } I executed up to 10 concurrent builds. All of them finished successfully.
            Hide
            nitram Martin Karing added a comment -

            I developed the test case and the fix on a Windows 10 development box.
            I encountered the issue on a windows 2012 r2 server.

            On both systems the hanging happened consistently. How ever on Windows 2012 I never encountered the problem in case only one script was executing. But starting with two scripts there was a fair chance of one of the scripts hanging. My test case on Windows 10 caused the hanging problem in 100% of the tests even with a single running task.

            My best guess is that it is a timing issue, the test case I provided queries the exit status as fast as possible and causes the hanging issue this way. The problem is that the batch script does not create the file while the plugin is checking the file system for the presence of the file. I do not know why this happens, in my mind the file creation should be unrelated to the file presence check, but it seems on Windows systems it is not.

            Show
            nitram Martin Karing added a comment - I developed the test case and the fix on a Windows 10 development box. I encountered the issue on a windows 2012 r2 server. On both systems the hanging happened consistently. How ever on Windows 2012 I never encountered the problem in case only one script was executing. But starting with two scripts there was a fair chance of one of the scripts hanging. My test case on Windows 10 caused the hanging problem in 100% of the tests even with a single running task. My best guess is that it is a timing issue, the test case I provided queries the exit status as fast as possible and causes the hanging issue this way. The problem is that the batch script does not create the file while the plugin is checking the file system for the presence of the file. I do not know why this happens, in my mind the file creation should be unrelated to the file presence check, but it seems on Windows systems it is not.
            amuniz Antonio Muñiz made changes -
            Status Reopened [ 4 ] In Progress [ 3 ]
            Hide
            amuniz Antonio Muñiz added a comment -

            Martin Karing Thanks for the info. I'm going to move the master to a Windows box too as it's the only thing I see my environment differing from yours.

            Show
            amuniz Antonio Muñiz added a comment - Martin Karing Thanks for the info. I'm going to move the master to a Windows box too as it's the only thing I see my environment differing from yours.
            Hide
            amuniz Antonio Muñiz added a comment -

            The issue is reproducible only if two batch durable tasks run concurrently on the master node.

            Show
            amuniz Antonio Muñiz added a comment - The issue is reproducible only if two batch durable tasks run concurrently on the master node.
            Hide
            amuniz Antonio Muñiz added a comment -

            For some reason the result file is not being created, this line must be failing to execute (but I can not find any log):

            echo %ERRORLEVEL% > "[JENKINS_HOME]\my-job@tmp\durable-acc8d5a4\jenkins-result.txt"
            

            While the build was hanging, I manually executed jenkins-wrap.bat and it made the execution to finish (as the result file was created).

            Trying to see why the result file is not being written.

            Show
            amuniz Antonio Muñiz added a comment - For some reason the result file is not being created, this line must be failing to execute (but I can not find any log): echo %ERRORLEVEL% > "[JENKINS_HOME]\my-job@tmp\durable-acc8d5a4\jenkins-result.txt" While the build was hanging, I manually executed jenkins-wrap.bat and it made the execution to finish (as the result file was created). Trying to see why the result file is not being written.
            Hide
            danielbeck Daniel Beck added a comment -

            Wild guess: Spaces in JENKINS_HOME path when installing using the installer?

            Show
            danielbeck Daniel Beck added a comment - Wild guess: Spaces in JENKINS_HOME path when installing using the installer?
            Hide
            amuniz Antonio Muñiz added a comment -

            No. It's reproducible for me without spaces in the path (and didn't use the installer but direct java -jar mode).

            Show
            amuniz Antonio Muñiz added a comment - No. It's reproducible for me without spaces in the path (and didn't use the installer but direct java -jar mode).
            danielbeck Daniel Beck made changes -
            Labels 2.0 testfest workflow 2.0 2.0-planned testfest workflow
            Hide
            amuniz Antonio Muñiz added a comment -

            Not a regression in core at least, reproduced in 1.651.1 (Pipeline Durable Task Step 2.0 + Durable Task 1.9)

            Show
            amuniz Antonio Muñiz added a comment - Not a regression in core at least, reproduced in 1.651.1 (Pipeline Durable Task Step 2.0 + Durable Task 1.9)
            Hide
            dpd_30 Daniel Daugherty added a comment -

            Antonio, If you apply my pull request (20) you will see the logs for the jenkins-wraper.bat to include the echo command run. But you will no longer experince the error. Just as you will not enconter the error if you apply Martins pull (21). The main action that both change is that a reference to Launcher.ProcStarter ps is maintained after doLaunch() is called. This to me speaks of a possible GC issue where the Proc is destroyed before task completes. And that causes the wraper to not finish. But when I watch performance monitor in windows as the job is runing I don't see the command prompts being killed early. So may locking issue as Martin mentioned earlier. Where checking for the existince of the result file prevents the result from being created. But that does not explain why keeping a reference to the PS instance causes the bug to no longer happen. That speaks more to something happening due to GC. If this was C I would call it a use after free errror. Where PS is nolonger reference but expected to continue to do things.

            Show
            dpd_30 Daniel Daugherty added a comment - Antonio, If you apply my pull request (20) you will see the logs for the jenkins-wraper.bat to include the echo command run. But you will no longer experince the error. Just as you will not enconter the error if you apply Martins pull (21). The main action that both change is that a reference to Launcher.ProcStarter ps is maintained after doLaunch() is called. This to me speaks of a possible GC issue where the Proc is destroyed before task completes. And that causes the wraper to not finish. But when I watch performance monitor in windows as the job is runing I don't see the command prompts being killed early. So may locking issue as Martin mentioned earlier. Where checking for the existince of the result file prevents the result from being created. But that does not explain why keeping a reference to the PS instance causes the bug to no longer happen. That speaks more to something happening due to GC. If this was C I would call it a use after free errror. Where PS is nolonger reference but expected to continue to do things.
            Hide
            nitram Martin Karing added a comment -

            Daniel Daugherty: Actually it does explain why it works when maintaining the reference to the process causes the bug to disappear. My pull requests works because it does not monitor the presence of the file, but it waits until the process is terminated and only after the process is no longer present is looks for the file. This way there is no file checking done as long as the process is active and the batch file is able to create the result file without any problems.

            Show
            nitram Martin Karing added a comment - Daniel Daugherty : Actually it does explain why it works when maintaining the reference to the process causes the bug to disappear. My pull requests works because it does not monitor the presence of the file, but it waits until the process is terminated and only after the process is no longer present is looks for the file. This way there is no file checking done as long as the process is active and the batch file is able to create the result file without any problems.
            Hide
            amuniz Antonio Muñiz added a comment -

            This way there is no file checking done as long as the process is active

            Right. As I noted in the PR, keeping that reference is just what this plugin is trying to avoid.

            I think the GC theory is probably the culprit. Perhaps if we keep a transient instance field private transient Launcher.ProcStarter ps in WindowsBatchScript it is prevented to be collected. I'm currently testing this option.

            Show
            amuniz Antonio Muñiz added a comment - This way there is no file checking done as long as the process is active Right. As I noted in the PR, keeping that reference is just what this plugin is trying to avoid. I think the GC theory is probably the culprit. Perhaps if we keep a transient instance field private transient Launcher.ProcStarter ps in WindowsBatchScript it is prevented to be collected. I'm currently testing this option.
            Hide
            amuniz Antonio Muñiz added a comment -

            No, it does not work.

            Show
            amuniz Antonio Muñiz added a comment - No, it does not work.
            Hide
            nitram Martin Karing added a comment -

            This issue is really annoying. I tried to track it with the SysInternals Process Monitor. As soon as the monitor runs the issue does not happen any more.

            Also I tried to alter the wrapping batch file to check if the file was created after writing the error code and if not try again. This does not resolve the issue. It seems like the batch file "sees" the jenkins-result.txt during it's execution.

            Show
            nitram Martin Karing added a comment - This issue is really annoying. I tried to track it with the SysInternals Process Monitor. As soon as the monitor runs the issue does not happen any more. Also I tried to alter the wrapping batch file to check if the file was created after writing the error code and if not try again. This does not resolve the issue. It seems like the batch file "sees" the jenkins-result.txt during it's execution.
            Hide
            nitram Martin Karing added a comment -

            Okay, the reason why the loop in the batch file did not work is that something is killing the entire batch file structure before it finishes. I have no idea why this happens and I can't track it because process monitor seems to do something that stops this from happening.

            I was able to get it running by changing the script so the execution of the wrapper is done by a additional "start" command. This cause command line windows to popup all over my desktop, but it allowed the entire thing to execute properly. This approach has the massive disadvantage that there is no way to terminate the script process in case the script itself hangs or something like this because it runs fully detached and there is no reference to the process.

            On the other hand the purpose of this plugin is to allow scripts to run across restarts of Jenkins. So it has to run as a detached process from Jenkins, so the JVM doesn't tear it down along with it, but we need a serializable reference to the process so it's possible to locate it again. Just in case it is required to terminate it after a Jenkins reboot.

            Show
            nitram Martin Karing added a comment - Okay, the reason why the loop in the batch file did not work is that something is killing the entire batch file structure before it finishes. I have no idea why this happens and I can't track it because process monitor seems to do something that stops this from happening. I was able to get it running by changing the script so the execution of the wrapper is done by a additional " start " command. This cause command line windows to popup all over my desktop, but it allowed the entire thing to execute properly. This approach has the massive disadvantage that there is no way to terminate the script process in case the script itself hangs or something like this because it runs fully detached and there is no reference to the process. On the other hand the purpose of this plugin is to allow scripts to run across restarts of Jenkins. So it has to run as a detached process from Jenkins, so the JVM doesn't tear it down along with it, but we need a serializable reference to the process so it's possible to locate it again. Just in case it is required to terminate it after a Jenkins reboot.
            Hide
            flyingosprey KK G added a comment -

            It's related to https://issues.jenkins-ci.org/browse/JENKINS-33164. Just attach a simple repro from that bug:
            Pipeline code is:
            node('master') {
            for(int i=0; i < 100; ++i)
            { bat('echo "Hello from batch file."' + i.toString()) }}
            Click "build now" 5 times.
            All 5 jobs got stuck on windows OS. Please help. Thanks.

            Show
            flyingosprey KK G added a comment - It's related to https://issues.jenkins-ci.org/browse/JENKINS-33164 . Just attach a simple repro from that bug: Pipeline code is: node('master') { for(int i=0; i < 100; ++i) { bat('echo "Hello from batch file."' + i.toString()) }} Click "build now" 5 times. All 5 jobs got stuck on windows OS. Please help. Thanks.
            amuniz Antonio Muñiz made changes -
            Link This issue is related to JENKINS-33164 [ JENKINS-33164 ]
            amuniz Antonio Muñiz made changes -
            Link This issue is related to JENKINS-33749 [ JENKINS-33749 ]
            Hide
            amuniz Antonio Muñiz added a comment -

            I was not able to reproduce the issue in a debug session and did not manage to diagnose why jenkins-wrapper.bat is not fully executed (so jenkins-result.txt is not created) and the bat step never finishes. If someone with more Windows background can throw some light here, it would be great.

            Perhaps the additional start command proposed by Martin Karing is the less ugly fix, what others think? Jesse Glick ?

            In the meantime, the workaround is to use a build agent (other than master), even being in the same physical machine.

            Show
            amuniz Antonio Muñiz added a comment - I was not able to reproduce the issue in a debug session and did not manage to diagnose why jenkins-wrapper.bat is not fully executed (so jenkins-result.txt is not created) and the bat step never finishes. If someone with more Windows background can throw some light here, it would be great. Perhaps the additional start command proposed by Martin Karing is the less ugly fix, what others think? Jesse Glick ? In the meantime, the workaround is to use a build agent (other than master), even being in the same physical machine.
            Hide
            nitram Martin Karing added a comment -

            I was able to track down that the batch process is forcefully terminated.
            If you run the wrapper batch by hand and close the command line window before the process finishes you get exactly the same behaviour. The main and the child processes are terminated and no files are created. The only thing that is attached to the command line actually is java. So the termination has to come from there.

            The thing I wonder is: Can all this even work across a jenkins restart? If Java terminates its child processes this would kill no command line execution no matter what.

            I think there are solutions to work around this using powershell or the scripting host. But those may be blocked on the host system.

            Show
            nitram Martin Karing added a comment - I was able to track down that the batch process is forcefully terminated. If you run the wrapper batch by hand and close the command line window before the process finishes you get exactly the same behaviour. The main and the child processes are terminated and no files are created. The only thing that is attached to the command line actually is java. So the termination has to come from there. The thing I wonder is: Can all this even work across a jenkins restart? If Java terminates its child processes this would kill no command line execution no matter what. I think there are solutions to work around this using powershell or the scripting host. But those may be blocked on the host system.
            Hide
            flyingosprey KK G added a comment -

            Just gave a try. "workaround is to use a build agent (other than master), even being in the same physical machine." really works! Thanks. At least, I can proceed.

            BTW, I notice that for the same machine, master node has info, "Windows Server 2012 R2 (x86)", while client has info, "Windows Server 2012 R2 (amd64)". I doubt if the bug trigger corner case related to machine architecture.

            Show
            flyingosprey KK G added a comment - Just gave a try. "workaround is to use a build agent (other than master), even being in the same physical machine." really works! Thanks. At least, I can proceed. BTW, I notice that for the same machine, master node has info, "Windows Server 2012 R2 (x86)", while client has info, "Windows Server 2012 R2 (amd64)". I doubt if the bug trigger corner case related to machine architecture.
            flyingosprey KK G made changes -
            Environment Jenkins 2.0RC1
            Windows 2008r2, Windows 2012r2, Windows 7
            Jenkins 2.0RC1, Jenkins LTS
            Windows 2008r2, Windows 2012r2, Windows 7
            flyingosprey KK G made changes -
            Environment Jenkins 2.0RC1, Jenkins LTS
            Windows 2008r2, Windows 2012r2, Windows 7
            Jenkins 2.0RC1, Jenkins LTS
            Windows 2008r2, Windows 2012r2, Windows 7, Window 10
            flyingosprey KK G made changes -
            Environment Jenkins 2.0RC1, Jenkins LTS
            Windows 2008r2, Windows 2012r2, Windows 7, Window 10
            Jenkins 2.0RC1, Jenkins LTS
            Windows 2008r2, Windows 2012r2, Windows 7, Windows 10
            flyingosprey KK G made changes -
            Environment Jenkins 2.0RC1, Jenkins LTS
            Windows 2008r2, Windows 2012r2, Windows 7, Windows 10
            Jenkins 2.0RC1, Jenkins 1.6* LTS
            Windows 2008r2, Windows 2012r2, Windows 7, Windows 10
            jglick Jesse Glick made changes -
            Component/s workflow-plugin [ 18820 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 21 (Web Link)" [ 14227 ]
            Hide
            jglick Jesse Glick added a comment -

            Can all this even work across a jenkins restart?

            If you are using a master executor, not generally. (On Unix, it works under some conditions but not others.)

            You are strongly recommended to use an agent rather than master executors in general. In particular, if you have any kind of layered security on your Jenkins installation—whereby people configuring jobs (or permitted to edit build scripts in SCM) are not Jenkins administrators—you must not have a master executor, or any pretense at security is gone. Even if only one physical computer is available, you must configure a separate service account for builds.

            All that said, if the problem can be fixed—or at least clearly diagnosed and reported—without breaking anything for the more general use case of an agent on another machine, obviously we want to apply a fix.

            Show
            jglick Jesse Glick added a comment - Can all this even work across a jenkins restart? If you are using a master executor, not generally. (On Unix, it works under some conditions but not others.) You are strongly recommended to use an agent rather than master executors in general. In particular, if you have any kind of layered security on your Jenkins installation—whereby people configuring jobs (or permitted to edit build scripts in SCM) are not Jenkins administrators—you must not have a master executor, or any pretense at security is gone. Even if only one physical computer is available, you must configure a separate service account for builds. All that said, if the problem can be fixed—or at least clearly diagnosed and reported—without breaking anything for the more general use case of an agent on another machine, obviously we want to apply a fix.
            jglick Jesse Glick made changes -
            Labels 2.0 2.0-planned testfest workflow 2.0 2.0-planned testfest windows workflow
            Hide
            luebbe Lübbe Onken added a comment -

            This hanging batch bug has bitten me heavily too. Failing batch jobs always terminated, successful jobs never did.
            A working solution for me is to explicitely return an exit code from any batch call. I'm on Windows 7 Professional.

            So:
            {{
            echo 'Successful step'
            bat '''dir
            exit /B %ERRORLEVEL%'''

            echo 'Failing step'
            bat '''find /c "_no.file" "_no.file"
            exit /B %ERRORLEVEL%'''

            echo 'Never execute step'
            bat '''dir"
            exit /B %ERRORLEVEL%'''
            }}

            successfully terminates step one and returns from the batch execution, the second step fails the build and the third step never gets executed.

            Can somebody please confirm that this solution works for them too?

            Show
            luebbe Lübbe Onken added a comment - This hanging batch bug has bitten me heavily too. Failing batch jobs always terminated, successful jobs never did. A working solution for me is to explicitely return an exit code from any batch call. I'm on Windows 7 Professional. So: {{ echo 'Successful step' bat '''dir exit /B %ERRORLEVEL%''' echo 'Failing step' bat '''find /c "_no.file" "_no.file" exit /B %ERRORLEVEL%''' echo 'Never execute step' bat '''dir" exit /B %ERRORLEVEL%''' }} successfully terminates step one and returns from the batch execution, the second step fails the build and the third step never gets executed. Can somebody please confirm that this solution works for them too?
            Hide
            luebbe Lübbe Onken added a comment - - edited

            Looks like I was too optimistic. The solution always worked with short running batch jobs, like dir, but it didn't with long running jobs, like a NAnt build.
            Is it possible that there is a race condition? Some state is checked very quickly after a task is started. A simple "dir" is quick enough to deliver the result in time and a slower task isn't?

            Show
            luebbe Lübbe Onken added a comment - - edited Looks like I was too optimistic. The solution always worked with short running batch jobs, like dir, but it didn't with long running jobs, like a NAnt build. Is it possible that there is a race condition? Some state is checked very quickly after a task is started. A simple "dir" is quick enough to deliver the result in time and a slower task isn't?
            Hide
            saucistophe Christophe Carpentier added a comment -

            That would explain the iconsistent results during my tests. Weird.
            Anyway, I've encountered this both on failed and successful jobs.

            Show
            saucistophe Christophe Carpentier added a comment - That would explain the iconsistent results during my tests. Weird. Anyway, I've encountered this both on failed and successful jobs.
            amuniz Antonio Muñiz made changes -
            Status In Progress [ 3 ] Open [ 1 ]
            Hide
            erates Rens Hoskens added a comment - - edited

            Have the same issue on Windows Server 2008 R2. Hope it will get fixed soon (or a plain maven command would be usefull as well)

            node {
                mvn 'clean package -DskipTests=true'
            }
            
            def mvn(args) {
                bat "${tool 'Maven 3.3.9'}/bin/mvn ${args}"
            }
            
            Show
            erates Rens Hoskens added a comment - - edited Have the same issue on Windows Server 2008 R2. Hope it will get fixed soon (or a plain maven command would be usefull as well) node { mvn 'clean package -DskipTests=true' } def mvn(args) { bat "${tool 'Maven 3.3.9'}/bin/mvn ${args}" }
            Hide
            sonneveldsmartward Nick Sonneveld added a comment - - edited

            I have mentioned in other related tickets but I just want to point out that I have seen this behaviour with a linux master and multiple windows agents (with 5-10 executors on each). You could try this example code. I haven't tested it but it's similar to our Jenkinsfile where we have branches doing chunks of a test. Cancelling the job in the middle of execution sometimes puts the agents in a weird state too.

            def branches = [:]
            
            for (int i = 0; i < 64; i++) {
            	def id = "branch-${i}"
            	branches[id] = {
            		node ('windows') {
            			for (int j = 0; j < 8; j++) {
            			    bat 'ping 127.0.0.1 -n 10' 
            			}
            		}
            	}
            }
            
            parallel branches
            
            Show
            sonneveldsmartward Nick Sonneveld added a comment - - edited I have mentioned in other related tickets but I just want to point out that I have seen this behaviour with a linux master and multiple windows agents (with 5-10 executors on each). You could try this example code. I haven't tested it but it's similar to our Jenkinsfile where we have branches doing chunks of a test. Cancelling the job in the middle of execution sometimes puts the agents in a weird state too. def branches = [:] for ( int i = 0; i < 64; i++) { def id = "branch-${i}" branches[id] = { node ( 'windows' ) { for ( int j = 0; j < 8; j++) { bat 'ping 127.0.0.1 -n 10' } } } } parallel branches
            Hide
            gijskuijer Gijs Kuijer added a comment -

            I have the same exact issue on a windows server 2012 R2 with a Jenkins 2.1 installations and all plugins fully updated.
            I have installed the Github organization folder plugin to scan my organization.

            My jenkins file has a simple batch job to use MSBuild to build our project and a batch job for analysis of sonar.
            The job randomly hangs after one of these two jobs.

            Is there any progress on this issue?

            Show
            gijskuijer Gijs Kuijer added a comment - I have the same exact issue on a windows server 2012 R2 with a Jenkins 2.1 installations and all plugins fully updated. I have installed the Github organization folder plugin to scan my organization. My jenkins file has a simple batch job to use MSBuild to build our project and a batch job for analysis of sonar. The job randomly hangs after one of these two jobs. Is there any progress on this issue?
            Hide
            maaltan maaltan natlaam added a comment -

            I have this issue also. my batch invocation is:

            bat '''
            call %BUILD_CONFIG_PATH%
            setenv.cmd
            perl <custom build manager script that typically runs for 90 minutes>
            '''

            This has broken the job completely. I cannot terminate the job, nor can i start another (this is an incremental build with a fixed workspace location so i dont want to run concurrently). If i restart jenkins the job restarts and immediately hangs again.

            Are there any workaround short of full uninstall/reinstall of jenkins to recover this job setup?

            Show
            maaltan maaltan natlaam added a comment - I have this issue also. my batch invocation is: bat ''' call %BUILD_CONFIG_PATH% setenv.cmd perl <custom build manager script that typically runs for 90 minutes> ''' This has broken the job completely. I cannot terminate the job, nor can i start another (this is an incremental build with a fixed workspace location so i dont want to run concurrently). If i restart jenkins the job restarts and immediately hangs again. Are there any workaround short of full uninstall/reinstall of jenkins to recover this job setup?
            Hide
            ttt43ttt Wilson Tian added a comment - - edited

            I encounter this issue too. I'm running a maven job using

            bat "${mavenHome}\\bin\\mvn clean package"

            . But the job always hangs at last and never exits.
            Is there any workaround?

            Show
            ttt43ttt Wilson Tian added a comment - - edited I encounter this issue too. I'm running a maven job using bat "${mavenHome}\\bin\\mvn clean package " . But the job always hangs at last and never exits. Is there any workaround?
            Hide
            maaltan maaltan natlaam added a comment -

            I found a workaround (better than reinstalling at least).

            1. Shutdown jenkins service
            2. go to <install>/jobs/<jobname>/ and delete the <jobnumber> folder.
            3. Restart jenkins.

            There is probably a flag somewhere in that folder you can set to prevent the job from "restarting" after restart.

            Since ive hit this bug about 80% of the runs I've tried so far. this workaround is unusable in any kind of production environment. At least you can run the job again though. I guess another workaround would be allow multiple instances of the job to run and clean up the zombies once a day or something?

            Also, it seems this happens more when i view the console output via jenkins ui while the job is running.

            Show
            maaltan maaltan natlaam added a comment - I found a workaround (better than reinstalling at least). 1. Shutdown jenkins service 2. go to <install>/jobs/<jobname>/ and delete the <jobnumber> folder. 3. Restart jenkins. There is probably a flag somewhere in that folder you can set to prevent the job from "restarting" after restart. Since ive hit this bug about 80% of the runs I've tried so far. this workaround is unusable in any kind of production environment. At least you can run the job again though. I guess another workaround would be allow multiple instances of the job to run and clean up the zombies once a day or something? Also, it seems this happens more when i view the console output via jenkins ui while the job is running.
            Hide
            sonneveldsmartward Nick Sonneveld added a comment - - edited

            Another workaround that doesn't involve deleting jobs (but also isn't a long term solution) is to realise that batch steps create two batch files in the @tmp directory (which is relative to where batch is run, so it might be in the workspace if you've changed the directory, or just outside it): a jenkins-main.bat and a jenkins-wrap.bat. The main bat file contains your commands. The wrap bat file will run the main bat, pipe output to a log file and finally writes a result file.

            The bug concerns the wrap bat file not completing so the result file is never written. You can search for the file and run the final line manually (looks like echo %errorlevel% > ...\jenkins-result.txt), or run the wrap batch file again if you don't mind it performing the same operation again.

            Show
            sonneveldsmartward Nick Sonneveld added a comment - - edited Another workaround that doesn't involve deleting jobs (but also isn't a long term solution) is to realise that batch steps create two batch files in the @tmp directory (which is relative to where batch is run, so it might be in the workspace if you've changed the directory, or just outside it): a jenkins-main.bat and a jenkins-wrap.bat. The main bat file contains your commands. The wrap bat file will run the main bat, pipe output to a log file and finally writes a result file. The bug concerns the wrap bat file not completing so the result file is never written. You can search for the file and run the final line manually (looks like echo %errorlevel% > ...\jenkins-result.txt), or run the wrap batch file again if you don't mind it performing the same operation again.
            Hide
            maaltan maaltan natlaam added a comment -

            cmd /c ""<script> > ".../jenkins-log.txt"" 2>&1
            echo %ERRORLEVEL% > "...\..@tmp\durable-cf7a3b23\jenkins-result.txt"

            Perhaps using "call" will work better. that will leverage the current cmd shell to execute the batch. Ive found it more stable than launching a second cmd from a batch file. "start" is another option. That gives you a subshell that is detached from the main shell. There are parameters that prevent that though. other bonuses of start is ability to set process priorities,etc.

            If you know of the jar/class i need to hack off hand to make this change, i'll try to give it a shot today.

            I am probably going to end up grabbing the jenkins source at some point but that will probably be later next week if then.

            Show
            maaltan maaltan natlaam added a comment - cmd /c ""<script> > ".../jenkins-log.txt"" 2>&1 echo %ERRORLEVEL% > "...\..@tmp\durable-cf7a3b23\jenkins-result.txt" Perhaps using "call" will work better. that will leverage the current cmd shell to execute the batch. Ive found it more stable than launching a second cmd from a batch file. "start" is another option. That gives you a subshell that is detached from the main shell. There are parameters that prevent that though. other bonuses of start is ability to set process priorities,etc. If you know of the jar/class i need to hack off hand to make this change, i'll try to give it a shot today. I am probably going to end up grabbing the jenkins source at some point but that will probably be later next week if then.
            Hide
            sonneveldsmartward Nick Sonneveld added a comment -

            There is a pull request being worked on by Martin Karing you might want to look at and comment on. Link is attached to this issue https://github.com/jenkinsci/durable-task-plugin/pull/21

            Show
            sonneveldsmartward Nick Sonneveld added a comment - There is a pull request being worked on by Martin Karing you might want to look at and comment on. Link is attached to this issue https://github.com/jenkinsci/durable-task-plugin/pull/21
            Hide
            maaltan maaltan natlaam added a comment -

            First off, i found a better workaround. First go to the console screen for the job. click the abort button in upper right area. scroll to bottom and wait about 10 seconds. You will see a link allowing you to force kill the job.

            Started At: 05-06-2016 20:00:08
            Ended At: 05-06-2016 20:02:19
            Build Lasted: 2 minutes 10 seconds
            Highest Error Code: 0
            <hang here>
            Aborted by admin
            Sending interrupt signal to process
            Click here to forcibly terminate running steps
            Terminating bat
            [Pipeline] }
            [Pipeline] // node
            [Pipeline] End of Pipeline
            Finished: ABORTED

            Unfortunately I'm seeing an almost 100% chance of hangs on my machine so still pretty useless.

            -------------------------

            I acquired Karing's code and tested it. it terminates my bat build steps after about 3-5 seconds no matter what the state is. I didnt dig too much into that.

            I reverted to baseline and tried my suggestions. None of them work. In fact, i can't prove that the script call from jenkins-wrap.bat ever returns...

            here is my current attempt at jenkins-wrap.bat:
            cmd /c ""...\jenkins-main.bat"" > "...\jenkins-log.txt" 2>&1
            :retry
            echo writing jenkins.results >> "...\jenkins-log.txt"
            echo %ERRORLEVEL% > "...\jenkins-result.txt"
            if not exist "...\jenkins-result.txt" goto retry

            It is supposed to jackhammer that results file until it is created. I see no "writing jenkins.results" in the logs, therefore the wrapper script is terminating early. The same thing happens if i replace cmd /c with call or start.
            (note: "..." is a placeholder for my real paths not some kind of relative path thing. sorry for confusion.)

            Show
            maaltan maaltan natlaam added a comment - First off, i found a better workaround. First go to the console screen for the job. click the abort button in upper right area. scroll to bottom and wait about 10 seconds. You will see a link allowing you to force kill the job. Started At: 05-06-2016 20:00:08 Ended At: 05-06-2016 20:02:19 Build Lasted: 2 minutes 10 seconds Highest Error Code: 0 <hang here> Aborted by admin Sending interrupt signal to process Click here to forcibly terminate running steps Terminating bat [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: ABORTED Unfortunately I'm seeing an almost 100% chance of hangs on my machine so still pretty useless. ------------------------- I acquired Karing's code and tested it. it terminates my bat build steps after about 3-5 seconds no matter what the state is. I didnt dig too much into that. I reverted to baseline and tried my suggestions. None of them work. In fact, i can't prove that the script call from jenkins-wrap.bat ever returns... here is my current attempt at jenkins-wrap.bat: cmd /c ""...\jenkins-main.bat"" > "...\jenkins-log.txt" 2>&1 :retry echo writing jenkins.results >> "...\jenkins-log.txt" echo %ERRORLEVEL% > "...\jenkins-result.txt" if not exist "...\jenkins-result.txt" goto retry It is supposed to jackhammer that results file until it is created. I see no "writing jenkins.results" in the logs, therefore the wrapper script is terminating early. The same thing happens if i replace cmd /c with call or start. (note: "..." is a placeholder for my real paths not some kind of relative path thing. sorry for confusion.)
            vdragon Camille Baud made changes -
            Comment [ Sucessfulyl reproduced with:
            {code:java}
            node("master") {
              for( int i = 0; i < 100; ++i ) {
                echo "i=${i}"
                bat "ping JENKINSSRVMGR"
              }
            }
            {code}
            ran multiple times; ]
            Hide
            teilo James Nord added a comment -

            I believe I have a 100% reproducible test-case for this issue

            It seems that in the case that the parent process has been killed (e.g. the slave dies) then all though the script terminates successfully and the wrapper terminates successfully (checked with process monitor) there is no attempt to create the result file.
            where the parent process has not been killed I never see this issue.

            All I needed to do to fix the issue I was observing is add @echo off as the first line of the wrapper script. Basically I believe it is trying to echo the commands to be run before running the commands and as there is no longer anything consuming the wrappers input/output when echoing the command it is doomed to fail (but strangely not with an exit code that implies something died!??!

            Show
            teilo James Nord added a comment - I believe I have a 100% reproducible test-case for this issue It seems that in the case that the parent process has been killed (e.g. the slave dies) then all though the script terminates successfully and the wrapper terminates successfully (checked with process monitor) there is no attempt to create the result file. where the parent process has not been killed I never see this issue. All I needed to do to fix the issue I was observing is add @echo off as the first line of the wrapper script. Basically I believe it is trying to echo the commands to be run before running the commands and as there is no longer anything consuming the wrappers input/output when echoing the command it is doomed to fail (but strangely not with an exit code that implies something died!??!
            Hide
            vyazelenko Dmitry Vyazelenko added a comment -

            I'm also having a build hang with simple pipeline that uses bat script to run Gradle tasks:

            node {
                timeout(time: 10, unit: 'MINUTES') {
                    timestamps {
                        stage 'Checkout'
                        git ...
                        
                        stage 'Tests'
                        bat 'gradlew test'
                        step([$class: 'JUnitResultArchiver', testResults: 'build/test-results/*.xml'])
                    }
                }
            }
            
            Show
            vyazelenko Dmitry Vyazelenko added a comment - I'm also having a build hang with simple pipeline that uses bat script to run Gradle tasks: node { timeout(time: 10, unit: 'MINUTES' ) { timestamps { stage 'Checkout' git ... stage 'Tests' bat 'gradlew test' step([$class: 'JUnitResultArchiver' , testResults: 'build/test-results/*.xml' ]) } } }
            vyazelenko Dmitry Vyazelenko made changes -
            Priority Critical [ 2 ] Blocker [ 1 ]
            Hide
            teilo James Nord added a comment - - edited

            For anyone observing the issue you can try installing the build from PR24 or PR21 and see if this resolves your issue. (I would start with PR24 first as it is a much smaller change, but then I am biased!)

            Show
            teilo James Nord added a comment - - edited For anyone observing the issue you can try installing the build from PR24 or PR21 and see if this resolves your issue. (I would start with PR24 first as it is a much smaller change, but then I am biased!)
            saucistophe Christophe Carpentier made changes -
            Attachment How.png [ 32760 ]
            Hide
            saucistophe Christophe Carpentier added a comment -

            PR24 fixes my particular issue.

            Show
            saucistophe Christophe Carpentier added a comment - PR24 fixes my particular issue.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: James Nord
            Path:
            src/main/java/org/jenkinsci/plugins/durabletask/WindowsBatchScript.java
            http://jenkins-ci.org/commit/durable-task-plugin/d156ebfbcdb70666757ff48127d0597bd5891a61
            Log:
            JENKINS-34150 Fixes my observed issue.

            I have a reproducable tests case in a propratary implementation using this
            code that is 100% reproducable.
            The simple "@echo off" fixes the failing test for me.

            It seems that in the case that the parent process has been killed (e.g.
            the slave dies) then all though the script terminates successfully and the
            wrapper terminates successfully (checked with process monitor) there is no
            attempt to create the result file.
            where the parent process has not been killed I never see this issue.
            All I needed to do to fix the issue I was observing is add @echo off as
            the first line of the wrapper script. Basically I believe it is trying to
            echo the commands to be run before running the commands and as there is no
            longer anything consuming the wrappers input/output when echoing the
            command it is doomed to fail (but strangely not with an exit code that
            implies something died!??!

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: James Nord Path: src/main/java/org/jenkinsci/plugins/durabletask/WindowsBatchScript.java http://jenkins-ci.org/commit/durable-task-plugin/d156ebfbcdb70666757ff48127d0597bd5891a61 Log: JENKINS-34150 Fixes my observed issue. I have a reproducable tests case in a propratary implementation using this code that is 100% reproducable. The simple "@echo off" fixes the failing test for me. It seems that in the case that the parent process has been killed (e.g. the slave dies) then all though the script terminates successfully and the wrapper terminates successfully (checked with process monitor) there is no attempt to create the result file. where the parent process has not been killed I never see this issue. All I needed to do to fix the issue I was observing is add @echo off as the first line of the wrapper script. Basically I believe it is trying to echo the commands to be run before running the commands and as there is no longer anything consuming the wrappers input/output when echoing the command it is doomed to fail (but strangely not with an exit code that implies something died!??!
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/org/jenkinsci/plugins/durabletask/WindowsBatchScript.java
            http://jenkins-ci.org/commit/durable-task-plugin/8a2537cf28c826ad91d4ce14cd657712364c8953
            Log:
            Merge pull request #24 from jtnord/jenkins-34150

            [FIXED JENKINS-34150] Fixes my observed issue.

            Compare: https://github.com/jenkinsci/durable-task-plugin/compare/0f09bb54a1b7...8a2537cf28c8

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/durabletask/WindowsBatchScript.java http://jenkins-ci.org/commit/durable-task-plugin/8a2537cf28c826ad91d4ce14cd657712364c8953 Log: Merge pull request #24 from jtnord/jenkins-34150 [FIXED JENKINS-34150] Fixes my observed issue. Compare: https://github.com/jenkinsci/durable-task-plugin/compare/0f09bb54a1b7...8a2537cf28c8
            scm_issue_link SCM/JIRA link daemon made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            svanoort Sam Van Oort added a comment -

            Most important single-line-of-code change I've seen recently.

            Show
            svanoort Sam Van Oort added a comment - Most important single-line-of-code change I've seen recently.
            Hide
            junkimu Junichi Kimura added a comment - - edited

            The test job below used to consistently get stuck at the 8mins stage. After upgrading the Durable Task Plugin to 1.10, the job passed successfully (once so far).

            node {
               stage '1min'
               bat '''
            @ECHO OFF
            FOR /L %%A IN (0,1,60) DO (
              ECHO %%A
              PING 192.0.2.1 -n 1 -w 1000 >NUL
            )
            EXIT /B 0
            '''
               stage '5mins'
               bat '''
            @ECHO OFF
            FOR /L %%A IN (0,1,300) DO (
              ECHO %%A
              PING 192.0.2.1 -n 1 -w 1000 >NUL
            )
            EXIT /B 0
            '''
               stage '8mins'
               bat '''
            @ECHO OFF
            FOR /L %%A IN (0,1,480) DO (
              ECHO %%A
              PING 192.0.2.1 -n 1 -w 1000 >NUL
            )
            EXIT /B 0
            '''
               stage '10mins'
               bat '''
            @ECHO OFF
            FOR /L %%A IN (0,1,600) DO (
              ECHO %%A
              PING 192.0.2.1 -n 1 -w 1000 >NUL
            )
            EXIT /B 0
            '''
               stage '30mins'
               bat '''
            @ECHO OFF
            FOR /L %%A IN (0,1,1800) DO (
              ECHO %%A
              PING 192.0.2.1 -n 1 -w 1000 >NUL
            )
            EXIT /B 0
            '''
               stage '1hr'
               bat '''
            @ECHO OFF
            FOR /L %%A IN (0,1,3600) DO (
              ECHO %%A
              PING 192.0.2.1 -n 1 -w 1000 >NUL
            )
            EXIT /B 0
            '''
            }
            
            Show
            junkimu Junichi Kimura added a comment - - edited The test job below used to consistently get stuck at the 8mins stage. After upgrading the Durable Task Plugin to 1.10, the job passed successfully (once so far). node { stage '1min' bat ''' @ECHO OFF FOR /L %%A IN (0,1,60) DO ( ECHO %%A PING 192.0.2.1 -n 1 -w 1000 >NUL ) EXIT /B 0 ''' stage '5mins' bat ''' @ECHO OFF FOR /L %%A IN (0,1,300) DO ( ECHO %%A PING 192.0.2.1 -n 1 -w 1000 >NUL ) EXIT /B 0 ''' stage '8mins' bat ''' @ECHO OFF FOR /L %%A IN (0,1,480) DO ( ECHO %%A PING 192.0.2.1 -n 1 -w 1000 >NUL ) EXIT /B 0 ''' stage '10mins' bat ''' @ECHO OFF FOR /L %%A IN (0,1,600) DO ( ECHO %%A PING 192.0.2.1 -n 1 -w 1000 >NUL ) EXIT /B 0 ''' stage '30mins' bat ''' @ECHO OFF FOR /L %%A IN (0,1,1800) DO ( ECHO %%A PING 192.0.2.1 -n 1 -w 1000 >NUL ) EXIT /B 0 ''' stage '1hr' bat ''' @ECHO OFF FOR /L %%A IN (0,1,3600) DO ( ECHO %%A PING 192.0.2.1 -n 1 -w 1000 >NUL ) EXIT /B 0 ''' }
            Hide
            gijskuijer Gijs Kuijer added a comment -

            Great job! Totally solves my issues!

            Show
            gijskuijer Gijs Kuijer added a comment - Great job! Totally solves my issues!
            Hide
            dpd_30 Daniel Daugherty added a comment -

            Initial testing here show issue is resolved
            Just getting back from vacation so nice to see this resolved. Thanks all for the good work.

            Show
            dpd_30 Daniel Daugherty added a comment - Initial testing here show issue is resolved Just getting back from vacation so nice to see this resolved. Thanks all for the good work.
            Hide
            rufer7 Marc Rufer added a comment -

            Thanks guys. Great job. Updating to the newest version of the durable-task-plugin solved the issue for me as well!

            Show
            rufer7 Marc Rufer added a comment - Thanks guys. Great job. Updating to the newest version of the durable-task-plugin solved the issue for me as well!
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 170200 ] JNJira + In-Review [ 198805 ]
            abayer Andrew Bayer made changes -
            Labels 2.0 2.0-planned testfest windows workflow 2.0 2.0-planned pipeline testfest windows workflow
            abayer Andrew Bayer made changes -
            Labels 2.0 2.0-planned pipeline testfest windows workflow 2.0 2.0-planned pipeline testfest windows
            hrmpw Patrick Wolf made changes -
            Link This issue is related to JENKINS-33904 [ JENKINS-33904 ]
            hrmpw Patrick Wolf made changes -
            Link This issue is related to JENKINS-33456 [ JENKINS-33456 ]
            hrmpw Patrick Wolf made changes -
            Link This issue is related to JENKINS-32000 [ JENKINS-32000 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 25 (Web Link)" [ 15197 ]
            Hide
            lidl Sven Brosi added a comment -

            Hello,

            since the version 1.20 of the Durable Task plugin we encounter the same behavior described above. 

            Our worker nodes have Win2012 and Win2018 environment.

            A quick solution was to downgrade the version of this plugin to 1.18.

            Then everything works again in the DSL Jenkinsfiles with bat(ch) step.

             

             

             

             

            Show
            lidl Sven Brosi added a comment - Hello, since the version 1.20 of the Durable Task plugin we encounter the same behavior described above.  Our worker nodes have Win2012 and Win2018 environment. A quick solution was to downgrade the version of this plugin to 1.18. Then everything works again in the DSL Jenkinsfiles with bat(ch) step.        
            lidl Sven Brosi made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            Hide
            stevenfoster Steven Foster added a comment -

            I'm encountering the same issue after updating to 1.20 from 1.18

            Show
            stevenfoster Steven Foster added a comment - I'm encountering the same issue after updating to 1.20 from 1.18
            svanoort Sam Van Oort made changes -
            Attachment durable-task.hpi [ 41773 ]
            Hide
            svanoort Sam Van Oort added a comment -

            Steven Foster Does it work if you set 'returnStdOut: true'? If so, I have an attached hotfix for you to try – please let us know if this resolves it. durable-task.hpi

            Show
            svanoort Sam Van Oort added a comment - Steven Foster Does it work if you set 'returnStdOut: true'? If so, I have an attached hotfix for you to try – please let us know if this resolves it. durable-task.hpi
            Hide
            svanoort Sam Van Oort added a comment -

            Sven Brosi Does it work if you set 'returnStdOut: true', and if so, please try the attached hotfix and let us know if this resolves it.

            Show
            svanoort Sam Van Oort added a comment - Sven Brosi Does it work if you set 'returnStdOut: true', and if so, please try the attached hotfix and let us know if this resolves it.
            Hide
            stevenfoster Steven Foster added a comment -

            returnStdout: true has the same result. confirmed the process is finished on the machine.

            Show
            stevenfoster Steven Foster added a comment - returnStdout: true has the same result. confirmed the process is finished on the machine.
            Hide
            svanoort Sam Van Oort added a comment -

            Steven Foster In the control directory in the build agent's workspace for this job (where there's a jenkins-main.bat and jenkins-wrap.bat) do you see a jenkins-result.txt file, a jenkins-result.txt.tmp, or both?

            I'm trying to figure out what actually triggered this because it did not fail any of our unit tests that should have explicitly covered this functionality.

            Show
            svanoort Sam Van Oort added a comment - Steven Foster In the control directory in the build agent's workspace for this job (where there's a jenkins-main.bat and jenkins-wrap.bat) do you see a jenkins-result.txt file, a jenkins-result.txt.tmp, or both? I'm trying to figure out what actually triggered this because it did not fail any of our unit tests that should have explicitly covered this functionality.
            Hide
            stevenfoster Steven Foster added a comment -

            just the .tmp

            Show
            stevenfoster Steven Foster added a comment - just the .tmp
            Hide
            svanoort Sam Van Oort added a comment -

            Steven Foster Okay, that means the "move" operation failed to rename the file, which should basically never happen. Would you be able to hop on #jenkins IRC briefly to discuss (I'm svanoort there)? It should be a quick and trivial fix but since we can't reproduce the issue in our own environment, it would be super-helpful to be able to see a case where it happened.

            Show
            svanoort Sam Van Oort added a comment - Steven Foster Okay, that means the "move" operation failed to rename the file, which should basically never happen. Would you be able to hop on #jenkins IRC briefly to discuss (I'm svanoort there)? It should be a quick and trivial fix but since we can't reproduce the issue in our own environment, it would be super-helpful to be able to see a case where it happened.
            Hide
            lidl Sven Brosi added a comment - - edited

            Sam Van Oort: Can not test it again, as it is a corporate jenkins and i can not  up- and downgrade the plugins on the fly.

            How can i still help you?

            Show
            lidl Sven Brosi added a comment - - edited Sam Van Oort : Can not test it again, as it is a corporate jenkins and i can not  up- and downgrade the plugins on the fly. How can i still help you?
            svanoort Sam Van Oort made changes -
            Attachment durable-task.hpi [ 41776 ]
            Hide
            svanoort Sam Van Oort added a comment -

            Sven Brosi Don't worry about it, Steven Foster was thankfully available to help debug in an environment where this is reproducible (thanks!).

            I'm attaching one more hotfix version for him to try out which should fully resolve the issue. durable-task.hpi

            Show
            svanoort Sam Van Oort added a comment - Sven Brosi Don't worry about it, Steven Foster was thankfully available to help debug in an environment where this is reproducible (thanks!). I'm attaching one more hotfix version for him to try out which should fully resolve the issue. durable-task.hpi
            svanoort Sam Van Oort made changes -
            Link This issue is related to JENKINS-50025 [ JENKINS-50025 ]
            Hide
            svanoort Sam Van Oort added a comment -

            This issue duplicates symptoms of JENKINS-50025 but the root causes are significantly different, so that is being tracked separately.

            Show
            svanoort Sam Van Oort added a comment - This issue duplicates symptoms of JENKINS-50025 but the root causes are significantly different, so that is being tracked separately.
            svanoort Sam Van Oort made changes -
            Link This issue duplicates JENKINS-50025 [ JENKINS-50025 ]
            Hide
            svanoort Sam Van Oort added a comment -

            Sven Brosi Steven Foster I'm closing THIS issue because while the result is the same this has a different cause and is resolved in JENKINS-50025.

            Show
            svanoort Sam Van Oort added a comment - Sven Brosi Steven Foster I'm closing THIS issue because while the result is the same this has a different cause and is resolved in JENKINS-50025 .
            svanoort Sam Van Oort made changes -
            Status Reopened [ 4 ] Closed [ 6 ]
            Resolution Fixed [ 1 ]
            Hide
            svanoort Sam Van Oort added a comment -

            Sven Brosi Steven Foster Released fix after review and testing as durable-task-plugin 1.21

            Show
            svanoort Sam Van Oort added a comment - Sven Brosi Steven Foster Released fix after review and testing as durable-task-plugin 1.21

              People

              • Assignee:
                amuniz Antonio Muñiz
                Reporter:
                dpd_30 Daniel Daugherty
              • Votes:
                18 Vote for this issue
                Watchers:
                34 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: