Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34712

"master is offline" preventing Pipeline from executing

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Our own Jenkins Pipeline projects seem to be getting stuck in this state of "master is offline" when attempting to run on our clusters which have zero executors assigned to the master node.

      It's unclear what, past a service restart, will clear this up

      Steps to reproduce:

      1. Start a pipeline job
      2. forcing the master to run out of storage
      3. shutdown master, clear up storage
      4. restart master, confirm it's up
      5. observe that it is still marked as offline for a long time. 30+ minutes

        Attachments

          Issue Links

            Activity

            rtyler R. Tyler Croy created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Component/s core [ 15593 ]
            Component/s workflow-plugin [ 18820 ]
            Labels workflow
            Assignee Jesse Glick [ jglick ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 170892 ] JNJira + In-Review [ 184080 ]
            abayer Andrew Bayer made changes -
            Labels workflow pipeline workflow
            abayer Andrew Bayer made changes -
            Labels pipeline workflow pipeline
            recampbell Ryan Campbell made changes -
            Description Our own Jenkins Pipeline projects seem to be getting stuck in this state of "master is offline" when attempting to run on our clusters which have zero executors assigned to the master node.

            It's unclear what, past a service restart, will clear this up :(

            h3. Thread Dump

            {code}
            Channel reader thread: trusted-agent-1

            "Channel reader thread: trusted-agent-1" Id=87 Group=main WAITING on com.trilead.ssh2.channel.Channel@78493ab5
            at java.lang.Object.wait(Native Method)
            - waiting on com.trilead.ssh2.channel.Channel@78493ab5
            at java.lang.Object.wait(Object.java:503)
            at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:212)
            at com.trilead.ssh2.channel.Channel$Output.read(Channel.java:127)
            at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:936)
            at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58)
            at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:79)
            at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
            at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
            at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
            at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
            at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            Channel reader thread: trusted-agent-2

            "Channel reader thread: trusted-agent-2" Id=90 Group=main WAITING on com.trilead.ssh2.channel.Channel@67e91886
            at java.lang.Object.wait(Native Method)
            - waiting on com.trilead.ssh2.channel.Channel@67e91886
            at java.lang.Object.wait(Object.java:503)
            at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:212)
            at com.trilead.ssh2.channel.Channel$Output.read(Channel.java:127)
            at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:936)
            at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58)
            at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:79)
            at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
            at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
            at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
            at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
            at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            Computer.threadPoolForRemoting [#8]

            "Computer.threadPoolForRemoting [#8]" Id=132 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@4a44a3c2
            at sun.misc.Unsafe.park(Native Method)
            - waiting on java.util.concurrent.SynchronousQueue$TransferStack@4a44a3c2
            at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
            at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
            at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            Computer.threadPoolForRemoting [#9]

            "Computer.threadPoolForRemoting [#9]" Id=133 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@4a44a3c2
            at sun.misc.Unsafe.park(Native Method)
            - waiting on java.util.concurrent.SynchronousQueue$TransferStack@4a44a3c2
            at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
            at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
            at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            DestroyJavaVM

            "DestroyJavaVM" Id=17 Group=main RUNNABLE

            FilePath.localPool [#4]

            "FilePath.localPool [#4]" Id=137 Group=main RUNNABLE
            at sun.management.ThreadImpl.dumpThreads0(Native Method)
            at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
            at hudson.Functions.getThreadInfos(Functions.java:1200)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:98)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:95)
            at hudson.remoting.LocalChannel$1.call(LocalChannel.java:52)
            at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            Number of locked synchronizers = 1
            - java.util.concurrent.ThreadPoolExecutor$Worker@4dd9587c

            Handling GET /threadDump from 172.31.29.27 : RequestHandlerThread[#4] Jenkins/threadDump.jelly

            "Handling GET /threadDump from 172.31.29.27 : RequestHandlerThread[#4] Jenkins/threadDump.jelly" Id=25 Group=main RUNNABLE
            at java.lang.System.identityHashCode(Native Method)
            at java.io.ObjectOutputStream$HandleTable.hash(ObjectOutputStream.java:2359)
            at java.io.ObjectOutputStream$HandleTable.lookup(ObjectOutputStream.java:2292)
            at java.io.ObjectOutputStream$ReplaceTable.lookup(ObjectOutputStream.java:2398)
            at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1112)
            at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377)
            at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1173)
            at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
            at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:440)
            at java.lang.Throwable.writeObject(Throwable.java:985)
            - locked java.lang.Exception@3bcc975f
            at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
            at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
            at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
            at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
            at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
            at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:440)
            at hudson.remoting.RemoteInvocationHandler.writeObject(RemoteInvocationHandler.java:290)
            at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
            at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
            at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
            at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
            at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
            at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
            at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
            at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
            at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
            at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
            at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
            at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
            at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
            at hudson.remoting.Command.writeTo(Command.java:83)
            at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:43)
            at hudson.remoting.Channel.send(Channel.java:582)
            - locked hudson.remoting.Channel@56434d6
            at hudson.remoting.Request.callAsync(Request.java:205)
            at hudson.remoting.Channel.callAsync(Channel.java:807)
            at hudson.util.RemotingDiagnostics.getThreadDumpAsync(RemotingDiagnostics.java:92)
            at jenkins.model.Jenkins.getAllThreadDumps(Jenkins.java:3073)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
            at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
            at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
            at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
            at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
            at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
            at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
            at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
            at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
            at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
            at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
            at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
            at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
            at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
            at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
            at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
            at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
            at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
            at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
            at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
            at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
            at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
            at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
            at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:92)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
            at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
            at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
            at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
            at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
            at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
            at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
            at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
            at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
            at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
            at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
            at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
            at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
            at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
            at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
            at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
            at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
            at org.eclipse.jetty.server.Server.handle(Server.java:370)
            at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
            at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
            at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
            at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
            at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
            at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
            at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
            at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
            at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            Number of locked synchronizers = 1
            - java.util.concurrent.ThreadPoolExecutor$Worker@5f69743c

            HashSessionScavenger-0

            "HashSessionScavenger-0" Id=10 Group=main TIMED_WAITING on java.util.TaskQueue@3031917d
            at java.lang.Object.wait(Native Method)
            - waiting on java.util.TaskQueue@3031917d
            at java.util.TimerThread.mainLoop(Timer.java:552)
            at java.util.TimerThread.run(Timer.java:505)

            Jenkins cron thread

            "Jenkins cron thread" Id=18 Group=main WAITING on java.util.TaskQueue@5fdde855
            at java.lang.Object.wait(Native Method)
            - waiting on java.util.TaskQueue@5fdde855
            at java.lang.Object.wait(Object.java:503)
            at java.util.TimerThread.mainLoop(Timer.java:526)
            at java.util.TimerThread.run(Timer.java:505)

            Jenkins UDP 33848 monitoring thread

            "Jenkins UDP 33848 monitoring thread" Id=98 Group=main RUNNABLE (in native)
            at java.net.PlainDatagramSocketImpl.receive0(Native Method)
            - locked java.net.PlainDatagramSocketImpl@2b23ce36
            at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:146)
            - locked java.net.PlainDatagramSocketImpl@2b23ce36
            at java.net.DatagramSocket.receive(DatagramSocket.java:817)
            - locked java.net.DatagramPacket@3e0a57
            - locked java.net.MulticastSocket@6918d5e0
            at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:83)

            jenkins.util.Timer [#10]

            "jenkins.util.Timer [#10]" Id=47 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at sun.misc.Unsafe.park(Native Method)
            - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
            at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            jenkins.util.Timer [#1]

            "jenkins.util.Timer [#1]" Id=19 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at sun.misc.Unsafe.park(Native Method)
            - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
            at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            jenkins.util.Timer [#2]

            "jenkins.util.Timer [#2]" Id=27 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at sun.misc.Unsafe.park(Native Method)
            - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
            at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            jenkins.util.Timer [#3]

            "jenkins.util.Timer [#3]" Id=28 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at sun.misc.Unsafe.park(Native Method)
            - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
            at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            jenkins.util.Timer [#4]

            "jenkins.util.Timer [#4]" Id=29 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at sun.misc.Unsafe.park(Native Method)
            - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
            at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            jenkins.util.Timer [#5]

            "jenkins.util.Timer [#5]" Id=30 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at sun.misc.Unsafe.park(Native Method)
            - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
            at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            jenkins.util.Timer [#6]

            "jenkins.util.Timer [#6]" Id=31 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at sun.misc.Unsafe.park(Native Method)
            - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
            at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            jenkins.util.Timer [#7]

            "jenkins.util.Timer [#7]" Id=32 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at sun.misc.Unsafe.park(Native Method)
            - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
            at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            jenkins.util.Timer [#8]

            "jenkins.util.Timer [#8]" Id=33 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at sun.misc.Unsafe.park(Native Method)
            - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
            at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            jenkins.util.Timer [#9]

            "jenkins.util.Timer [#9]" Id=46 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at sun.misc.Unsafe.park(Native Method)
            - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
            at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
            at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            JmDNS(ip-172-31-50-95.local.).State.Timer

            "JmDNS(ip-172-31-50-95.local.).State.Timer" Id=102 Group=main TIMED_WAITING on java.util.TaskQueue@1a0134b0
            at java.lang.Object.wait(Native Method)
            - waiting on java.util.TaskQueue@1a0134b0
            at java.util.TimerThread.mainLoop(Timer.java:552)
            at java.util.TimerThread.run(Timer.java:505)

            JmDNS(ip-172-31-50-95.local.).Timer

            "JmDNS(ip-172-31-50-95.local.).Timer" Id=101 Group=main TIMED_WAITING on java.util.TaskQueue@452a1c26
            at java.lang.Object.wait(Native Method)
            - waiting on java.util.TaskQueue@452a1c26
            at java.util.TimerThread.mainLoop(Timer.java:552)
            at java.util.TimerThread.run(Timer.java:505)

            LauncherControlThread[ControlPort=-1]

            "LauncherControlThread[ControlPort=-1]" Id=15 Group=main TIMED_WAITING
            at java.lang.Thread.sleep(Native Method)
            at winstone.Launcher.run(Launcher.java:247)
            at java.lang.Thread.run(Thread.java:745)

            NioSocketAcceptor-1

            "NioSocketAcceptor-1" Id=69 Group=main RUNNABLE (in native)
            at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
            at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
            - locked sun.nio.ch.Util$2@5bcae695
            - locked java.util.Collections$UnmodifiableSet@7ee1d295
            - locked sun.nio.ch.EPollSelectorImpl@58b9691
            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
            at org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:238)
            at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:432)
            at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            Number of locked synchronizers = 1
            - java.util.concurrent.ThreadPoolExecutor$Worker@58b18d61

            Ping thread for channel hudson.remoting.Channel@25d663c8:trusted-agent-2

            "Ping thread for channel hudson.remoting.Channel@25d663c8:trusted-agent-2" Id=92 Group=main TIMED_WAITING
            at java.lang.Thread.sleep(Native Method)
            at hudson.remoting.PingThread.run(PingThread.java:90)

            Ping thread for channel hudson.remoting.Channel@56434d6:trusted-agent-1

            "Ping thread for channel hudson.remoting.Channel@56434d6:trusted-agent-1" Id=91 Group=main TIMED_WAITING
            at java.lang.Thread.sleep(Native Method)
            at hudson.remoting.PingThread.run(PingThread.java:90)

            RemoteInvocationHandler [#1]

            "RemoteInvocationHandler [#1]" Id=86 Group=main TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@fad7ec5
            at java.lang.Object.wait(Native Method)
            - waiting on java.lang.ref.ReferenceQueue$Lock@fad7ec5
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
            at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:415)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
            at java.lang.Thread.run(Thread.java:745)

            RequestHandlerThread[#1] Selector0

            "RequestHandlerThread[#1] Selector0" Id=13 Group=main RUNNABLE (in native)
            at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
            at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
            - locked sun.nio.ch.Util$2@1a352a35
            - locked java.util.Collections$UnmodifiableSet@18660b04
            - locked sun.nio.ch.EPollSelectorImpl@535d401f
            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
            at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
            at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
            at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            Number of locked synchronizers = 1
            - java.util.concurrent.ThreadPoolExecutor$Worker@710d806c

            RequestHandlerThread[#2] Acceptor0 SelectChannelConnector@0.0.0.0:8080

            "RequestHandlerThread[#2] Acceptor0 SelectChannelConnector@0.0.0.0:8080" Id=14 Group=main RUNNABLE (in native)
            at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
            at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
            - locked java.lang.Object@61c0559e
            at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
            at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
            at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            Number of locked synchronizers = 1
            - java.util.concurrent.ThreadPoolExecutor$Worker@23897e13

            RequestHandlerThread[#3]

            "RequestHandlerThread[#3]" Id=24 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5a8ff0f
            at sun.misc.Unsafe.park(Native Method)
            - waiting on java.util.concurrent.SynchronousQueue$TransferStack@5a8ff0f
            at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
            at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
            at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            RequestHandlerThread[#5]

            "RequestHandlerThread[#5]" Id=26 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5a8ff0f
            at sun.misc.Unsafe.park(Native Method)
            - waiting on java.util.concurrent.SynchronousQueue$TransferStack@5a8ff0f
            at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
            at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
            at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            RequestHandlerThread[#6]

            "RequestHandlerThread[#6]" Id=128 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5a8ff0f
            at sun.misc.Unsafe.park(Native Method)
            - waiting on java.util.concurrent.SynchronousQueue$TransferStack@5a8ff0f
            at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
            at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
            at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            SCMTrigger [#3]

            "SCMTrigger [#3]" Id=129 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@605b3ba5
            at sun.misc.Unsafe.park(Native Method)
            - waiting on java.util.concurrent.SynchronousQueue$TransferStack@605b3ba5
            at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
            at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
            at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            SocketListener(ip-172-31-50-95.local.)

            "SocketListener(ip-172-31-50-95.local.)" Id=100 Group=main RUNNABLE (in native)
            at java.net.PlainDatagramSocketImpl.receive0(Native Method)
            - locked java.net.PlainDatagramSocketImpl@4f9e2643
            at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:146)
            - locked java.net.PlainDatagramSocketImpl@4f9e2643
            at java.net.DatagramSocket.receive(DatagramSocket.java:817)
            - locked java.net.DatagramPacket@59a0b187
            - locked java.net.MulticastSocket@44e74409
            at javax.jmdns.impl.SocketListener.run(SocketListener.java:41)

            TCP slave agent listener port=0

            "TCP slave agent listener port=0" Id=97 Group=main RUNNABLE (in native)
            at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
            at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
            - locked java.lang.Object@5d819b07
            at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:95)

            Thread-5

            "Thread-5" Id=44 Group=main RUNNABLE (in native)
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(SocketInputStream.java:152)
            at java.net.SocketInputStream.read(SocketInputStream.java:122)
            at com.trilead.ssh2.crypto.cipher.CipherInputStream.fill_buffer(CipherInputStream.java:41)
            at com.trilead.ssh2.crypto.cipher.CipherInputStream.internal_read(CipherInputStream.java:52)
            at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:79)
            at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108)
            at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232)
            at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:693)
            at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
            at java.lang.Thread.run(Thread.java:745)

            Thread-6

            "Thread-6" Id=45 Group=main RUNNABLE (in native)
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(SocketInputStream.java:152)
            at java.net.SocketInputStream.read(SocketInputStream.java:122)
            at com.trilead.ssh2.crypto.cipher.CipherInputStream.fill_buffer(CipherInputStream.java:41)
            at com.trilead.ssh2.crypto.cipher.CipherInputStream.internal_read(CipherInputStream.java:52)
            at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:79)
            at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108)
            at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232)
            at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:693)
            at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
            at java.lang.Thread.run(Thread.java:745)

            Finalizer

            "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@646be43
            at java.lang.Object.wait(Native Method)
            - waiting on java.lang.ref.ReferenceQueue$Lock@646be43
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
            at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

            Java2D Disposer

            "Java2D Disposer" Id=11 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@f1cddfb
            at java.lang.Object.wait(Native Method)
            - waiting on java.lang.ref.ReferenceQueue$Lock@f1cddfb
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
            at sun.java2d.Disposer.run(Disposer.java:145)
            at java.lang.Thread.run(Thread.java:745)

            Reference Handler

            "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@e7523ed
            at java.lang.Object.wait(Native Method)
            - waiting on java.lang.ref.Reference$Lock@e7523ed
            at java.lang.Object.wait(Object.java:503)
            at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)

            Signal Dispatcher

            "Signal Dispatcher" Id=4 Group=system RUNNABLE

            trusted-agent-1

            Channel reader thread: channel

            "Channel reader thread: channel" Id=10 Group=main RUNNABLE (in native)
            at java.io.FileInputStream.readBytes(Native Method)
            at java.io.FileInputStream.read(FileInputStream.java:272)
            at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
            at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
            - locked java.io.BufferedInputStream@35df85a1
            at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
            at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
            at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
            at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
            at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            main

            "main" Id=1 Group=main TIMED_WAITING on hudson.remoting.Channel@a54a40c
            at java.lang.Object.wait(Native Method)
            - waiting on hudson.remoting.Channel@a54a40c
            at hudson.remoting.Channel.join(Channel.java:948)
            at hudson.remoting.Launcher.main(Launcher.java:534)
            at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:459)
            at hudson.remoting.Launcher.run(Launcher.java:239)
            at hudson.remoting.Launcher.main(Launcher.java:192)

            Ping thread for channel hudson.remoting.Channel@a54a40c:channel

            "Ping thread for channel hudson.remoting.Channel@a54a40c:channel" Id=11 Group=main TIMED_WAITING
            at java.lang.Thread.sleep(Native Method)
            at hudson.remoting.PingThread.run(PingThread.java:90)

            pool-1-thread-6 for channel

            "pool-1-thread-6 for channel" Id=19 Group=main RUNNABLE
            at sun.management.ThreadImpl.dumpThreads0(Native Method)
            at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
            at hudson.Functions.getThreadInfos(Functions.java:1200)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:98)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:95)
            at hudson.remoting.UserRequest.perform(UserRequest.java:120)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            Number of locked synchronizers = 1
            - java.util.concurrent.ThreadPoolExecutor$Worker@33cc093f

            RemoteInvocationHandler [#1]

            "RemoteInvocationHandler [#1]" Id=9 Group=main TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@65be28a8
            at java.lang.Object.wait(Native Method)
            - waiting on java.lang.ref.ReferenceQueue$Lock@65be28a8
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
            at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:415)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
            at java.lang.Thread.run(Thread.java:745)

            Finalizer

            "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@275f61c7
            at java.lang.Object.wait(Native Method)
            - waiting on java.lang.ref.ReferenceQueue$Lock@275f61c7
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
            at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

            process reaper

            "process reaper" Id=20 Group=system TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@1af6a16b
            at sun.misc.Unsafe.park(Native Method)
            - waiting on java.util.concurrent.SynchronousQueue$TransferStack@1af6a16b
            at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
            at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
            at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            Reference Handler

            "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@670c6402
            at java.lang.Object.wait(Native Method)
            - waiting on java.lang.ref.Reference$Lock@670c6402
            at java.lang.Object.wait(Object.java:503)
            at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)

            Signal Dispatcher

            "Signal Dispatcher" Id=4 Group=system RUNNABLE

            trusted-agent-2

            Channel reader thread: channel

            "Channel reader thread: channel" Id=10 Group=main RUNNABLE (in native)
            at java.io.FileInputStream.readBytes(Native Method)
            at java.io.FileInputStream.read(FileInputStream.java:272)
            at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
            at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
            - locked java.io.BufferedInputStream@2c17b55
            at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
            at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
            at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
            at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
            at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            main

            "main" Id=1 Group=main TIMED_WAITING on hudson.remoting.Channel@a54a40c
            at java.lang.Object.wait(Native Method)
            - waiting on hudson.remoting.Channel@a54a40c
            at hudson.remoting.Channel.join(Channel.java:948)
            at hudson.remoting.Launcher.main(Launcher.java:534)
            at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:459)
            at hudson.remoting.Launcher.run(Launcher.java:239)
            at hudson.remoting.Launcher.main(Launcher.java:192)

            Ping thread for channel hudson.remoting.Channel@a54a40c:channel

            "Ping thread for channel hudson.remoting.Channel@a54a40c:channel" Id=11 Group=main TIMED_WAITING
            at java.lang.Thread.sleep(Native Method)
            at hudson.remoting.PingThread.run(PingThread.java:90)

            pool-1-thread-6 for channel

            "pool-1-thread-6 for channel" Id=19 Group=main RUNNABLE
            at sun.management.ThreadImpl.dumpThreads0(Native Method)
            at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
            at hudson.Functions.getThreadInfos(Functions.java:1200)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:98)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:95)
            at hudson.remoting.UserRequest.perform(UserRequest.java:120)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

            Number of locked synchronizers = 1
            - java.util.concurrent.ThreadPoolExecutor$Worker@189a6c56

            RemoteInvocationHandler [#1]

            "RemoteInvocationHandler [#1]" Id=9 Group=main TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@6c880adc
            at java.lang.Object.wait(Native Method)
            - waiting on java.lang.ref.ReferenceQueue$Lock@6c880adc
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
            at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:415)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
            at java.lang.Thread.run(Thread.java:745)

            Finalizer

            "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@5781292e
            at java.lang.Object.wait(Native Method)
            - waiting on java.lang.ref.ReferenceQueue$Lock@5781292e
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
            at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

            Reference Handler

            "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@60b6b7e3
            at java.lang.Object.wait(Native Method)
            - waiting on java.lang.ref.Reference$Lock@60b6b7e3
            at java.lang.Object.wait(Object.java:503)
            at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)

            Signal Dispatcher

            "Signal Dispatcher" Id=4 Group=system RUNNABLE
            {code}
            Our own Jenkins Pipeline projects seem to be getting stuck in this state of "master is offline" when attempting to run on our clusters which have zero executors assigned to the master node.

            It's unclear what, past a service restart, will clear this up :(

            Steps to reproduce:

            # forcing the master to run out of storage
            # shutdown master, clear up storage
            # restart master, confirm it's up
            # observe that it is still marked as offline in CJOC for a long time. 30+ minutes
            recampbell Ryan Campbell made changes -
            Description Our own Jenkins Pipeline projects seem to be getting stuck in this state of "master is offline" when attempting to run on our clusters which have zero executors assigned to the master node.

            It's unclear what, past a service restart, will clear this up :(

            Steps to reproduce:

            # forcing the master to run out of storage
            # shutdown master, clear up storage
            # restart master, confirm it's up
            # observe that it is still marked as offline in CJOC for a long time. 30+ minutes
            Our own Jenkins Pipeline projects seem to be getting stuck in this state of "master is offline" when attempting to run on our clusters which have zero executors assigned to the master node.

            It's unclear what, past a service restart, will clear this up :(

            Steps to reproduce:

            # forcing the master to run out of storage
            # shutdown master, clear up storage
            # restart master, confirm it's up
            # observe that it is still marked as offline for a long time. 30+ minutes
            recampbell Ryan Campbell made changes -
            Attachment threaddump.txt [ 35645 ]
            recampbell Ryan Campbell made changes -
            Labels pipeline pipeline robustness
            recampbell Ryan Campbell made changes -
            Description Our own Jenkins Pipeline projects seem to be getting stuck in this state of "master is offline" when attempting to run on our clusters which have zero executors assigned to the master node.

            It's unclear what, past a service restart, will clear this up :(

            Steps to reproduce:

            # forcing the master to run out of storage
            # shutdown master, clear up storage
            # restart master, confirm it's up
            # observe that it is still marked as offline for a long time. 30+ minutes
            Our own Jenkins Pipeline projects seem to be getting stuck in this state of "master is offline" when attempting to run on our clusters which have zero executors assigned to the master node.

            It's unclear what, past a service restart, will clear this up :(

            Steps to reproduce:

            # Start a pipeline job
            # forcing the master to run out of storage
            # shutdown master, clear up storage
            # restart master, confirm it's up
            # observe that it is still marked as offline for a long time. 30+ minutes
            recampbell Ryan Campbell made changes -
            Labels pipeline robustness pipeline pipeline-hangs robustness
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal OSS-1525 (Web Link)" [ 18665 ]
            dnusbaum Devin Nusbaum made changes -
            Assignee Devin Nusbaum [ dnusbaum ]
            dnusbaum Devin Nusbaum made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            dnusbaum Devin Nusbaum made changes -
            Remote Link This issue links to "PR 3294 (Web Link)" [ 20073 ]
            danielbeck Daniel Beck made changes -
            Status In Progress [ 3 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            danielbeck Daniel Beck made changes -
            Labels pipeline pipeline-hangs robustness lts-candidate pipeline pipeline-hangs robustness
            olivergondza Oliver Gond┼ża made changes -
            Labels lts-candidate pipeline pipeline-hangs robustness 2.107.2-fixed pipeline pipeline-hangs robustness

              People

              • Assignee:
                dnusbaum Devin Nusbaum
                Reporter:
                rtyler R. Tyler Croy
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: