Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55700

deadlock remoting TCP agent connection handler

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Duplicate
    • Component/s: remoting
    • Labels:
    • Environment:
      Server: Jenkins 2.138.2
      Remoting: 3.25
      Java: 1.8.181
    • Similar Issues:
    • Released As:
      Remoting 3.27

      Description

      Found one Java-level deadlock:
      =============================
      "TCP agent connection handler #159021 with /10.116.113.11:54776":
        waiting for ownable synchronizer 0x00000002ce4a6770, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
        which is held by "IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/10.116.241.122:8181 remote=10.116.113.11/10.116.113.11:54776]] / Computer.threadPoolForRemoting [#12366]"
      "IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/10.116.241.122:8181 remote=10.116.113.11/10.116.113.11:54776]] / Computer.threadPoolForRemoting [#12366]":
        waiting to lock monitor 0x00007f75cc5900f8 (object 0x00000002ce4de778, a org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer),
        which is held by "TCP agent connection handler #159021 with /10.116.113.11:54776"
      
      Java stack information for the threads listed above:
      ===================================================
      "TCP agent connection handler #159021 with /10.116.113.11:54776":
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000002ce4a6770> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
      	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.getNextSend(ProtocolStack.java:857)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:686)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:518)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
      	- locked <0x00000002ce4a6800> (a java.lang.Object)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:692)
      	at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.doSend(ConnectionHeadersFilterLayer.java:497)
      	- locked <0x00000002ce4de778> (a org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:692)
      	at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157)
      	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.start(ChannelApplicationLayer.java:230)
      	at org.jenkinsci.remoting.protocol.ProtocolStack.init(ProtocolStack.java:201)
      	at org.jenkinsci.remoting.protocol.ProtocolStack.access$700(ProtocolStack.java:106)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Builder.build(ProtocolStack.java:554)
      	at org.jenkinsci.remoting.engine.JnlpProtocol4Handler.handle(JnlpProtocol4Handler.java:153)
      	at jenkins.slaves.JnlpSlaveAgentProtocol4.handle(JnlpSlaveAgentProtocol4.java:205)
      	at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:260)
      "IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/10.116.241.122:8181 remote=10.116.113.11/10.116.113.11:54776]] / Computer.threadPoolForRemoting [#12366]":
      	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvRemoved(FilterLayer.java:133)
      	- waiting to lock <0x00000002ce4de778> (a org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.getNextRecv(ProtocolStack.java:929)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:663)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:369)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
      	at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
      	at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:160)
      	at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      
      Found 1 deadlock.
       

        Attachments

          Issue Links

            Activity

            Hide
            jthompson Jeff Thompson added a comment -

            Looks like a duplicate to JENKINS-53569, which was fixed in Remoting 3.27. Both reports are connected to `FilterLayer.onRecvRemoved()`. I recommend updating.

            Show
            jthompson Jeff Thompson added a comment - Looks like a duplicate to JENKINS-53569 , which was fixed in Remoting 3.27. Both reports are connected to `FilterLayer.onRecvRemoved()`. I recommend updating.
            Hide
            jthompson Jeff Thompson added a comment -

            Looks like a duplicate with a released fix. If upgrading doesn't resolve the issue, please re-open.

            Show
            jthompson Jeff Thompson added a comment - Looks like a duplicate with a released fix. If upgrading doesn't resolve the issue, please re-open.

              People

              • Assignee:
                jthompson Jeff Thompson
                Reporter:
                brainsam Alexander Moiseenko
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: