Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-5977

Dead lock condition due to pipe clogging

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Similar Issues:

      Description

      The way the remoting is written, a certain race condition can result in a dead lock.

      The root cause of this is that Channel.ReaderThread performs blocking write operation to OutputStream when it's executing the ProxyOutputStream.Chunk command. This synchronous behavior was necessary to ensure the proper in-order data arrival guarantee.

      Now, the situation is, the master initiates a FilePath.copyRecursiveTo from the master to a slave. This creates a piped stream pair over Channel, where the sender on the master are directly sending data over Channel, which gets to FastPipedOutputStream on the slave, then the forwarded Callable on the slave is reading from the corresponding FastPipedInputStream.

      The callable executing on the slave needs to load a class from the master, which results in this callable making an callback to the master to fetch a class file. But while this class file arrives as a Command, the master can send enough byte sequence to FastPipedOutputStream to the point that the buffer fills up and the write operation blocks.

      At this point, the class file is stuck in the network waiting to be read by the command reader thread, but that won't be able to do that until there'll be some space in the buffer, and there won't be any space in the buffer until the class file is retrieved, hence the dead lock.

        Attachments

        1. 34838_hang_on_archiving.txt
          18 kB
        2. 34838_hang_on_emma_publisher.txt
          11 kB
        3. deadlockpatch.txt
          3 kB
        4. HUDSON-5977.stack
          17 kB
        5. log.zip
          15 kB

          Issue Links

            Activity

            kohsuke Kohsuke Kawaguchi created issue -
            kohsuke Kohsuke Kawaguchi made changes -
            Field Original Value New Value
            Attachment log.zip [ 19251 ]
            glundh glundh made changes -
            Attachment deadlockpatch.txt [ 19736 ]
            scm_issue_link SCM/JIRA link daemon made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            glundh glundh made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            Assignee kohsuke [ kohsuke ]
            glundh glundh made changes -
            Attachment 34838_hang_on_archiving.txt [ 19790 ]
            Attachment 34838_hang_on_emma_publisher.txt [ 19791 ]
            olamy Olivier Lamy made changes -
            Attachment JENKINS-5977.stack [ 19792 ]
            olamy Olivier Lamy made changes -
            Link This issue is blocking JENKINS-7572 [ JENKINS-7572 ]
            buckett Matthew Buckett made changes -
            Link This issue is related to JENKINS-3346 [ JENKINS-3346 ]
            esmalling Eric Smalling made changes -
            Link This issue is blocking JENKINS-7546 [ JENKINS-7546 ]
            kohsuke Kohsuke Kawaguchi made changes -
            Status Reopened [ 4 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            lkishalmi lkishalmi made changes -
            Link This issue duplicates JENKINS-5760 [ JENKINS-5760 ]
            lkishalmi lkishalmi made changes -
            Link This issue is related to JENKINS-7809 [ JENKINS-7809 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 136070 ] JNJira + In-Review [ 187104 ]

              People

              • Assignee:
                kohsuke Kohsuke Kawaguchi
                Reporter:
                kohsuke Kohsuke Kawaguchi
              • Votes:
                27 Vote for this issue
                Watchers:
                26 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: