Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-5413

SCM polling getting hung

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: remoting
    • Labels:
      None
    • Similar Issues:

      Description

      This is to track the problem originally reported here: http://n4.nabble.com/Polling-hung-td1310838.html#a1310838
      The referenced thread is relocated to http://jenkins.361315.n4.nabble.com/Polling-hung-td1310838.html

      What the problem boils down to is that many remote operations are performed synchronously causing the channel object to be locked while a response returns. In situations where a lengthy remote operations is using the channel, SCM polling can be blocked waiting for the monitor on the channel to be released. In extreme situations, all the polling threads can wind up waiting on object monitors for the channel objects, preventing further processing of polling tasks.

      Furthermore, if the slave dies, the locked channel object still exists in the master JVM. If no IOException is thrown to indicate the termination of the connection to the pipe, the channel can never be closed because Channel.close() itself is a sychronized operation.

        Attachments

        1. DUMP1.txt
          57 kB
        2. hung_scm_pollers_02.PNG
          hung_scm_pollers_02.PNG
          145 kB
        3. thread_dump_02.txt
          92 kB
        4. threads.vetted.txt
          163 kB

          Issue Links

            Activity

            dty Dean Yu created issue -
            mdillon mdillon made changes -
            Field Original Value New Value
            Attachment threads.vetted.txt [ 19185 ]
            lkishalmi lkishalmi made changes -
            Summary SVN polling on slaves getting hung SCM polling on slaves getting hung
            Component/s master-slave [ 15489 ]
            Component/s subversion [ 15485 ]
            jpshackelford jpshackelford made changes -
            Attachment thread_dump_02.txt [ 19535 ]
            Attachment hung_scm_pollers_02.PNG [ 19536 ]
            lkishalmi lkishalmi made changes -
            Link This issue is related to JENKINS-5760 [ JENKINS-5760 ]
            lars_kruse Lars Kruse made changes -
            Description This is to track the problem originally reported here: http://n4.nabble.com/Polling-hung-td1310838.html#a1310838

            What the problem boils down to is that many remote operations are performed synchronously causing the channel object to be locked while a response returns. In situations where a lengthy remote operations is using the channel, SCM polling can be blocked waiting for the monitor on the channel to be released. In extreme situations, all the polling threads can wind up waiting on object monitors for the channel objects, preventing further processing of polling tasks.

            Furthermore, if the slave dies, the locked channel object still exists in the master JVM. If no IOException is thrown to indicate the termination of the connection to the pipe, the channel can never be closed because Channel.close() itself is a sychronized operation.
            This is to track the problem originally reported here: http://n4.nabble.com/Polling-hung-td1310838.html#a1310838
            The referenced thread is relocated to http://jenkins.361315.n4.nabble.com/Polling-hung-td1310838.html

            What the problem boils down to is that many remote operations are performed synchronously causing the channel object to be locked while a response returns. In situations where a lengthy remote operations is using the channel, SCM polling can be blocked waiting for the monitor on the channel to be released. In extreme situations, all the polling threads can wind up waiting on object monitors for the channel objects, preventing further processing of polling tasks.

            Furthermore, if the slave dies, the locked channel object still exists in the master JVM. If no IOException is thrown to indicate the termination of the connection to the pipe, the channel can never be closed because Channel.close() itself is a sychronized operation.
            jstruck Jes Struck made changes -
            Assignee Jes Struck [ jstruck ]
            jstruck Jes Struck made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jstruck Jes Struck made changes -
            Status In Progress [ 3 ] Open [ 1 ]
            jstruck Jes Struck made changes -
            Assignee Jes Struck [ jstruck ]
            mabahj Markus made changes -
            Link This issue is related to JENKINS-12302 [ JENKINS-12302 ]
            dark_lx Alex Lorenz made changes -
            Summary SCM polling on slaves getting hung SCM polling getting hung
            Priority Major [ 3 ] Critical [ 2 ]
            lacostej lacostej made changes -
            Attachment DUMP1.txt [ 23052 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-19055 [ JENKINS-19055 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 135502 ] JNJira + In-Review [ 174320 ]
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal OSS-902 (Web Link)" [ 18805 ]

              People

              • Assignee:
                Unassigned
                Reporter:
                dty Dean Yu
              • Votes:
                140 Vote for this issue
                Watchers:
                145 Start watching this issue

                Dates

                • Created:
                  Updated: