Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37632

Bitbucket Pull Request plugin has no socket timeout and hangs indefinitely

    Details

    • Similar Issues:

      Description

      On average once per day, our Jenkins server stops polling all SCMs and pull requests because the BitBucket Pull Request Builder Plugin hangs indefinitely on a socket connection. See the below excerpt for the stack trace of the hanging Timer thread. The only way to recover is to restart the Jenkins server. All Jenkins cron timers are blocked on this thread, so all other jobs are impacted when this happens, and we cannot even use Jenkins to schedule restarts for recovery.

      We would love to see a socket timeout configuration added to ApiClient so that it would gracefully recover from the hanging socket read connection.

      Stack trace:

      "jenkins.util.Timer [#7]" Id=41 Group=main RUNNABLE (in native)
      	at java.net.SocketInputStream.socketRead0(Native Method)
      	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      	at java.net.SocketInputStream.read(SocketInputStream.java:170)
      	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
      	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
      	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
      	-  locked java.lang.Object@7bc6ab76
      	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
      	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
      	-  locked sun.security.ssl.AppInputStream@71767bcb
      	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
      	at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
      	-  locked java.io.BufferedInputStream@77029758
      	at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
      	at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
      	at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
      	at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
      	at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
      	at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
      	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
      	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
      	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
      	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
      	at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send(ApiClient.java:239)
      	at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.get(ApiClient.java:213)
      	at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.hasBuildStatus(ApiClient.java:139)
      	at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketRepository.isBuildTarget(BitbucketRepository.java:224)
      	at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketRepository.getTargetPullRequests(BitbucketRepository.java:94)
      	at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketPullRequestsBuilder.run(BitbucketPullRequestsBuilder.java:37)
      	at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketBuildTrigger.run(BitbucketBuildTrigger.java:187)
      	at hudson.triggers.Trigger.checkTriggers(Trigger.java:274)
      	at hudson.triggers.Trigger$Cron.doRun(Trigger.java:222)
      	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      	Number of locked synchronizers = 1
      	- java.util.concurrent.ThreadPoolExecutor$Worker@2619b6f0
      

        Attachments

          Activity

          Hide
          codemonky David Frascone added a comment -

          Thank you for isolating this failure. We've been tracking it here: https://github.com/nishio-dens/bitbucket-pullrequest-builder-plugin/issues/180 but did not know exactly where the thread was hanging, since, once in state, we were down!

          I'll try to get a patch testing over the weekend, and, if successful, expect a release in about a week. (Since we can't trigger the failure directly, I want to make sure we cause no harm for a week or so before pushing it to the masses.)

          Feel free to assign this ticket to me, or whatever you do for bugs in plugins. I'll keep it updated, if you want, or you can follow on the github tracker.

           

          -Dave

          Show
          codemonky David Frascone added a comment - Thank you for isolating this failure. We've been tracking it here: https://github.com/nishio-dens/bitbucket-pullrequest-builder-plugin/issues/180  but did not know exactly where the thread was hanging, since, once in state, we were down! I'll try to get a patch testing over the weekend, and, if successful, expect a release in about a week. (Since we can't trigger the failure directly, I want to make sure we cause no harm for a week or so before pushing it to the masses.) Feel free to assign this ticket to me, or whatever you do for bugs in plugins. I'll keep it updated, if you want, or you can follow on the github tracker.   -Dave
          Hide
          jpfund_sr Jeremy P added a comment -

          Assigned over to you and thanks for looking into this issue.  We don't run into it anymore over here for whatever reason (apparently mitigated over a core Jenkins upgrade at some point).

          Show
          jpfund_sr Jeremy P added a comment - Assigned over to you and thanks for looking into this issue.  We don't run into it anymore over here for whatever reason (apparently mitigated over a core Jenkins upgrade at some point).
          Hide
          codemonky David Frascone added a comment -

          So, the timeouts are set to 60000ms (60 seconds), in ApiClient.java here:

          HttpClientParams params = client.getParams();
          params.setConnectionManagerTimeout(DEFAULT_TIMEOUT);
          params.setSoTimeout(DEFAULT_TIMEOUT);
          

           

          I then added debug to the `send(HttpMethodBase req)` function, and the client's parameters contain 60000 in params.getSoTimeout()

           

          So, my question is: Is it being set correctly? And, if not in the apache commons Method Params (https://hc.apache.org/httpclient-3.x/apidocs/org/apache/commons/httpclient/params/HttpMethodParams.html), where should the timeout(s) be set, when using `client.executeMethod()`?

           

          Show
          codemonky David Frascone added a comment - So, the timeouts are set to 60000ms (60 seconds), in ApiClient.java here: HttpClientParams params = client.getParams(); params.setConnectionManagerTimeout(DEFAULT_TIMEOUT); params.setSoTimeout(DEFAULT_TIMEOUT);   I then added debug to the `send(HttpMethodBase req)` function, and the client's parameters contain 60000 in params.getSoTimeout()   So, my question is: Is it being set correctly? And, if not in the apache commons Method Params ( https://hc.apache.org/httpclient-3.x/apidocs/org/apache/commons/httpclient/params/HttpMethodParams.html),  where should the timeout(s) be set, when using `client.executeMethod()`?  
          Hide
          codemonky David Frascone added a comment -

          Relevant - this looks very similar to the plugin's implementation:

           

          https://stackoverflow.com/questions/27968669/socket-timeout-not-working-inputstream-read-blocks

          Show
          codemonky David Frascone added a comment - Relevant - this looks very similar to the plugin's implementation:   https://stackoverflow.com/questions/27968669/socket-timeout-not-working-inputstream-read-blocks
          Show
          codemonky David Frascone added a comment - Shot in the dark:  https://github.com/nishio-dens/bitbucket-pullrequest-builder-plugin/pull/183
          Hide
          codemonky David Frascone added a comment -

          Jeremy P: How did you find the hung thread and get that stack dump?

          I'm going to test my fix above over time, and I'll need to be able to dump threads when our server starts working strangely. (I'm going to put that PR code up live, as a test, since I have low confidence in the fix)

           

          Thanks in advance!

           

          -Dave

          Show
          codemonky David Frascone added a comment - Jeremy P : How did you find the hung thread and get that stack dump? I'm going to test my fix above over time, and I'll need to be able to dump threads when our server starts working strangely. (I'm going to put that PR code up live, as a test, since I have low confidence in the fix)   Thanks in advance!   -Dave
          Hide
          jpfund_sr Jeremy P added a comment -

          You can access `/threadDump` on a running Jenkins server to obtain the thread dumps.  See https://wiki.jenkins.io/display/JENKINS/Obtaining+a+thread+dump

          Show
          jpfund_sr Jeremy P added a comment - You can access `/threadDump` on a running Jenkins server to obtain the thread dumps.  See  https://wiki.jenkins.io/display/JENKINS/Obtaining+a+thread+dump

            People

            • Assignee:
              codemonky David Frascone
              Reporter:
              jpfund_sr Jeremy P
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: