Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60536

Jenkins job hangs in Jira call after git rev-list command

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: jira-plugin
    • Labels:
    • Environment:
    • Similar Issues:

      Description

      Recently Jenkins master  when a job is triggered it hangs around checkout

       Command that it seems to get stuck on:

      git rev-list --no-walk a4539ebbe8fe6da843fb1fe4c7b0ce5cf79a0647 # timeout=10 

      This issue only started happening sporadically from 2019-12-17 and is impacting our CI/CD pipeline. We rely on Jenkins to deploy to our environments.

       

      If you tried to kill the job, it is unresponsive.

      If you restart Jenkins master then the job that gets stuck is no longer exists

       

      Job Build Logs:

      Started by upstream project "parent-job" build number 5
      originally caused by:
       Branch indexing
      Checking out git <REPO> into /var/lib/jenkins/workspace/acceptance-tests@script to read acceptance-tests/Jenkinsfile-local
      using credential jenkins-git
       > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repository
       > git config remote.origin.url git@bitbucket.org:obillex/fundit.git # timeout=10
      Fetching upstream changes from git@bitbucket.org:obillex/fundit.git
       > git --version # timeout=10
      using GIT_SSH to set credentials Jenkins access to git
       > git fetch --tags --progress -- <REPO>+refs/heads/*:refs/remotes/origin/* # timeout=10
       > git rev-parse b6a1b7a159b86dd5a52b8385670397d568345c53^{commit} # timeout=10
      Checking out Revision b6a1b7a159b86dd5a52b8385670397d568345c53 (detached)
       > git config core.sparsecheckout # timeout=10
       > git checkout -f b6a1b7a159b86dd5a52b8385670397d568345c53 # timeout=10
      Commit message: "Commit message"
       > git rev-list --no-walk a4539ebbe8fe6da843fb1fe4c7b0ce5cf79a0647 # timeout=10

      This issue seems to be a regression of this resolved issue JENKINS-43106

        Attachments

          Issue Links

            Activity

            Hide
            markewaite Mark Waite added a comment -

            A new Jira integration has been created by Atlassian. You might consider trying the new integration, available from https://plugins.jenkins.io/atlassian-jira-software-cloud . A blog post announcing the new plugin will likely be published early in 2020.

            Show
            markewaite Mark Waite added a comment - A new Jira integration has been created by Atlassian. You might consider trying the new integration, available from https://plugins.jenkins.io/atlassian-jira-software-cloud . A blog post announcing the new plugin will likely be published early in 2020.
            Hide
            falcon benoit guerin added a comment -

            Hello Mark Waite

            Thanks for the news, and the pointers.

            This new plugin only handles build and deployment informations. My company uses the legacy Jira plugin to add comments on issues, and also the jira-steps plugin to enforce our workflow. I do not see how I will do this with this new plugin, until all these features are implemented on it. And I also think I am not alone with these usecases.

            This new Jira plugin is too young enough, and not complete enough to be considered as a drop in remplacement for the legacy Jira one. Plus the legacy is a hard dependency for blue ocean. 

            Show
            falcon benoit guerin added a comment - Hello Mark Waite Thanks for the news, and the pointers. This new plugin only handles build and deployment informations. My company uses the legacy Jira plugin to add comments on issues, and also the jira-steps plugin to enforce our workflow. I do not see how I will do this with this new plugin, until all these features are implemented on it. And I also think I am not alone with these usecases. This new Jira plugin is too young enough, and not complete enough to be considered as a drop in remplacement for the legacy Jira one. Plus the legacy is a hard dependency for blue ocean. 
            Hide
            markewaite Mark Waite added a comment -

            Thanks. You are correct. I had not read enough of the details of the new Jira plugin from Atlassian.

            Show
            markewaite Mark Waite added a comment - Thanks. You are correct. I had not read enough of the details of the new Jira plugin from Atlassian.
            Hide
            dfrance David France added a comment - - edited

            We've being seeing much the same problem stack that Kieran reported, trying to progress issues by a workflow action in Jira.  We first became aware of the problem on 20/12/2019.

            The locking thread:

            "Handling GET /view/Pipelines/job/.../457/ from a.b.c.d : qtp64133603-15 AbstractBuild/index.jelly SubversionChangeLogS
            et/digest.jelly" #15 prio=5 os_prio=0 cpu=8985.66ms elapsed=5244.95s tid=0x0000000064d07000 nid=0x1fdc waiting for monitor entry  [0
            x00000000665b7000]
               java.lang.Thread.State: BLOCKED (on object monitor)
                    at org.apache.http.nio.reactor.ssl.SSLIOSession.close(SSLIOSession.java:605)
                    - waiting to lock <0x000000008613f660> (a org.apache.http.nio.reactor.ssl.SSLIOSession)
                    at org.apache.http.impl.nio.NHttpConnectionBase.close(NHttpConnectionBase.java:511)
                    at org.apache.http.impl.nio.conn.CPoolEntry.closeConnection(CPoolEntry.java:75)
                    at org.apache.http.impl.nio.conn.CPoolEntry.close(CPoolEntry.java:101)
                    at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:423)
                    at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280)
                    at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.j
            ava:295)
                    at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
                    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
                    at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
                    at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
                    at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
                    at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseH
            ttpPromiseAsyncClient.java:36)
                    at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417)
                    at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364)
                    at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297)
                    at com.atlassian.jira.rest.client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianH
            ttpClientDecorator.java:83)
                    at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253)
                    at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.j
            ava:69)
                    at com.atlassian.jira.rest.client.internal.async.AsynchronousIssueRestClient.getIssue(AsynchronousIssueRestClient.java:186)
                    at com.atlassian.jira.rest.client.internal.async.AsynchronousIssueRestClient.getIssue(AsynchronousIssueRestClient.java:177)
                    at hudson.plugins.jira.JiraRestService.getIssue(JiraRestService.java:154)
                    at hudson.plugins.jira.JiraSession.getIssue(JiraSession.java:126)
                    at hudson.plugins.jira.JiraSite.lambda$getIssue$0(JiraSite.java:878)
                    at hudson.plugins.jira.JiraSite$$Lambda$303/0x000000002262d840.call(Unknown Source)
                    at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
                    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
                    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
                    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
                    - locked <0x000000008644f218> (a com.google.common.cache.LocalCache$StrongAccessEntry)
                    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
                    at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
                    at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
                    at hudson.plugins.jira.JiraSite.getIssue(JiraSite.java:873)
                    at hudson.plugins.jira.JiraChangeLogAnnotator.annotate(JiraChangeLogAnnotator.java:107)
                    at hudson.scm.ChangeLogSet$Entry.getMsgAnnotated(ChangeLogSet.java:252)
                    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.5/Native Method)
                    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.5/NativeMethodAccessorImpl.java:62)
                    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.5/DelegatingMethodAccessorImpl.java:43)
                    at java.lang.reflect.Method.invoke(java.base@11.0.5/Method.java:566)
                    at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
                    at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
                    at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
                    at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
                    at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
                    at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
                    at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
                    at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
            ...
            

             
            And a waiting thread:

            "Executor #1 for master : executing deployToUAT #1619" #647 daemon prio=5 os_prio=0 cpu=436.80ms elapsed=236.46s tid=0x000000006
            2b92000 nid=0x2590 waiting on condition  [0x000000006644d000]
               java.lang.Thread.State: WAITING (parking)
                    at jdk.internal.misc.Unsafe.park(java.base@11.0.5/Native Method)
                    - parking to wait for  <0x0000000083d46f58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
                    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.5/LockSupport.java:194)
                    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.5/AbstractQueuedSynchronizer.j
            ava:885)
                    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.5/AbstractQueuedSynchronizer.java:917)        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.5/AbstractQueuedSynchronizer.java:1240)
                    at java.util.concurrent.locks.ReentrantLock.lock(java.base@11.0.5/ReentrantLock.java:267)
                    at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:278)
                    at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.j
            ava:295)
                    at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
                    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
                    at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
                    at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
                    at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
                    at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseH
            ttpPromiseAsyncClient.java:36)
                    at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417)
                    at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364)
                    at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297)
                    at com.atlassian.jira.rest.client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianH
            ttpClientDecorator.java:83)
                    at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253)
                    at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.j
            ava:69)
                    at com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJqlImplGet(AsynchronousSearchRestClient.
            java:109)
                    at com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJql(AsynchronousSearchRestClient.java:94
            )
                    at hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:196)
                    at hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:136)
                    at hudson.plugins.jira.JiraSite.progressMatchingIssues(JiraSite.java:1043)
                    at hudson.plugins.jira.JiraIssueUpdateBuilder.perform(JiraIssueUpdateBuilder.java:105)
                    at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
                    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
                    at hudson.model.Build$BuildExecution.build(Build.java:206)
                    at hudson.model.Build$BuildExecution.doRun(Build.java:163)
                    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
                    at hudson.model.Run.execute(Run.java:1853)
                    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                    at hudson.model.ResourceController.execute(ResourceController.java:97)
                    at hudson.model.Executor.run(Executor.java:427) 

             

             The problem looks to be a problem with an earlier connection in the connection pool in the HttpAsyncClient. The earlier open connection is failing to be closed before processing the new request (see the first thread stack above).

            We did a couple of experiments in the jira-plugin source, and managed to find a way get the connection to complete.

            The first hope was that changing the HttpClientOptions.maxConnectionsPerHost value would stop the connection pool holding the connections, but forcing this (in the code) to either 1 or 2000 had no effect - it was still trying to close the previous (now available) connection.

            Instead, as a rough workaround we tried bypassing the connection pooling, by creating a new HttpAsyncClientBuilder against a new PoolingNHttpClientConnectionManager for each request. The change is fairly basic, but has enabled Jenkins to stay up so far.

            The change is on this fork. (Essentially it extracts a method in the constructor, to create the HttpAsyncClientBuilder, then also calls it from doExecute().)

            As mentioned, this was just a minimal/dirty change to get us up and running for now - it just bypasses connection reuse at the point of the request. For example an HttpAsyncClient without connection pooling would be cleaner.

            For reference this is working against JIRA Cloud.  We're not sure why the connections started failing to close, but this was enough to force our builds through for the time being.

            Show
            dfrance David France added a comment - - edited We've being seeing much the same problem stack that Kieran reported, trying to progress issues by a workflow action in Jira.  We first became aware of the problem on 20/12/2019. The locking thread: "Handling GET /view/Pipelines/job/.../457/ from a.b.c.d : qtp64133603-15 AbstractBuild/index.jelly SubversionChangeLogS et/digest.jelly" #15 prio=5 os_prio=0 cpu=8985.66ms elapsed=5244.95s tid=0x0000000064d07000 nid=0x1fdc waiting for monitor entry [0 x00000000665b7000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.http.nio.reactor.ssl.SSLIOSession.close(SSLIOSession.java:605) - waiting to lock <0x000000008613f660> (a org.apache.http.nio.reactor.ssl.SSLIOSession) at org.apache.http.impl.nio.NHttpConnectionBase.close(NHttpConnectionBase.java:511) at org.apache.http.impl.nio.conn.CPoolEntry.closeConnection(CPoolEntry.java:75) at org.apache.http.impl.nio.conn.CPoolEntry.close(CPoolEntry.java:101) at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:423) at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.j ava:295) at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129) at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75) at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108) at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseH ttpPromiseAsyncClient.java:36) at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417) at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364) at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297) at com.atlassian.jira.rest.client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianH ttpClientDecorator.java:83) at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253) at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.j ava:69) at com.atlassian.jira.rest.client.internal.async.AsynchronousIssueRestClient.getIssue(AsynchronousIssueRestClient.java:186) at com.atlassian.jira.rest.client.internal.async.AsynchronousIssueRestClient.getIssue(AsynchronousIssueRestClient.java:177) at hudson.plugins.jira.JiraRestService.getIssue(JiraRestService.java:154) at hudson.plugins.jira.JiraSession.getIssue(JiraSession.java:126) at hudson.plugins.jira.JiraSite.lambda$getIssue$0(JiraSite.java:878) at hudson.plugins.jira.JiraSite$$Lambda$303/0x000000002262d840.call(Unknown Source) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313) - locked <0x000000008644f218> (a com.google.common.cache.LocalCache$StrongAccessEntry) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228) at com.google.common.cache.LocalCache.get(LocalCache.java:3965) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764) at hudson.plugins.jira.JiraSite.getIssue(JiraSite.java:873) at hudson.plugins.jira.JiraChangeLogAnnotator.annotate(JiraChangeLogAnnotator.java:107) at hudson.scm.ChangeLogSet$Entry.getMsgAnnotated(ChangeLogSet.java:252) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.5/Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.5/NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.5/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(java.base@11.0.5/Method.java:566) at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314) at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185) at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) ...   And a waiting thread: "Executor #1 for master : executing deployToUAT #1619" #647 daemon prio=5 os_prio=0 cpu=436.80ms elapsed=236.46s tid=0x000000006 2b92000 nid=0x2590 waiting on condition [0x000000006644d000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.5/Native Method) - parking to wait for <0x0000000083d46f58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.5/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.5/AbstractQueuedSynchronizer.j ava:885) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.5/AbstractQueuedSynchronizer.java:917) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.5/AbstractQueuedSynchronizer.java:1240) at java.util.concurrent.locks.ReentrantLock.lock(java.base@11.0.5/ReentrantLock.java:267) at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:278) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.j ava:295) at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129) at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75) at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108) at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseH ttpPromiseAsyncClient.java:36) at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417) at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364) at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297) at com.atlassian.jira.rest.client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianH ttpClientDecorator.java:83) at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253) at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.j ava:69) at com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJqlImplGet(AsynchronousSearchRestClient. java:109) at com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJql(AsynchronousSearchRestClient.java:94 ) at hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:196) at hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:136) at hudson.plugins.jira.JiraSite.progressMatchingIssues(JiraSite.java:1043) at hudson.plugins.jira.JiraIssueUpdateBuilder.perform(JiraIssueUpdateBuilder.java:105) at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741) at hudson.model.Build$BuildExecution.build(Build.java:206) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) at hudson.model.Run.execute(Run.java:1853) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:427)    The problem looks to be a problem with an earlier connection in the connection pool in the HttpAsyncClient . The earlier open connection is failing to be closed before processing the new request (see the first thread stack above). We did a couple of experiments in the jira-plugin source, and managed to find a way get the connection to complete. The first hope was that changing the HttpClientOptions.maxConnectionsPerHost value would stop the connection pool holding the connections, but forcing this (in the code) to either 1 or 2000 had no effect - it was still trying to close the previous (now available) connection. Instead, as a rough workaround we tried bypassing the connection pooling, by creating a new HttpAsyncClientBuilder against a new PoolingNHttpClientConnectionManager for each request. The change is fairly basic, but has enabled Jenkins to stay up so far. The change is on this fork . (Essentially it extracts a method in the constructor, to create the HttpAsyncClientBuilder , then also calls it from doExecute() .) As mentioned, this was just a minimal/dirty change to get us up and running for now - it just bypasses connection reuse at the point of the request. For example an HttpAsyncClient without connection pooling would be cleaner. For reference this is working against JIRA Cloud.  We're not sure why the connections started failing to close, but this was enough to force our builds through for the time being.
            Hide
            rodrigc Craig Rodrigues added a comment - - edited

            David France can you submit a PR of what you've got working to https://github.com/jenkinsci/jira-plugin ?

            Show
            rodrigc Craig Rodrigues added a comment - - edited David France can you submit a PR of what you've got working to https://github.com/jenkinsci/jira-plugin ?

              People

              • Assignee:
                Unassigned
                Reporter:
                honestem Emma Phillips
              • Votes:
                6 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated: