Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40965

Gerrit trigger connection hangs while WatchDog attempts to reconnect.

    Details

    • Similar Issues:

      Description

      Hi,

      On some occasions when the connection between Jenkins and Gerrit server is down, the watchdog fails to re-establish the connection back. Many times, it works!

      Looks like the WatchDog attempting to shutdown the stream-events connection waits indefinitely, eventually BLOCKING many other Gerrit serverStatuses threads.

      The only way to recover is to restart jenkins. Could be similar to the issue already report JENKINS-15065

      ThreadDump

      "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx : http-bio-8443-exec-914" #1169869 daemon prio=5 os_prio=0 tid=0x00007fb90c3b6800 nid=0x2d2a in Object.wait() [0x00007fb8e4495000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	at java.lang.Thread.join(Thread.java:1245)
      	- locked <0x0000000545878ae0> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
      	at java.lang.Thread.join(Thread.java:1319)
      	at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
      	- locked <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	.
      	.
      	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
      	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
      	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
      	- locked <0x0000000560c9c5f8> (a org.apache.tomcat.util.net.SocketWrapper)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
         Locked ownable synchronizers:
      	- <0x0000000561678500> (a java.util.concurrent.ThreadPoolExecutor$Worker)
      
      "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-938 GerritManagement/serverStatuses.jelly" #1172250 daemon prio=5 os_prio=0 tid=0x00007fb904856000 nid=0x3e25 waiting for monitor entry [0x00007fb8d6a03000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
      	- waiting to lock <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
      	at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
      	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
      ..
      .
         Locked ownable synchronizers:
      	- <0x00000005c0101218> (a java.util.concurrent.ThreadPoolExecutor$Worker)
      
      "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-929 GerritManagement/serverStatuses.jelly" #1170917 daemon prio=5 os_prio=0 tid=0x00007fb93a009800 nid=0x6f08 waiting for monitor entry [0x00007fb8dc822000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
      	- waiting to lock <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
      	at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
      	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
      	at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
      	at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
      .
      .
         Locked ownable synchronizers:
      	- <0x000000056ee04ca0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
      
      
      "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx: http-bio-8443-exec-914 - priority:5 - threadId:0x00007fb90c3b6800 - nativeId:0x2d2a - state:WAITING
      stackTrace:
          java.lang.Thread.State: WAITING (on object monitor)
          at java.lang.Object.wait(Native Method)
          at java.lang.Thread.join(Thread.java:1245)
          - locked <0x0000000545878ae0> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
          at java.lang.Thread.join(Thread.java:1319)
          at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
          - locked <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      
      
      

      Catalina.out

      Jan 10, 2017 1:42:49 AM com.sonymobile.tools.gerrit.gerritevents.watchdog.StreamWatchdog run
      INFO: Last lively connection with Gerrit was 60 seconds ago; reconnecting.
      

      Thanks!
      Dilip Mahadevappa

        Attachments

          Issue Links

            Activity

            Hide
            dilipm79 Dilip Mahadevappa added a comment -

            We had another occurrence of this with the same stacktrace.

            Show
            dilipm79 Dilip Mahadevappa added a comment - We had another occurrence of this with the same stacktrace.
            Hide
            dilipm79 Dilip Mahadevappa added a comment -

            We had another occurrence of the same issue.. there was some 30 threads in BLOCKED waiting to lock states..

            *Gerrit version: 2.22.0 *

            "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-112 GerritManagement/serverStatuses.jelly" daemon prio=10 tid=0x00007f4aa4ac0000 nid=0x6664 waiting for monitor entry [0x00007f4a61a33000]
               java.lang.Thread.State: BLOCKED (on object monitor)
            	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
            	- waiting to lock <0x0000000543590590> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
            	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
            	at sun.reflect.GeneratedMethodAccessor827.invoke(Unknown Source)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:606)
            	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
            	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
            	at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
            	at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
            	at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
            	at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
            	at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
            	at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
            	at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
            	at org.apache.commons.jelly.parser.EscapingExpression.evaluate(EscapingExpression.java:24)
            	at org.apache.commons.jelly.impl.ExpressionScript.run(ExpressionScript.java:66)
            	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
            	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
            	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
            	at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
            	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
            	at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380)
            	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
            	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
            	at javax.servlet.http.HttpServlet.service(HttpServlet.java:728
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
            	at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
            	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
            	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
            	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
            	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
            	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
            	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
            	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:611)
            	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
            	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
            	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
            	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
            	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409)
            	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
            	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
            	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
            	- locked <0x000000057d42c0f0> (a org.apache.tomcat.util.net.SocketWrapper)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            	at java.lang.Thread.run(Thread.java:745)
            
               Locked ownable synchronizers:
            	- <0x000000057d42c240> (a java.util.concurrent.ThreadPoolExecutor$Worker)
            
            
            "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx : http-bio-8443-exec-98" daemon prio=10 tid=0x00007f4a54fb7000 nid=0x170d in Object.wait() [0x00007f4a6294c000]
               java.lang.Thread.State: WAITING (on object monitor)
            	at java.lang.Object.wait(Native Method)
            	at java.lang.Thread.join(Thread.java:1281)
            	- locked <0x0000000547091b00> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
            	at java.lang.Thread.join(Thread.java:1355)
            	at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
            	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
            	- locked <0x0000000543590590> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
            	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:606)
            	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:320)
            	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:163)
            	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
            	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
            	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
            	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
            	at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233)
            	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
            	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
            	at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380)
            	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
            	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
            	at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
            	at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
            	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
            	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
            	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
            	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
            	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
            	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
            	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:611)
            	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
            	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
            	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
            	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
            	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409)
            	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
            	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
            	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
            	- locked <0x000000054ac742f0> (a org.apache.tomcat.util.net.SocketWrapper)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            	at java.lang.Thread.run(Thread.java:745)
            
               Locked ownable synchronizers:
            	- <0x000000054ac743e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
            
            
            
            Show
            dilipm79 Dilip Mahadevappa added a comment - We had another occurrence of the same issue.. there was some 30 threads in BLOCKED waiting to lock states.. *Gerrit version: 2.22.0 * "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-112 GerritManagement/serverStatuses.jelly" daemon prio=10 tid=0x00007f4aa4ac0000 nid=0x6664 waiting for monitor entry [0x00007f4a61a33000] java.lang. Thread .State: BLOCKED (on object monitor) at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554) - waiting to lock <0x0000000543590590> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer) at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219) at sun.reflect.GeneratedMethodAccessor827.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314) at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185) at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74) at org.apache.commons.jelly.parser.EscapingExpression.evaluate(EscapingExpression.java:24) at org.apache.commons.jelly.impl.ExpressionScript.run(ExpressionScript.java:66) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135) at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:611) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315) - locked <0x000000057d42c0f0> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) Locked ownable synchronizers: - <0x000000057d42c240> (a java.util.concurrent.ThreadPoolExecutor$Worker) "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx : http-bio-8443-exec-98" daemon prio=10 tid=0x00007f4a54fb7000 nid=0x170d in Object .wait() [0x00007f4a6294c000] java.lang. Thread .State: WAITING (on object monitor) at java.lang. Object .wait(Native Method) at java.lang. Thread .join( Thread .java:1281) - locked <0x0000000547091b00> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection) at java.lang. Thread .join( Thread .java:1355) at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603) at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537) - locked <0x0000000543590590> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer) at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:320) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:163) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135) at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:611) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315) - locked <0x000000054ac742f0> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) Locked ownable synchronizers: - <0x000000054ac743e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
            Hide
            dilipm79 Dilip Mahadevappa added a comment -

            Whenever the issue is happening, the Gerrit trigger configure page itself hangs. Like see in this screenshot

            Show
            dilipm79 Dilip Mahadevappa added a comment - Whenever the issue is happening, the Gerrit trigger configure page itself hangs. Like see in this screenshot
            Hide
            pedersen Björn Pedersen added a comment -
            Show
            pedersen Björn Pedersen added a comment - Possibly fixed by https://github.com/sonyxperiadev/gerrit-events/pull/62/ ?
            Hide
            rsandell rsandell added a comment -

            Released in 2.23.3

            Show
            rsandell rsandell added a comment - Released in 2.23.3

              People

              • Assignee:
                hektve Hector David
                Reporter:
                dilipm79 Dilip Mahadevappa
              • Votes:
                7 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: