Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37921

SSE failures and deadlocking with github auth plugin installed (needs investigation)

    Details

    • Epic Link:
    • Sprint:
      1.0-beta-1, 1.0-b05/b-06
    • Similar Issues:

      Description

      When running blue ocean with github auth enabled, SSE events often don't arrive when running in mvn hpi:run mode. Jenkins can "lock up" (all web threads busy) and blocking SSE threads were noted in a thread dump.

      This hasn't been observed running in the wild (eg on dogfood), but is worth investigation as it is a bit of a DOS problem if it is "real".

      To reproduce:

      run blueocean via mvn hpi:run
      Setup brand new workspace
      Install github authorization plugin https://wiki.jenkins-ci.org/display/JENKINS/GitHub+OAuth+Plugin
      Follow its instructions for setting up an app and callback
      Create a pipeline (simple one will do, anything really, a few shell steps)
      Run the pipeline a few times from blue ocean - notice SSE events will not arrive
      Keep clicking around.
      (see video link above for what I do to see this).

      When the dashboard looks "broken" or it stops responding, try to go to the classic UI and you will see it is waiting on a socket. It is at this point I took a thread dump.

      Thread dump that showed blocking SSE threads:

      "Handling POST /jenkins/sse-gateway/configure from 0:0:0:0:0:0:0:1 : qtp1780682194-650" #650 prio=5 os_prio=31 tid=0x00007fec9106a800 nid=0xac4b waiting for monitor entry [0x0000700003dcd000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at org.jenkinsci.plugins.ssegateway.Endpoint.doConfigure(Endpoint.java:165)
      	- waiting to lock <0x00000007b5b05cd8> (a java.lang.String)
      	at sun.reflect.GeneratedMethodAccessor211.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:483)
      	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324)
      	at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:52)
      	at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
      	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:249)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:117)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.jenkinsci.plugins.ssegateway.Endpoint.process(Endpoint.java:88)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:58)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:499)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
      	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
      	at java.lang.Thread.run(Thread.java:745)
      

      A video of this happening and what it looks like visually: http://www.giphy.com/gifs/3o6ZtaL9BGmvTVlqEw

        Attachments

          Issue Links

            Activity

            Hide
            jamesdumay James Dumay added a comment -

            This should prevent the thread starvation https://github.com/jenkinsci/sse-gateway-plugin/pull/8

            Show
            jamesdumay James Dumay added a comment - This should prevent the thread starvation https://github.com/jenkinsci/sse-gateway-plugin/pull/8
            Hide
            michaelneale Michael Neale added a comment - - edited

            Tom FENNELLY I think this is worth investigating, including james proposed fixes (alarming for a product manager to pick up concurrent code but here we are !).

            This hasn't been seen "in the wild" but myself and Ivan can reliably reproduce this. I took a thread dump to see what was locking things up. Currently only impacting when github oauth plugin is installed (and only in some cases) but it may highlight some underlying issue that may bite hard.

            Its only marked as important as this does stop you from accessing classic for a while (it does seem to get unblocked eventually).

            If this isn't as serious as it looks, can bump down the priority.

            Show
            michaelneale Michael Neale added a comment - - edited Tom FENNELLY I think this is worth investigating, including james proposed fixes (alarming for a product manager to pick up concurrent code but here we are !). This hasn't been seen "in the wild" but myself and Ivan can reliably reproduce this. I took a thread dump to see what was locking things up. Currently only impacting when github oauth plugin is installed (and only in some cases) but it may highlight some underlying issue that may bite hard. Its only marked as important as this does stop you from accessing classic for a while (it does seem to get unblocked eventually). If this isn't as serious as it looks, can bump down the priority.
            Hide
            tfennelly Tom FENNELLY added a comment -

            Closing it as the work here is being looked at under JENKINS-38252

            Show
            tfennelly Tom FENNELLY added a comment - Closing it as the work here is being looked at under JENKINS-38252

              People

              • Assignee:
                tfennelly Tom FENNELLY
                Reporter:
                michaelneale Michael Neale
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: