Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49038

Jenkins does not start due to a deadlock

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Labels:
      None
    • Environment:
      Windows Server 2012 R2
      Jenkins 2.102
      Oracle Java HotSpot(TM) 64-Bit Server VM version 25.144-b01

    • Similar Issues:

      Description

      Recent changes on the affected server:
      Jenkins-Update to 2.102, including all installed plugins
      Update for WIndows 2012 R2

      Jenkins refuses to start due to a thread deadlock during the startup process. This deadlock also affects HTTP request workers; so the server does not respond to HTTP requests anymore.

      The stacks below show the deadlock between thread pool-6-thread-16 (a.k.a. GitSCM.onLoaded) and pool-6-thread-11. The last stack shows how RequestHandlerThread1 is blocked by the lock owned by GitSCM.onLoaded.

      ------------------------------------------------------------------------------
      pool-6-thread-16

      Name: GitSCM.onLoaded
      State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@29986f0 owned by: PreventRefreshFilter.initAutoRefreshFilter
      Total blocked: 91 Total waited: 39

      Stack trace:
      sun.misc.Unsafe.park(Native Method)
      java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
      java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
      java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
      com.google.inject.internal.CycleDetectingLock$CycleDetectingLockFactory$ReentrantCycleDetectingLock.lockOrDetectPotentialLocksCycle(CycleDetectingLock.java:160)
      com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:136)
      hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:424)
      com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
      com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
      com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
      com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
      hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:386)
      hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:377)
      hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:482)
      hudson.ExtensionList.load(ExtensionList.java:366)
      hudson.ExtensionList.ensureLoaded(ExtensionList.java:304)

      • locked hudson.ExtensionList$Lock@5cfdc450
        hudson.ExtensionList.iterator(ExtensionList.java:158)
        jenkins.model.Jenkins.getDescriptorByType(Jenkins.java:1519)
        hudson.plugins.git.GitSCM.onLoaded(GitSCM.java:1859)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:498)
        hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
        hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
        org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
        jenkins.model.Jenkins$5.runTask(Jenkins.java:1063)
        org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
        org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)

      ------------------------------------------------------------------------------
      pool-6-thread-11

      Name: PreventRefreshFilter.initAutoRefreshFilter
      State: BLOCKED on hudson.ExtensionList$Lock@5cfdc450 owned by: GitSCM.onLoaded
      Total blocked: 10 Total waited: 41

      Stack trace:
      hudson.ExtensionList.ensureLoaded(ExtensionList.java:303)
      hudson.ExtensionList.iterator(ExtensionList.java:158)
      hudson.ExtensionList.get(ExtensionList.java:149)
      org.jenkinsci.plugins.scriptsecurity.scripts.ScriptApproval.get(ScriptApproval.java:97)
      org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript.configuring(SecureGroovyScript.java:126)
      org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript.readResolve(SecureGroovyScript.java:102)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:498)
      com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadResolve(SerializationMethodInvoker.java:66)
      hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:271)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
      hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
      hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
      com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
      com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
      hudson.util.XStream2.unmarshal(XStream2.java:147)
      hudson.util.XStream2.unmarshal(XStream2.java:118)
      com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
      hudson.XmlFile.unmarshal(XmlFile.java:179)
      hudson.XmlFile.unmarshal(XmlFile.java:162)
      hudson.model.Descriptor.load(Descriptor.java:893)

      • locked hudson.plugins.claim.ClaimConfig@1d46c412
        hudson.plugins.claim.ClaimConfig.<init>(ClaimConfig.java:24)
        hudson.plugins.claim.ClaimConfig$$FastClassByGuice$$74f9760b.newInstance(<generated>)
        com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
        com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61)
        com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105)
        com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
        com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
        com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
        com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
        hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:424)
        com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
        com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
        com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051)
        jenkins.ProxyInjector.getInstance(ProxyInjector.java:98)
        hudson.init.TaskMethodFinder.lookUp(TaskMethodFinder.java:124)
        hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:102)
        hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
        org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
        jenkins.model.Jenkins$5.runTask(Jenkins.java:1063)
        org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
        org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)

      ------------------------------------------------------------------------------
      RequestHandlerThread1

      Name: Handling GET / from 10.10.15.31 : RequestHandlerThread1 HudsonIsLoading/index.jelly
      State: BLOCKED on hudson.ExtensionList$Lock@5cfdc450 owned by: GitSCM.onLoaded
      Total blocked: 1 Total waited: 0

      Stack trace:
      hudson.ExtensionList.ensureLoaded(ExtensionList.java:303)
      hudson.ExtensionList.iterator(ExtensionList.java:158)
      jenkins.JenkinsHttpSessionListener.sessionCreated(JenkinsHttpSessionListener.java:50)
      org.eclipse.jetty.server.session.SessionHandler.newHttpSession(SessionHandler.java:810)
      org.eclipse.jetty.server.Request.getSession(Request.java:1550)
      org.eclipse.jetty.server.Request.getSession(Request.java:1523)
      javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:279)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:498)
      org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
      org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
      org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
      org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
      org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
      org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
      hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
      org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
      org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
      org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
      org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:140)
      org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:30)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      hudson.security.HudsonFilter.doFilter(HudsonFilter.java:169)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
      org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
      org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
      org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
      org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
      org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
      org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
      org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
      org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
      org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
      org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      org.eclipse.jetty.server.Server.handle(Server.java:564)
      org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
      org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
      org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
      org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
      org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
      org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
      org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
      org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126)
      org.eclipse.jetty.io.ManagedSelector$$Lambda$16/549293029.run(Unknown Source)
      winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      java.lang.Thread.run(Thread.java:748)

        Attachments

          Issue Links

            Activity

            Hide
            flybeetlecricket Fly Cricket added a comment -

            The issue is occuring everytime.

            Show
            flybeetlecricket Fly Cricket added a comment - The issue is occuring everytime.
            Hide
            greybird Arnaud TAMAILLON added a comment -

            Fly Cricket, can you test https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fclaim-plugin/detail/PR-53/1/artifacts version please, to tell me if situation is better ?

            I'm still not managing to reproduce, but am under the impression that it is probably linked to the actual list of plugins and maybe threads used by the task reactor lib, as on my dev pc, the deadlock does not occur.

            Show
            greybird Arnaud TAMAILLON added a comment - Fly Cricket , can you test https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fclaim-plugin/detail/PR-53/1/artifacts version please, to tell me if situation is better ? I'm still not managing to reproduce, but am under the impression that it is probably linked to the actual list of plugins and maybe threads used by the task reactor lib, as on my dev pc, the deadlock does not occur.
            Hide
            flybeetlecricket Fly Cricket added a comment -

            I've uploaded the HPI via the Jenkins GUI. Jenkins could be restarted without a deadlock. A second restart also worked.

             

            Show
            flybeetlecricket Fly Cricket added a comment - I've uploaded the HPI via the Jenkins GUI. Jenkins could be restarted without a deadlock. A second restart also worked.  
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Arnaud
            Path:
            src/main/java/hudson/plugins/claim/http/PreventRefreshFilter.java
            http://jenkins-ci.org/commit/claim-plugin/16e22544affcff18b8e4b4ae01036ad3482de698
            Log:
            JENKINS-49038 Jenkins does not start due to a deadlock

            the idea is to get the ClaimConfig retrieval out of guice logic.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Arnaud Path: src/main/java/hudson/plugins/claim/http/PreventRefreshFilter.java http://jenkins-ci.org/commit/claim-plugin/16e22544affcff18b8e4b4ae01036ad3482de698 Log: JENKINS-49038 Jenkins does not start due to a deadlock the idea is to get the ClaimConfig retrieval out of guice logic.
            Hide
            greybird Arnaud TAMAILLON added a comment -

            Thanks for your help!

            Fixed in 2.14.1

            Show
            greybird Arnaud TAMAILLON added a comment - Thanks for your help! Fixed in 2.14.1

              People

              • Assignee:
                greybird Arnaud TAMAILLON
                Reporter:
                flybeetlecricket Fly Cricket
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: