Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20239

Deadlock when disabling a build which is being scheduled

    Details

    • Similar Issues:

      Description

      A deadlock can occur when a job is disabled while it is being scheduled in the queue.

      Found one Java-level deadlock:
      =============================
      "Handling GET / : http-8106-21":
        waiting to lock monitor 0x00007fdc673f2580 (object 0x00000000da9051e8, a hudson.model.Queue),
        which is held by "Jenkins cron thread"
      "Jenkins cron thread":
        waiting to lock monitor 0x00007fdc886ccce8 (object 0x00000000dde70f88, a hudson.model.FreeStyleProject),
        which is held by "Handling POST /job/xxxx/configSubmit : http-8106-3"
      "Handling POST /job/xxx/configSubmit : http-8106-3":
        waiting to lock monitor 0x00007fdc673f2580 (object 0x00000000da9051e8, a hudson.model.Queue),
        which is held by "Jenkins cron thread"
       
      Java stack information for the threads listed above:
      ===================================================
      "Handling GET / : http-8106-21":
      	at hudson.model.Queue.getItems(Queue.java:641)
      	- waiting to lock <0x00000000da9051e8> (a hudson.model.Queue)
      	at hudson.model.Queue$CachedItemList.get(Queue.java:196)
      	at hudson.model.Queue.getApproximateItemsQuickly(Queue.java:671)
      	at hudson.model.View.getApproximateQueueItemsQuickly(View.java:457)
      	at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
      	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
      	at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
      	at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
      	at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
      	at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
      	at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
      	at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
      	at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
      	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:58)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98)
      	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
      	at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98)
      	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
      	at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
      	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
      	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
      	at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
      	at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:666)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:736)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:214)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at hudson.ResponseHeaderFilter.doFilter(ResponseHeaderFilter.java:100)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
      	at hudson.plugins.cloudbees.account.EC2CloudListener$2.doFilter(EC2CloudListener.java:57)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
      	at org.jvnet.hudson.plugins.cors.JenkinsCorsFilter.doFilter(JenkinsCorsFilter.java:38)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at com.cloudbees.jenkins.plugins.dacsecurity.auth.VerifyHostFilter.doFilter(VerifyHostFilter.java:32)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at com.cloudbees.jenkins.plugins.dacsecurity.auth.CloudBeesBasicAuthFilter.doFilter(CloudBeesBasicAuthFilter.java:122)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
      	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:554)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
      	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
      	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
      	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
      	at java.lang.Thread.run(Unknown Source)
      "Jenkins cron thread":
      	at hudson.model.Project.getBuildersList(Project.java:107)
      	- waiting to lock <0x00000000dde70f88> (a hudson.model.FreeStyleProject)
      	at hudson.model.Project.getResourceActivities(Project.java:136)
      	at hudson.model.AbstractProject.getResourceList(AbstractProject.java:1340)
      	at hudson.model.Queue.isBuildBlocked(Queue.java:939)
      	at hudson.model.Queue.maintain(Queue.java:1024)
      	- locked <0x00000000da9051e8> (a hudson.model.Queue)
      	at hudson.model.Queue$MaintainTask.doRun(Queue.java:1763)
      	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
      	at java.util.TimerThread.mainLoop(Unknown Source)
      	at java.util.TimerThread.run(Unknown Source)
      "Handling POST /job/xxxx/configSubmit : http-8106-3":
      	at hudson.model.Queue.cancel(Queue.java:591)
      	- waiting to lock <0x00000000da9051e8> (a hudson.model.Queue)
      	at hudson.model.AbstractProject.makeDisabled(AbstractProject.java:673)
      	at hudson.model.AbstractProject.submit(AbstractProject.java:1895)
      	at hudson.model.Project.submit(Project.java:195)
      	at hudson.model.Job.doConfigSubmit(Job.java:1076)
      	- locked <0x00000000dde70f88> (a hudson.model.FreeStyleProject)
      	at hudson.model.AbstractProject.doConfigSubmit(AbstractProject.java:753)
      	at sun.reflect.GeneratedMethodAccessor1434.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
      	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:214)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at hudson.ResponseHeaderFilter.doFilter(ResponseHeaderFilter.java:100)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
      	at hudson.plugins.cloudbees.account.EC2CloudListener$2.doFilter(EC2CloudListener.java:57)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
      	at org.jvnet.hudson.plugins.cors.JenkinsCorsFilter.doFilter(JenkinsCorsFilter.java:38)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at com.cloudbees.jenkins.plugins.dacsecurity.auth.VerifyHostFilter.doFilter(VerifyHostFilter.java:32)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at com.cloudbees.jenkins.plugins.dacsecurity.auth.CloudBeesBasicAuthFilter.doFilter(CloudBeesBasicAuthFilter.java:122)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
      	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:554)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
      	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
      	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
      	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
      	at java.lang.Thread.run(Unknown Source)
       
      Found 1 deadlock.
      

        Attachments

          Activity

          There are no comments yet on this issue.

            People

            • Assignee:
              Unassigned
              Reporter:
              recampbell Ryan Campbell
            • Votes:
              2 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: