Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19446

Deadlock while parallel deletion/rename of jobs

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core
    • Labels:
    • Environment:
      RedHat 5.6 java 1.6. jenkins 1.522
    • Similar Issues:

      Description

      While parallel deletion/renaming of jobs we have following deadlock:

      Found one Java-level deadlock:
      =============================
      "Handling POST /jenkins/view/QA/view/Verification/createItem : http-bio-8080-exec-189":
        waiting to lock monitor 0x000000004eba7ca8 (object 0x000000070032c4c8, a hudson.model.Hudson),
        which is held by "Handling POST /jenkins/view/RE/view/Onsite/view/Dev/job/RE-Onsite-Build-ATG-Stg/doRename : http-bio-8080-exec-140"
      "Handling POST /jenkins/view/RE/view/Onsite/view/Dev/job/RE-Onsite-Build-ATG-Stg/doRename : http-bio-8080-exec-140":
        waiting to lock monitor 0x000000004f70e1e0 (object 0x0000000752e08ea0, a hudson.maven.MavenModuleSet),
        which is held by "Handling POST /jenkins/view/Maintenance/job/__tst1__/doDelete : http-bio-8080-exec-166"
      "Handling POST /jenkins/view/Maintenance/job/__tst1__/doDelete : http-bio-8080-exec-166":
        waiting to lock monitor 0x000000004eba7ca8 (object 0x000000070032c4c8, a hudson.model.Hudson),
        which is held by "Handling POST /jenkins/view/RE/view/Onsite/view/Dev/job/RE-Onsite-Build-ATG-Stg/doRename : http-bio-8080-exec-140"
      

      Dump of threads:

      "Handling POST /jenkins/view/QA/view/Verification/createItem : http-bio-8080-exec-189":
              at jenkins.model.Jenkins.doCreateItem(Jenkins.java:2889)
              - waiting to lock <0x000000070032c4c8> (a hudson.model.Hudson)
              at jenkins.model.Jenkins.doCreateItem(Jenkins.java:312)
              at hudson.model.ListView.doCreateItem(ListView.java:262)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:297)
              at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:160)
              at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:95)
              at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:586)
              at org.kohsuke.stapler.Stapler.service(Stapler.java:217)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
              at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
              at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:208)
              at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:181)
              at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)
              at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
      
      "Handling POST /jenkins/view/RE/view/Onsite/view/Dev/job/RE-Onsite-Build-ATG-Stg/doRename : http-bio-8080-exec-140":
              at hudson.model.AbstractProject.save(AbstractProject.java:278)
              - waiting to lock <0x0000000752e08ea0> (a hudson.maven.MavenModuleSet)
              at hudson.plugins.copyartifact.CopyArtifact$ListenerImpl.onRenamed(CopyArtifact.java:331)
              at hudson.model.AbstractItem.renameTo(AbstractItem.java:294)
              - locked <0x0000000709ca3ef8> (a hudson.model.FreeStyleProject)
              - locked <0x000000070032c4c8> (a hudson.model.Hudson)
              at hudson.model.Job.renameTo(Job.java:575)
              at hudson.model.Job.doDoRename(Job.java:1294)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:297)
              at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:29)
              at org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:398)
              at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:160)
              at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:95)
              at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:586)
              at org.kohsuke.stapler.Stapler.service(Stapler.java:217)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
      
      "Handling POST /jenkins/view/Maintenance/job/__tst1__/doDelete : http-bio-8080-exec-166":
              at jenkins.model.Jenkins.save(Jenkins.java:2635)
              - waiting to lock <0x000000070032c4c8> (a hudson.model.Hudson)
              at jenkins.model.Jenkins.onDeleted(Jenkins.java:2435)
              at jenkins.model.Jenkins.onDeleted(Jenkins.java:312)
              at hudson.model.AbstractItem.invokeOnDeleted(AbstractItem.java:524)
              at hudson.model.AbstractItem.delete(AbstractItem.java:511)
              - locked <0x0000000752e08ea0> (a hudson.maven.MavenModuleSet)
              at hudson.model.Job.delete(Job.java:585)
              - locked <0x0000000752e08ea0> (a hudson.maven.MavenModuleSet)
              at hudson.model.AbstractProject.doDoDelete(AbstractProject.java:1915)
              at sun.reflect.GeneratedMethodAccessor1652.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:297)
              at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:29)
              at org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:398)
              at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:160)
              at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:95)
              at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:586)
      

        Attachments

          Issue Links

            Activity

            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            core/src/main/java/hudson/model/AbstractItem.java
            http://jenkins-ci.org/commit/jenkins/41baac80fbf559aa3eb30c635abe0386815fd67f
            Log:
            [FIXED JENKINS-19446]

            Defer the notification to avoid deadlock.

            (cherry picked from commit a5755cb3e1f901c98a2263a3ae1851489cb8e47b)

            Conflicts:
            changelog.html

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/model/AbstractItem.java http://jenkins-ci.org/commit/jenkins/41baac80fbf559aa3eb30c635abe0386815fd67f Log: [FIXED JENKINS-19446] Defer the notification to avoid deadlock. (cherry picked from commit a5755cb3e1f901c98a2263a3ae1851489cb8e47b) Conflicts: changelog.html
            Hide
            jglick Jesse Glick added a comment -

            Filed JENKINS-22001 for the problematic fix.

            Show
            jglick Jesse Glick added a comment - Filed JENKINS-22001 for the problematic fix.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/hudson/model/ListView.java
            core/src/main/java/jenkins/model/Jenkins.java
            http://jenkins-ci.org/commit/jenkins/3aa0dd5368444728b608bce9790e59983dabbc8a
            Log:
            JENKINS-19446 Saving Jenkins configuration in onRenamed/onDeleted is often overkill.
            This is already handled by ListView.Listener (other kinds of views are now responsible for their own listeners).
            And handled better—if there are no actual changes
            (because you have no such views, or they do not mention this job), there is no need to save.
            Also moving the actual saving in those remaining cases out of the lock on the ListView itself, just in case.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/model/ListView.java core/src/main/java/jenkins/model/Jenkins.java http://jenkins-ci.org/commit/jenkins/3aa0dd5368444728b608bce9790e59983dabbc8a Log: JENKINS-19446 Saving Jenkins configuration in onRenamed/onDeleted is often overkill. This is already handled by ListView.Listener (other kinds of views are now responsible for their own listeners). And handled better—if there are no actual changes (because you have no such views, or they do not mention this job), there is no need to save. Also moving the actual saving in those remaining cases out of the lock on the ListView itself, just in case.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/hudson/model/AbstractItem.java
            core/src/main/java/hudson/model/Job.java
            test/src/test/java/hudson/model/ListViewTest.java
            http://jenkins-ci.org/commit/jenkins/00d27176f3a452e38bd0caafebe78c7bc2fa822f
            Log:
            [FIXED JENKINS-22001] Simpler fix of JENKINS-19446 that does not introduce asynchronous behavior.

            Compare: https://github.com/jenkinsci/jenkins/compare/334ce1be691c...00d27176f3a4

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/model/AbstractItem.java core/src/main/java/hudson/model/Job.java test/src/test/java/hudson/model/ListViewTest.java http://jenkins-ci.org/commit/jenkins/00d27176f3a452e38bd0caafebe78c7bc2fa822f Log: [FIXED JENKINS-22001] Simpler fix of JENKINS-19446 that does not introduce asynchronous behavior. Compare: https://github.com/jenkinsci/jenkins/compare/334ce1be691c...00d27176f3a4
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3204
            JENKINS-19446 Saving Jenkins configuration in onRenamed/onDeleted is often overkill. (Revision 3aa0dd5368444728b608bce9790e59983dabbc8a)
            [FIXED JENKINS-22001] Simpler fix of JENKINS-19446 that does not introduce asynchronous behavior. (Revision 00d27176f3a452e38bd0caafebe78c7bc2fa822f)

            Result = SUCCESS
            Jesse Glick : 3aa0dd5368444728b608bce9790e59983dabbc8a
            Files :

            • core/src/main/java/jenkins/model/Jenkins.java
            • core/src/main/java/hudson/model/ListView.java

            Jesse Glick : 00d27176f3a452e38bd0caafebe78c7bc2fa822f
            Files :

            • test/src/test/java/hudson/model/ListViewTest.java
            • core/src/main/java/hudson/model/AbstractItem.java
            • core/src/main/java/hudson/model/Job.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3204 JENKINS-19446 Saving Jenkins configuration in onRenamed/onDeleted is often overkill. (Revision 3aa0dd5368444728b608bce9790e59983dabbc8a) [FIXED JENKINS-22001] Simpler fix of JENKINS-19446 that does not introduce asynchronous behavior. (Revision 00d27176f3a452e38bd0caafebe78c7bc2fa822f) Result = SUCCESS Jesse Glick : 3aa0dd5368444728b608bce9790e59983dabbc8a Files : core/src/main/java/jenkins/model/Jenkins.java core/src/main/java/hudson/model/ListView.java Jesse Glick : 00d27176f3a452e38bd0caafebe78c7bc2fa822f Files : test/src/test/java/hudson/model/ListViewTest.java core/src/main/java/hudson/model/AbstractItem.java core/src/main/java/hudson/model/Job.java

              People

              • Assignee:
                Unassigned
                Reporter:
                joesephz Joe Zeldin
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: