Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22560

Deadlocked executors on slaves since 1.556

    Details

    • Similar Issues:

      Description

      Could it be that the change in JENKINS-22131 is causing deadlocks? If I downgrade to 1.555, there is no problem, all versions from 1.556 show the same behaviour: More and more executors are deadlocked. To me it looks if two concurrent jobs do something which locks each other.

      Warning, the following threads are deadlocked : Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568, Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569
      
      The threads are blocked in:
      hudson.model.Run.save(Run.java:1856) (#1568)
      hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178) (#1569)
      
      Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569
      hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178)
      jenkins.model.lazy.LazyBuildMixIn$RunMixIn.dropLinks(LazyBuildMixIn.java:336)
      hudson.model.AbstractBuild.dropLinks(AbstractBuild.java:193)
      hudson.model.RunMap.removeValue(RunMap.java:120)
      hudson.model.RunMap.remove(RunMap.java:86)
      jenkins.model.lazy.LazyBuildMixIn.removeRun(LazyBuildMixIn.java:216)
      hudson.model.AbstractProject.removeRun(AbstractProject.java:1002)
      hudson.model.AbstractProject.removeRun(AbstractProject.java:145)
      hudson.model.Run.removeRunFromParent(Run.java:1478)
      hudson.model.Run.delete(Run.java:1473)
      hudson.tasks.LogRotator.perform(LogRotator.java:124)
      hudson.model.Job.logRotate(Job.java:441)
      hudson.model.Run.execute(Run.java:1751)
      hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      hudson.model.ResourceController.execute(ResourceController.java:88)
      hudson.model.Executor.run(Executor.java:231)
      
      Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568
      hudson.model.Run.save(Run.java:1856)
      hudson.plugins.changelog_history.ChangeLogHistoryRunListener.copyChangeLogs(ChangeLogHistoryRunListener.java:82)
      hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:50)
      hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:40)
      hudson.model.listeners.RunListener.fireDeleted(RunListener.java:244)
      hudson.model.Run.delete(Run.java:1447)
      hudson.tasks.LogRotator.perform(LogRotator.java:124)
      hudson.model.Job.logRotate(Job.java:441)
      hudson.model.Run.execute(Run.java:1751)
      hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      hudson.model.ResourceController.execute(ResourceController.java:88)
      hudson.model.Executor.run(Executor.java:231)
      

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            Or just go back to 1.555 (or whatever) temporarily; I hope the fix will be in 1.560.

            Show
            jglick Jesse Glick added a comment - Or just go back to 1.555 (or whatever) temporarily; I hope the fix will be in 1.560.
            Hide
            bruce Dirk Kuypers added a comment -

            I am back to 1.555. But I was not sure if it is fixable in short time. Then I will change nothing and will be happy to run the ultimative test for you. I think this issue is not easily reproducible.

            Show
            bruce Dirk Kuypers added a comment - I am back to 1.555. But I was not sure if it is fixable in short time. Then I will change nothing and will be happy to run the ultimative test for you. I think this issue is not easily reproducible.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            changelog.html
            core/src/main/java/hudson/model/AbstractBuild.java
            core/src/main/java/hudson/model/Run.java
            core/src/main/java/jenkins/model/lazy/LazyBuildMixIn.java
            http://jenkins-ci.org/commit/jenkins/a5d848bcec45100ca4f12303a8c619664f2eb945
            Log:
            [FIXED JENKINS-22560] Avoid deadlock by making AbstractBuild.runMixIn final.
            (Forgot that Run’s are unmarshalled in place after a special constructor is called, so there is no need for readResolve or other tricks.)
            Also calling RunListener.onDeleted outside of the Run lock to avoid problems with things like ChangeLogHistoryRunListener.

            Compare: https://github.com/jenkinsci/jenkins/compare/b35f834b6651...a5d848bcec45

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/java/hudson/model/AbstractBuild.java core/src/main/java/hudson/model/Run.java core/src/main/java/jenkins/model/lazy/LazyBuildMixIn.java http://jenkins-ci.org/commit/jenkins/a5d848bcec45100ca4f12303a8c619664f2eb945 Log: [FIXED JENKINS-22560] Avoid deadlock by making AbstractBuild.runMixIn final. (Forgot that Run’s are unmarshalled in place after a special constructor is called, so there is no need for readResolve or other tricks.) Also calling RunListener.onDeleted outside of the Run lock to avoid problems with things like ChangeLogHistoryRunListener. Compare: https://github.com/jenkinsci/jenkins/compare/b35f834b6651...a5d848bcec45
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3292
            [FIXED JENKINS-22560] Avoid deadlock by making AbstractBuild.runMixIn final. (Revision a5d848bcec45100ca4f12303a8c619664f2eb945)

            Result = SUCCESS
            Jesse Glick : a5d848bcec45100ca4f12303a8c619664f2eb945
            Files :

            • core/src/main/java/hudson/model/AbstractBuild.java
            • core/src/main/java/jenkins/model/lazy/LazyBuildMixIn.java
            • changelog.html
            • core/src/main/java/hudson/model/Run.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3292 [FIXED JENKINS-22560] Avoid deadlock by making AbstractBuild.runMixIn final. (Revision a5d848bcec45100ca4f12303a8c619664f2eb945) Result = SUCCESS Jesse Glick : a5d848bcec45100ca4f12303a8c619664f2eb945 Files : core/src/main/java/hudson/model/AbstractBuild.java core/src/main/java/jenkins/model/lazy/LazyBuildMixIn.java changelog.html core/src/main/java/hudson/model/Run.java
            Hide
            bruce Dirk Kuypers added a comment -

            Just a short feedback: Jenkins ver. 1.560-SNAPSHOT (private-04/10/2014 15:41 GMT-jenkins) is running since about 8 hours now without a deadlock, so I think the bug is gone. Thanks a lot for the fast fix!

            Show
            bruce Dirk Kuypers added a comment - Just a short feedback: Jenkins ver. 1.560-SNAPSHOT (private-04/10/2014 15:41 GMT-jenkins) is running since about 8 hours now without a deadlock, so I think the bug is gone. Thanks a lot for the fast fix!

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                bruce Dirk Kuypers
              • Votes:
                2 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: