Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46934

Deadlock killing jobs using build step

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Observed a case of many builds of a job all being unkillable, with thread dumps like

      "Running CpsFlowExecution[Owner[.../24191:... #24191]]" ... state=BLOCKED ...
          - waiting to lock <...> (a org.jenkinsci.plugins.workflow.cps.CpsStepContext)
            owned by "Handling POST /job/.../2690/kill from ..." ...
          at org.jenkinsci.plugins.workflow.cps.CpsStepContext.onFailure(CpsStepContext.java:326)
          at org.jenkinsci.plugins.workflow.support.steps.build.BuildTriggerListener.onDeleted(BuildTriggerListener.java:62)
          at hudson.model.listeners.RunListener.fireDeleted(RunListener.java:271)
          at hudson.model.Run.delete(Run.java:1501)
          at hudson.tasks.LogRotator.perform(LogRotator.java:145)
          at hudson.model.Job.logRotate(Job.java:467)
          at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:678)
          at ...
      "Handling POST /job/.../2690/kill from ..." ... state=WAITING ...
          - waiting on <...> (a java.util.HashMap)
          - locked <...> (a java.util.HashMap)
          at java.lang.Object.wait(Native Method)
          at java.lang.Object.wait(Object.java:502)
          at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:892)
          at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getFlowExecution(CpsStepContext.java:473)
          at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getNode(CpsStepContext.java:317)
          at org.jenkinsci.plugins.workflow.cps.CpsStepContext.scheduleNextRun(CpsStepContext.java:382)
          at org.jenkinsci.plugins.workflow.cps.CpsStepContext.completed(CpsStepContext.java:340)
          at org.jenkinsci.plugins.workflow.cps.CpsStepContext.onSuccess(CpsStepContext.java:333)
          at org.jenkinsci.plugins.workflow.support.steps.build.BuildTriggerListener.onCompleted(BuildTriggerListener.java:48)
          at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:211)
          at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:663)
          at org.jenkinsci.plugins.workflow.job.WorkflowRun.doKill(WorkflowRun.java:418)
          at ...
      

      Root cause (reason for hang of Owner.get on #2690) unknown, but there were some malformed build.xml files.

        Attachments

          Activity

          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java
          http://jenkins-ci.org/commit/workflow-job-plugin/63fdbe8c2d0f74f97e8f3b92cc566e2fff52bdf3
          Log:
          JENKINS-46934 Avoid blocking forever due to log rotation.
          · Call logRotate asynchronously.
          · Impose a timeout on Owner.get.
          · While waiting for Owner.get, record the build in the thread name.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java http://jenkins-ci.org/commit/workflow-job-plugin/63fdbe8c2d0f74f97e8f3b92cc566e2fff52bdf3 Log: JENKINS-46934 Avoid blocking forever due to log rotation. · Call logRotate asynchronously. · Impose a timeout on Owner.get. · While waiting for Owner.get, record the build in the thread name.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java
          http://jenkins-ci.org/commit/workflow-job-plugin/c750cfecc010ed5c9dd045ceb973bb2e4937f965
          Log:
          Merge pull request #70 from jglick/logRotate-hang-JENKINS-46934

          JENKINS-46934 Avoid blocking forever due to log rotation

          Compare: https://github.com/jenkinsci/workflow-job-plugin/compare/3cb7c953b958...c750cfecc010

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java http://jenkins-ci.org/commit/workflow-job-plugin/c750cfecc010ed5c9dd045ceb973bb2e4937f965 Log: Merge pull request #70 from jglick/logRotate-hang- JENKINS-46934 JENKINS-46934 Avoid blocking forever due to log rotation Compare: https://github.com/jenkinsci/workflow-job-plugin/compare/3cb7c953b958...c750cfecc010
          Hide
          kshultz Karl Shultz added a comment -

          Apologies for the inadvertent reassignments. Some JIRA keyboard shortcut must've gotten me.

          Show
          kshultz Karl Shultz added a comment - Apologies for the inadvertent reassignments. Some JIRA keyboard shortcut must've gotten me.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/main/java/org/jenkinsci/plugins/workflow/support/steps/build/BuildTriggerListener.java
          http://jenkins-ci.org/commit/pipeline-build-step-plugin/696921e476ddcf315be6808f87c9211fd9bb9371
          Log:
          JENKINS-46934 StepContext.onFailure could block; we should not block listeners on it.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/workflow/support/steps/build/BuildTriggerListener.java http://jenkins-ci.org/commit/pipeline-build-step-plugin/696921e476ddcf315be6808f87c9211fd9bb9371 Log: JENKINS-46934 StepContext.onFailure could block; we should not block listeners on it.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Sam Van Oort
          Path:
          src/main/java/org/jenkinsci/plugins/workflow/support/steps/build/BuildTriggerListener.java
          src/test/java/org/jenkinsci/plugins/workflow/support/steps/build/BuildTriggerStepTest.java
          http://jenkins-ci.org/commit/pipeline-build-step-plugin/269bccee26540da208440de574e3f6d62b7e31dc
          Log:
          Merge pull request #16 from jglick/BuildTriggerListener.onDeleted-JENKINS-46934

          JENKINS-46934 StepContext.onFailure could block; we should not block listeners on it

          Compare: https://github.com/jenkinsci/pipeline-build-step-plugin/compare/3dfefdec1f7b...269bccee2654

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Sam Van Oort Path: src/main/java/org/jenkinsci/plugins/workflow/support/steps/build/BuildTriggerListener.java src/test/java/org/jenkinsci/plugins/workflow/support/steps/build/BuildTriggerStepTest.java http://jenkins-ci.org/commit/pipeline-build-step-plugin/269bccee26540da208440de574e3f6d62b7e31dc Log: Merge pull request #16 from jglick/BuildTriggerListener.onDeleted- JENKINS-46934 JENKINS-46934 StepContext.onFailure could block; we should not block listeners on it Compare: https://github.com/jenkinsci/pipeline-build-step-plugin/compare/3dfefdec1f7b...269bccee2654
          Hide
          jglick Jesse Glick added a comment -

          Fixes merged though not yet released.

          Show
          jglick Jesse Glick added a comment - Fixes merged though not yet released.
          Hide
          jglick Jesse Glick added a comment - - edited

          Have workflow-job-2.15; still awaiting release of pipeline-build-step by Andrew Bayer I suppose.

          Show
          jglick Jesse Glick added a comment - - edited Have workflow-job-2.15 ; still awaiting release of pipeline-build-step by Andrew Bayer I suppose.

            People

            • Assignee:
              jglick Jesse Glick
              Reporter:
              jglick Jesse Glick
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: