Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59083

Builds stuck on "is already in progress" forever

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: workflow-job-plugin
    • Labels:
    • Environment:
    • Similar Issues:
    • Released As:
      workflow-job 2.35

      Description

      After upgrading our jenkins master and plugins, some of our build jobs - specifically it seemed like most pipeline jobs - were stuck waiting for previous build to complete, even though it already completed.

       

      This seems to have been caused by JENKINS-46076.

      After donwngrading the plugin to version 2.32, all builds were working correctly.

       

      From some testing, it seems that calling

      Jenkins.instance.getItemByFullName(...).getBuildByNumber(..).isLogUpdated()
      

      On the previous build (in the sample case build #185) returns true, although the build is completed and the log seems to have been completed.

      This issue happened immediately after restarting the jenkins instance with any of our pipeline jobs

        Attachments

        1. Capture.PNG
          Capture.PNG
          70 kB
        2. Capture2.PNG
          Capture2.PNG
          66 kB
        3. failed.PNG
          failed.PNG
          70 kB
        4. image.png
          image.png
          19 kB
        5. plugins.txt
          5 kB
        6. stuck-build-2.tar.gz
          16 kB
        7. tdump
          75 kB

          Issue Links

            Activity

            Hide
            hellspam Roy Arnon added a comment -

            Yes, I do see a stack trace like that in the log:

            Aug 27, 2019 7:37:09 PM org.jenkinsci.plugins.workflow.job.WorkflowRun getListener
            WARNING: null
            java.lang.IllegalStateException: trying to open a build log on Lint.Hooks #1923 after it has completed
                    at org.jenkinsci.plugins.workflow.job.WorkflowRun.getListener(WorkflowRun.java:219)
                    at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$300(WorkflowRun.java:133)
                    at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.getListener(WorkflowRun.java:955)
                    at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getListener(EnvActionImpl.java:77)
                    at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getEnvironment(EnvActionImpl.java:69)
                    at org.jenkinsci.plugin.taboola.ElasticSearchManager.getConsoleEnvs(ElasticSearchManager.java:277)
                    at org.jenkinsci.plugin.taboola.ElasticSearchManager.addCommonDataAndSend(ElasticSearchManager.java:110)
                    at org.jenkinsci.plugin.taboola.ElasticSearchManager.sendStageData(ElasticSearchManager.java:179)
                    at org.jenkinsci.plugin.taboola.events.TaboolaStagesListener.onNewHead(TaboolaStagesListener.java:145)
                    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463)
                    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458)
                    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:37)
                    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
                    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
                    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                    at java.lang.Thread.run(Thread.java:748)
            
            
            Show
            hellspam Roy Arnon added a comment - Yes, I do see a stack trace like that in the log: Aug 27, 2019 7:37:09 PM org.jenkinsci.plugins.workflow.job.WorkflowRun getListener WARNING: null java.lang.IllegalStateException: trying to open a build log on Lint.Hooks #1923 after it has completed at org.jenkinsci.plugins.workflow.job.WorkflowRun.getListener(WorkflowRun.java:219) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$300(WorkflowRun.java:133) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.getListener(WorkflowRun.java:955) at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getListener(EnvActionImpl.java:77) at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getEnvironment(EnvActionImpl.java:69) at org.jenkinsci.plugin.taboola.ElasticSearchManager.getConsoleEnvs(ElasticSearchManager.java:277) at org.jenkinsci.plugin.taboola.ElasticSearchManager.addCommonDataAndSend(ElasticSearchManager.java:110) at org.jenkinsci.plugin.taboola.ElasticSearchManager.sendStageData(ElasticSearchManager.java:179) at org.jenkinsci.plugin.taboola.events.TaboolaStagesListener.onNewHead(TaboolaStagesListener.java:145) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:37) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748)
            Hide
            hellspam Roy Arnon added a comment -

            Actually, taking a look at it, it seems it has to do with one of our own plugins - I will look into that.

            Show
            hellspam Roy Arnon added a comment - Actually, taking a look at it, it seems it has to do with one of our own plugins - I will look into that.
            Hide
            jglick Jesse Glick added a comment -

            workflow-job #140 should suppress the stack trace, which at least in this case seems to be innocuous.

            Show
            jglick Jesse Glick added a comment - workflow-job #140 should suppress the stack trace, which at least in this case seems to be innocuous.
            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            Ok, at least one way that this could be happening should be fixed as of Pipeline: Job Plugin 2.35. If you end up finding this ticket when running Pipeline: Job Plugin 2.35 or higher because you are seeing the following error in your logs, please file a new ticket, and include the full stack trace:

            java.lang.IllegalStateException: trying to open a build log on … after it has completed
            

            EDIT: Note that if you are not running Pipeline: Groovy Plugin 2.74 or later, you will see this error message for one case that has already been fixed, so I would recommend upgrading to Pipeline: Groovy Plugin 2.74 or later.

            Show
            dnusbaum Devin Nusbaum added a comment - - edited Ok, at least one way that this could be happening should be fixed as of Pipeline: Job Plugin 2.35. If you end up finding this ticket when running Pipeline: Job Plugin 2.35 or higher because you are seeing the following error in your logs, please file a new ticket, and include the full stack trace: java.lang.IllegalStateException: trying to open a build log on … after it has completed EDIT: Note that if you are not running Pipeline: Groovy Plugin 2.74 or later, you will see this error message for one case that has already been fixed, so I would recommend upgrading to Pipeline: Groovy Plugin 2.74 or later.
            Hide
            jglick Jesse Glick added a comment -

            upgrading to Pipeline: Job Plugin 2.35 or newer which will automatically update Pipeline: Groovy Plugin as well

            No it will not—this is just a test dependency.

            Plugin-Dependencies: workflow-api:2.36,workflow-step-api:2.20,workflow-support:3.3
            
            Show
            jglick Jesse Glick added a comment - upgrading to Pipeline: Job Plugin 2.35 or newer which will automatically update Pipeline: Groovy Plugin as well No it will not—this is just a test dependency. Plugin-Dependencies: workflow-api:2.36,workflow-step-api:2.20,workflow-support:3.3

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                hellspam Roy Arnon
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: