Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55287

Pipeline: Failure to load flow node: FlowNode was not found in storage for head

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Blocker
    • Resolution: Unresolved
    • Component/s: workflow-cps-plugin
    • Labels:
      None
    • Environment:
      Jenkins ver. 2.138.2 Pipeline: Groovy 2.61
    • Similar Issues:

      Description

      IMPORTANT: NOTE FROM A MAINTAINER:

      STOP!  YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS!

      (sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

      Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ($JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/) of the build that failed and upload it here along with the Jenkins system logs, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any messages in the Jenkins system logs that seem relevant, etc. Thanks!

       

      Creating placeholder flownodes because failed loading originals.
      java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
       at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
       at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
       at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
       at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
       at hudson.model.RunMap.retrieve(RunMap.java:225)
       at hudson.model.RunMap.retrieve(RunMap.java:57)
       at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
       at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
       at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
       at hudson.model.RunMap.getById(RunMap.java:205)
       at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
       at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)
       at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
       at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
       at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
       at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
       at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
       at jenkins.model.Jenkins.<init>(Jenkins.java:975)
       at hudson.model.Hudson.<init>(Hudson.java:85)
       at hudson.model.Hudson.<init>(Hudson.java:81)
       at hudson.WebAppMain$3.run(WebAppMain.java:233)
      Finished: FAILURE
      

        Attachments

          Issue Links

            Activity

            Hide
            bitwiseman Liam Newman added a comment -

            IMPORTANT: NOTE FROM A MAINTAINER:

            STOP!  YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS!

            (sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

            We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ($JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc. Thanks!

            Ramon Leon
            **Sorry, I don't think we can get that information from ci.jenkins.io - the infra team is overloaded and we need the zip of the build folder immediately (or very soon) after the error.

             

            Show
            bitwiseman Liam Newman added a comment - IMPORTANT: NOTE FROM A MAINTAINER: STOP!  YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS! (sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information) We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ( $JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/ ) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc. Thanks! Ramon Leon **Sorry, I don't think we can get that information from ci.jenkins.io - the infra team is overloaded and we need the zip of the build folder immediately (or very soon) after the error.  
            Hide
            rodrigc Craig Rodrigues added a comment - - edited

            Liam Newman I think a lot of pressure is being put on the end user to perform weird steps in order to help debug this problem. This is very hard to do. Since the problem
            is intermittent, it is hard to catch.

            Why don't you modify the pipeline plugin so at the point that the pipeline tries to throw an exception with message but FlowNode was not found in storage for head, the plugin zips up the entire job directory, and puts it in some place like $TEMPDIR ?
            You can release a version of the plugin with this change.
            That way the end user doesn't have to perform a lot of weird steps, but they can simply retrieve a zip file with all the stuff the is needed for the pipeline plugin maintainer to diagnose and root cause the problem?

            Show
            rodrigc Craig Rodrigues added a comment - - edited Liam Newman I think a lot of pressure is being put on the end user to perform weird steps in order to help debug this problem. This is very hard to do. Since the problem is intermittent, it is hard to catch. Why don't you modify the pipeline plugin so at the point that the pipeline tries to throw an exception with message but FlowNode was not found in storage for head , the plugin zips up the entire job directory, and puts it in some place like $TEMPDIR ? You can release a version of the plugin with this change. That way the end user doesn't have to perform a lot of weird steps, but they can simply retrieve a zip file with all the stuff the is needed for the pipeline plugin maintainer to diagnose and root cause the problem?
            Hide
            portchavz Steven Chaves added a comment -

            I don't have rights to zip my Build Job folder.

            Show
            portchavz Steven Chaves added a comment - I don't have rights to zip my Build Job folder.
            Hide
            famod Falko Modler added a comment - - edited

            Devin Nusbaum & Liam Newman

            I have three zipped build folders (all from around the same time, March 12th) but did not yet have the time to redact sensitive data. Do you have some hints what and where to look for (regarding sensitive data)?

            Show
            famod Falko Modler added a comment - - edited Devin Nusbaum & Liam Newman I have three zipped build folders (all from around the same time, March 12th) but did not yet have the time to redact sensitive data. Do you have some hints what and where to look for (regarding sensitive data)?
            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            I was looking into this issue recently and wanted to check whether the code that saves Pipelines using the PERFORMANCE_OPTIMIZED durability level when Jenkins is shutting down (CpsFlowExecution.suspendAll) is running successfully in these cases to make sure that the problem isn't something like the service configuration for Jenkins not giving it enough time to shut down.

            For users seeing the problem, please check the Jenkins system logs for the shutdown that occurred before you saw this error. Do you see messages similar to these, in particular for the build that failed with the exception in the description of this ticket?

            • About to try to checkpoint the program for buildCpsFlowExecution[Owner[YourJobName/BuildNumber:YourJobName #BuildNumber]]
            • Trying to save program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash
            • Finished saving program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash

            Those messages should come in groups of three, in that order, and are logged at INFO level, and there should not be any WARNING messages interspersed between them or surrounding them. There should be a set of those messages for each Pipeline running using the PERFORMANCE_OPTIMIZED durability level when Jenkins shuts down. If you have configured Jenkins to run as a service, I would also check the service logs to see if Jenkins has been shutting down successfully or if there are any warnings/errors when Jenkins shuts down.

            Falko Modler It depends on what your builds are doing and what you consider sensitive. The main files to check are build.xml, which contains a textual form of the Pipeline, and log, which contains the build logs. You might want to check for internal hostnames or IP addresses, unmasked passwords or API keys in your build log, etc.

            Show
            dnusbaum Devin Nusbaum added a comment - - edited I was looking into this issue recently and wanted to check whether the code that saves Pipelines using the PERFORMANCE_OPTIMIZED durability level when Jenkins is shutting down ( CpsFlowExecution.suspendAll ) is running successfully in these cases to make sure that the problem isn't something like the service configuration for Jenkins not giving it enough time to shut down. For users seeing the problem, please check the Jenkins system logs for the shutdown that occurred before you saw this error. Do you see messages similar to these, in particular for the build that failed with the exception in the description of this ticket? About to try to checkpoint the program for buildCpsFlowExecution[Owner[YourJobName/BuildNumber:YourJobName #BuildNumber]] Trying to save program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash Finished saving program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash Those messages should come in groups of three, in that order, and are logged at INFO level, and there should not be any WARNING messages interspersed between them or surrounding them. There should be a set of those messages for each Pipeline running using the PERFORMANCE_OPTIMIZED durability level when Jenkins shuts down. If you have configured Jenkins to run as a service, I would also check the service logs to see if Jenkins has been shutting down successfully or if there are any warnings/errors when Jenkins shuts down. Falko Modler It depends on what your builds are doing and what you consider sensitive. The main files to check are build.xml , which contains a textual form of the Pipeline, and log , which contains the build logs. You might want to check for internal hostnames or IP addresses, unmasked passwords or API keys in your build log, etc.

              People

              • Assignee:
                Unassigned
                Reporter:
                haorui658 Rui Hao
              • Votes:
                29 Vote for this issue
                Watchers:
                41 Start watching this issue

                Dates

                • Created:
                  Updated: