Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55287

Pipeline: Failure to load flow node: FlowNode was not found in storage for head

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Blocker
    • Resolution: Unresolved
    • Component/s: workflow-cps-plugin
    • Labels:
      None
    • Environment:
      Jenkins ver. 2.138.2 Pipeline: Groovy 2.61
    • Similar Issues:

      Description

      IMPORTANT: NOTE FROM A MAINTAINER:

      STOP! YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS!

      (sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

      Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ($JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/) of the build that failed and upload it here along with the Jenkins system logs, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any messages in the Jenkins system logs that seem relevant, etc. In addition, please check service or other system level logs for Jenkins to see if there are any issues with Jenkins taking too long to shut down or anything like that. Thanks!

      The main thing we are currently looking for is whether these messages are present in the Jenkins logs right before Jenkins shut down for the build which has the error:

      • About to try to checkpoint the program for buildCpsFlowExecutionOwner[YourJobName/BuildNumber:YourJobName #BuildNumber]]
      • Trying to save program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash
      • Finished saving program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash

      If these messages are not present, it means that Jenkins was unable to save the Pipeline, so the error is expected. If that is the case, fixing the issue probably requires changes to Jenkins packaging to configure longer service timeouts on shutdown, or totally changing how PERFORMANCE_OPTIMIZED works. If the messages are present, then something else is happening.

      Exception:

      Creating placeholder flownodes because failed loading originals.
      java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
       at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
       at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
       at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
       at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
       at hudson.model.RunMap.retrieve(RunMap.java:225)
       at hudson.model.RunMap.retrieve(RunMap.java:57)
       at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
       at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
       at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
       at hudson.model.RunMap.getById(RunMap.java:205)
       at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
       at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)
       at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
       at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
       at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
       at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
       at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
       at jenkins.model.Jenkins.<init>(Jenkins.java:975)
       at hudson.model.Hudson.<init>(Hudson.java:85)
       at hudson.model.Hudson.<init>(Hudson.java:81)
       at hudson.WebAppMain$3.run(WebAppMain.java:233)
      Finished: FAILURE
      

        Attachments

          Issue Links

            Activity

            Hide
            famod Falko Modler added a comment - - edited

            Devin Nusbaum & Liam Newman

            I have three zipped build folders (all from around the same time, March 12th) but did not yet have the time to redact sensitive data. Do you have some hints what and where to look for (regarding sensitive data)?

            Show
            famod Falko Modler added a comment - - edited Devin Nusbaum & Liam Newman I have three zipped build folders (all from around the same time, March 12th) but did not yet have the time to redact sensitive data. Do you have some hints what and where to look for (regarding sensitive data)?
            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            I was looking into this issue recently and wanted to check whether the code that saves Pipelines using the PERFORMANCE_OPTIMIZED durability level when Jenkins is shutting down (CpsFlowExecution.suspendAll) is running successfully in these cases to make sure that the problem isn't something like the service configuration for Jenkins not giving it enough time to shut down.

            For users seeing the problem, please check the Jenkins system logs for the shutdown that occurred before you saw this error. Do you see messages similar to these, in particular for the build that failed with the exception in the description of this ticket?

            • About to try to checkpoint the program for buildCpsFlowExecution[Owner[YourJobName/BuildNumber:YourJobName #BuildNumber]]
            • Trying to save program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash
            • Finished saving program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash

            Those messages should come in groups of three, in that order, and are logged at INFO level, and there should not be any WARNING messages interspersed between them or surrounding them. There should be a set of those messages for each Pipeline running using the PERFORMANCE_OPTIMIZED durability level when Jenkins shuts down. If you have configured Jenkins to run as a service, I would also check the service logs to see if Jenkins has been shutting down successfully or if there are any warnings/errors when Jenkins shuts down.

            Falko Modler It depends on what your builds are doing and what you consider sensitive. The main files to check are build.xml, which contains a textual form of the Pipeline, and log, which contains the build logs. You might want to check for internal hostnames or IP addresses, unmasked passwords or API keys in your build log, etc.

            Show
            dnusbaum Devin Nusbaum added a comment - - edited I was looking into this issue recently and wanted to check whether the code that saves Pipelines using the PERFORMANCE_OPTIMIZED durability level when Jenkins is shutting down ( CpsFlowExecution.suspendAll ) is running successfully in these cases to make sure that the problem isn't something like the service configuration for Jenkins not giving it enough time to shut down. For users seeing the problem, please check the Jenkins system logs for the shutdown that occurred before you saw this error. Do you see messages similar to these, in particular for the build that failed with the exception in the description of this ticket? About to try to checkpoint the program for buildCpsFlowExecution[Owner[YourJobName/BuildNumber:YourJobName #BuildNumber]] Trying to save program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash Finished saving program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash Those messages should come in groups of three, in that order, and are logged at INFO level, and there should not be any WARNING messages interspersed between them or surrounding them. There should be a set of those messages for each Pipeline running using the PERFORMANCE_OPTIMIZED durability level when Jenkins shuts down. If you have configured Jenkins to run as a service, I would also check the service logs to see if Jenkins has been shutting down successfully or if there are any warnings/errors when Jenkins shuts down. Falko Modler It depends on what your builds are doing and what you consider sensitive. The main files to check are build.xml , which contains a textual form of the Pipeline, and log , which contains the build logs. You might want to check for internal hostnames or IP addresses, unmasked passwords or API keys in your build log, etc.
            Hide
            famod Falko Modler added a comment -

            Devin Nusbaum I am in the process of redacting sensitive content. I will have to remove some files and folders. Is there anything useful for you in workspace-files?

            Can I send you the archive in a less public way somehow?

            Show
            famod Falko Modler added a comment - Devin Nusbaum I am in the process of redacting sensitive content. I will have to remove some files and folders. Is there anything useful for you in workspace-files ? Can I send you the archive in a less public way somehow?
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Restored the priority set by the reporters

            Show
            oleg_nenashev Oleg Nenashev added a comment - Restored the priority set by the reporters
            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            Falko Modler You can email it to me: dnusbaum at cloudbees.com. And no, I don't need to see workspace files. The main thing I want to see is the Jenkins system logs and or/service logs around the time of the last Jenkins restart to see if the build in question was able to be saved before the shutdown or not.

            Show
            dnusbaum Devin Nusbaum added a comment - - edited Falko Modler You can email it to me: dnusbaum at cloudbees.com. And no, I don't need to see workspace files. The main thing I want to see is the Jenkins system logs and or/service logs around the time of the last Jenkins restart to see if the build in question was able to be saved before the shutdown or not.

              People

              • Assignee:
                Unassigned
                Reporter:
                haorui658 Rui Hao
              • Votes:
                32 Vote for this issue
                Watchers:
                44 Start watching this issue

                Dates

                • Created:
                  Updated: