Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55923

CpsFlowExecution writes to logs after compression

    Details

    • Type: Bug
    • Status: Fixed but Unreleased (View Workflow)
    • Priority: Critical
    • Resolution: Incomplete
    • Component/s: workflow-cps-plugin
    • Labels:
      None
    • Environment:
      Will edit with additional details once authorized to disclose.
      Jenkins: 2.121.1
      Pipeline 2.5
    • Similar Issues:

      Description

      We've encountered occasional hung threads living far longer than their jobs, causing system instability. Root cause is that after build logs are compressed, an additional line is appended, 'Creating placeholder flownodes because failed loading originals.', which corrupts the gz archive. If we remove the appended line, the log can be extracted.

      The workaround is to move the build folder on the master, kill any remaining threads, and often we must reboot the master. This has happened multiple times so far, and we've setup thread duration monitoring jobs to detect threads & builds over X ms. Advice on additional ways of capturing relevant log information would be appreciated.

      The only place I've found the offending line is:
      https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L640

      Closest existing issue was: https://issues.jenkins-ci.org/browse/JENKINS-50199?jql=text%20~%20%22Creating%20placeholder%20flownodes%20because%20failed%20loading%20originals.%22

        Attachments

          Activity

          steven_williams Steven Williams created issue -
          steven_williams Steven Williams made changes -
          Field Original Value New Value
          Description We've encountered occasional hung threads living far longer than their jobs, causing system instability. Root cause is that after build logs are compressed, and additional line is appended, 'Creating placeholder flownodes because failed loading originals.', which corrupts the gz archive. If we remove the appended line, the log could be extracted.

          The workaround is to move the build folder on the master, kill any remaining threads, and often we must reboot the master. This has happened multiple times so far, and we've setup thread duration monitoring jobs to detect threads & builds over X ms. Advice on additional ways of capturing relevant log information would be appreciated.

          The only place I've found the offending line is:
          [https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L640]

          Closest existing issue was: https://issues.jenkins-ci.org/browse/JENKINS-50199?jql=text%20~%20%22Creating%20placeholder%20flownodes%20because%20failed%20loading%20originals.%22
          We've encountered occasional hung threads living far longer than their jobs, causing system instability. Root cause is that after build logs are compressed, an additional line is appended, 'Creating placeholder flownodes because failed loading originals.', which corrupts the gz archive. If we remove the appended line, the log could be extracted.

          The workaround is to move the build folder on the master, kill any remaining threads, and often we must reboot the master. This has happened multiple times so far, and we've setup thread duration monitoring jobs to detect threads & builds over X ms. Advice on additional ways of capturing relevant log information would be appreciated.

          The only place I've found the offending line is:
           [https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L640]

          Closest existing issue was: https://issues.jenkins-ci.org/browse/JENKINS-50199?jql=text%20~%20%22Creating%20placeholder%20flownodes%20because%20failed%20loading%20originals.%22
          steven_williams Steven Williams made changes -
          Description We've encountered occasional hung threads living far longer than their jobs, causing system instability. Root cause is that after build logs are compressed, an additional line is appended, 'Creating placeholder flownodes because failed loading originals.', which corrupts the gz archive. If we remove the appended line, the log could be extracted.

          The workaround is to move the build folder on the master, kill any remaining threads, and often we must reboot the master. This has happened multiple times so far, and we've setup thread duration monitoring jobs to detect threads & builds over X ms. Advice on additional ways of capturing relevant log information would be appreciated.

          The only place I've found the offending line is:
           [https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L640]

          Closest existing issue was: https://issues.jenkins-ci.org/browse/JENKINS-50199?jql=text%20~%20%22Creating%20placeholder%20flownodes%20because%20failed%20loading%20originals.%22
          We've encountered occasional hung threads living far longer than their jobs, causing system instability. Root cause is that after build logs are compressed, an additional line is appended, 'Creating placeholder flownodes because failed loading originals.', which corrupts the gz archive. If we remove the appended line, the log can be extracted.

          The workaround is to move the build folder on the master, kill any remaining threads, and often we must reboot the master. This has happened multiple times so far, and we've setup thread duration monitoring jobs to detect threads & builds over X ms. Advice on additional ways of capturing relevant log information would be appreciated.

          The only place I've found the offending line is:
           [https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L640]

          Closest existing issue was: https://issues.jenkins-ci.org/browse/JENKINS-50199?jql=text%20~%20%22Creating%20placeholder%20flownodes%20because%20failed%20loading%20originals.%22
          Hide
          steven_williams Steven Williams added a comment -

          This may have the same root cause as https://issues.jenkins-ci.org/browse/JENKINS-50199
          We are on a slightly older version of the Pipeline Job plugin than the fix. Will test and soak on 2.25+

          Show
          steven_williams Steven Williams added a comment - This may have the same root cause as  https://issues.jenkins-ci.org/browse/JENKINS-50199 We are on a slightly older version of the Pipeline Job plugin than the fix. Will test and soak on 2.25+
          steven_williams Steven Williams made changes -
          Status Open [ 1 ] Fixed but Unreleased [ 10203 ]
          Resolution Incomplete [ 4 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              steven_williams Steven Williams
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: