Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38381

[JEP-210] Optimize log handling in Pipeline and Durable Task

    XMLWordPrintable

    Details

    • Sprint:
      Pipeline - October, Pipeline - April 2018
    • Similar Issues:

      Description

      Extracted from JENKINS-30896:

      The current design of LogActionImpl, using $id.log, was considered the minimum necessary for a working 1.0 release, not a serious implementation. It has a major problem: when there is a large amount of output, WorkflowRun.copyLogs must duplicate it all to log, doubling disk space requirements per build.

      It would be better to keep a single log file for the build. LogActionImpl should deprecated in favor of an implementation that simply stores a rangeset of offsets into that file. When parallel blocks are producing concurrent output, the single log file will be a bit jumbled (probably still human-readable in most cases), but the rangesets will keep track of what output came from where. The final output produced by WorkflowRun will still be processed to split at line boundaries, add in thread labels, etc. (TBD how and whether JENKINS-30777 could be supported in this mode.)

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            Released in beta.

            Show
            jglick Jesse Glick added a comment - Released in beta.
            Hide
            jglick Jesse Glick added a comment -

            I filed a JEP submission for this.

            Show
            jglick Jesse Glick added a comment - I filed a JEP submission for this.
            Hide
            jglick Jesse Glick added a comment -

            Hoping for some review activity soon.

            Show
            jglick Jesse Glick added a comment - Hoping for some review activity soon.
            Hide
            webrat Andreas Sieferlinger added a comment -

            any update on this? its currently marked as in progress.

            Show
            webrat Andreas Sieferlinger added a comment - any update on this? its currently marked as in progress.
            Hide
            jglick Jesse Glick added a comment -

            Yesterday I redesigned the API to more clearly separate layers of responsibility and allow cloud-based implementations to perform better. I have abandoned the logstash PoC implementation.

            Show
            jglick Jesse Glick added a comment - Yesterday I redesigned the API to more clearly separate layers of responsibility and allow cloud-based implementations to perform better. I have abandoned the logstash PoC implementation.

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                jglick Jesse Glick
              • Votes:
                35 Vote for this issue
                Watchers:
                81 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: