Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38381

[JEP-210] Optimize log handling in Pipeline and Durable Task

    XMLWordPrintable

    Details

    • Sprint:
      Pipeline - October, Pipeline - April 2018
    • Similar Issues:

      Description

      Extracted from JENKINS-30896:

      The current design of LogActionImpl, using $id.log, was considered the minimum necessary for a working 1.0 release, not a serious implementation. It has a major problem: when there is a large amount of output, WorkflowRun.copyLogs must duplicate it all to log, doubling disk space requirements per build.

      It would be better to keep a single log file for the build. LogActionImpl should deprecated in favor of an implementation that simply stores a rangeset of offsets into that file. When parallel blocks are producing concurrent output, the single log file will be a bit jumbled (probably still human-readable in most cases), but the rangesets will keep track of what output came from where. The final output produced by WorkflowRun will still be processed to split at line boundaries, add in thread labels, etc. (TBD how and whether JENKINS-30777 could be supported in this mode.)

        Attachments

          Issue Links

            Activity

            jglick Jesse Glick created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Epic Link JENKINS-38313 [ 174485 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-30896 [ JENKINS-30896 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-30777 [ JENKINS-30777 ]
            jglick Jesse Glick made changes -
            Description See discussion in JENKINS-30896 about log files. Extracted from JENKINS-30896:

            The current design of {{LogActionImpl}}, using {{$id.log}}, was considered the minimum necessary for a working 1.0 release, not a serious implementation. It has a major problem: when there is a large amount of output, {{WorkflowRun.copyLogs}} must duplicate it all to {{log}}, doubling disk space requirements per build.

            It would be better to keep a single log file for the build. {{LogActionImpl}} should deprecated in favor of an implementation that simply stores a rangeset of offsets into that file. When {{parallel}} blocks are producing concurrent output, the single log file will be a bit jumbled (probably still human-readable in most cases), but the rangesets will keep track of what output came from where. The final output produced by {{WorkflowRun}} will still be processed to split at line boundaries, add in thread labels, etc. (TBD how and whether JENKINS-30777 could be supported in this mode.)
            jglick Jesse Glick made changes -
            Component/s durable-task-plugin [ 18622 ]
            Component/s workflow-durable-task-step-plugin [ 21715 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-31096 [ JENKINS-31096 ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-37575 [ JENKINS-37575 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-job PR 27 (Web Link)" [ 14876 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "jenkins-test-harness PR 37 (Web Link)" [ 14903 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-api PR 17 (Web Link)" [ 14904 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-support PR 15 (Web Link)" [ 14905 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "durable-task PR 29 (Web Link)" [ 14906 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-durable-task-step PR 21 (Web Link)" [ 14916 ]
            jglick Jesse Glick made changes -
            Component/s workflow-api-plugin [ 21711 ]
            oleg_nenashev Oleg Nenashev made changes -
            Summary Optimize log handling Optimize log handling in Pipeline and Durable Task
            jglick Jesse Glick made changes -
            Remote Link This issue links to "jenkins-test-harness PR 38 (Web Link)" [ 14924 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-27394 [ JENKINS-27394 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "logstash PR 18 (Web Link)" [ 14949 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "logstash PR 18 (Web Link)" [ 15104 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "logstash PR 18 (Web Link)" [ 14949 ]
            jglick Jesse Glick made changes -
            Link This issue relates to SECURITY-382 [ SECURITY-382 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-40613 [ JENKINS-40613 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-40934 [ JENKINS-40934 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-14435 [ JENKINS-14435 ]
            jglick Jesse Glick made changes -
            Link This issue is blocked by JENKINS-40934 [ JENKINS-40934 ]
            drulli Ulli Hafner made changes -
            Link This issue relates to JENKINS-32191 [ JENKINS-32191 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-40934 [ JENKINS-40934 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-44500 [ JENKINS-44500 ]
            jbochenski Jakub Bochenski made changes -
            Link This issue relates to JENKINS-44712 [ JENKINS-44712 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-44958 [ JENKINS-44958 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is blocking JENKINS-44483 [ JENKINS-44483 ]
            jamesdumay James Dumay made changes -
            Labels performance cloudbees-internal-pipeline performance
            jamesdumay James Dumay made changes -
            Component/s pipeline [ 21692 ]
            michaelneale Michael Neale made changes -
            Remote Link This issue links to "Some CB internal notes (Web Link)" [ 17289 ]
            michaelneale Michael Neale made changes -
            Status In Review [ 10005 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-45693 [ JENKINS-45693 ]
            jglick Jesse Glick made changes -
            Link This issue is blocking JENKINS-45553 [ JENKINS-45553 ]
            svanoort Sam Van Oort made changes -
            Epic Link JENKINS-38313 [ 174485 ] JENKINS-47170 [ 185575 ]
            jamesdumay James Dumay made changes -
            Sprint Pipeline - October [ 406 ]
            svanoort Sam Van Oort made changes -
            Assignee Jesse Glick [ jglick ] Sam Van Oort [ svanoort ]
            jamesdumay James Dumay made changes -
            Sprint Pipeline - October [ 406 ] Pipeline - October, Pipeline - December [ 406, 446 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-27035 [ JENKINS-27035 ]
            danielbeck Daniel Beck made changes -
            Link This issue blocks JENKINS-47192 [ JENKINS-47192 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-48344 [ JENKINS-48344 ]
            jglick Jesse Glick made changes -
            Link This issue blocks JENKINS-38313 [ JENKINS-38313 ]
            svanoort Sam Van Oort made changes -
            Labels cloudbees-internal-pipeline performance cloudbees-internal-pipeline performance project-cheetah
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal ARC-24 (Web Link)" [ 20158 ]
            vivek Vivek Pandey made changes -
            Sprint Pipeline - October, Pipeline - December [ 406, 446 ] Pipeline - October, Pipeline - March 2018 [ 406, 506 ]
            jbochenski Jakub Bochenski made changes -
            Link This issue is blocked by JENKINS-44195 [ JENKINS-44195 ]
            jbriden Jenn Briden made changes -
            Labels cloudbees-internal-pipeline performance project-cheetah blocked cloudbees-internal-pipeline performance project-cheetah
            jglick Jesse Glick made changes -
            Link This issue is blocked by JENKINS-52165 [ JENKINS-52165 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-37575 [ JENKINS-37575 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "durable-task PR 29 (Web Link)" [ 14906 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "JEP PR 153 (Web Link)" [ 21229 ]
            jglick Jesse Glick made changes -
            Assignee Sam Van Oort [ svanoort ] Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Link This issue is blocked by JENKINS-52729 [ JENKINS-52729 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "JEP PR 153 (Web Link)" [ 21229 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "JEP-210 (Web Link)" [ 21242 ]
            jglick Jesse Glick made changes -
            Summary Optimize log handling in Pipeline and Durable Task [JEP-210] Optimize log handling in Pipeline and Durable Task
            maxpayne15 Mukesh Suthar made changes -
            Rank Ranked higher
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            jglick Jesse Glick made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-54073 [ JENKINS-54073 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-54078 [ JENKINS-54078 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-54081 [ JENKINS-54081 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-54133 [ JENKINS-54133 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-54304 [ JENKINS-54304 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-54128 [ JENKINS-54128 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-41845 [ JENKINS-41845 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-54566 [ JENKINS-54566 ]

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                jglick Jesse Glick
              • Votes:
                35 Vote for this issue
                Watchers:
                81 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: