Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47192

RFC: Compress pipeline stage logs?

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We are extensively using multibranch pipelines to build our system and we're struggling to manage the sheer quantity of build logs generated.

      On inspection we're seeing roughly 250M of logs (uncompressed) for every build. The compress-build-log-plugin would seem to be the answer and indeed having added this to the properties of the pipeline the log file is compressed to about 2.5M. But, the individual stage logs are not compressed and so are consuming roughly 125M as the logs are entirely replicated so that the pipeline UI can show logs for each step.

      Should the compress-build-log-plugin be extended to also find all the *.log files and compress them as well? Not sure what the pipeline UI would think to this though.

        Attachments

          Issue Links

            Activity

            Hide
            danielbeck Daniel Beck added a comment -

            Pipeline doesn't support transparently decompressing compressed logs in the fly. That needs to be added first, otherwise there's no point in gzipping the logs.

            Seems to be around LogActionImpl in workflow-support.

            Show
            danielbeck Daniel Beck added a comment - Pipeline doesn't support transparently decompressing compressed logs in the fly. That needs to be added first, otherwise there's no point in gzipping the logs. Seems to be around LogActionImpl in workflow-support.
            Hide
            marksyms Mark Syms added a comment -

            I thought that might be the case. Worth doing though?

            Show
            marksyms Mark Syms added a comment - I thought that might be the case. Worth doing though?
            Hide
            fajran Fajran Rusadi added a comment -

            Hi, I would also like to have this feature implemented. It would save me a ton of storage space if I could compress the log files and still able to read them.

            I did try to change the LogActionImpl class and it worked on my local test – https://github.com/fajran/workflow-support-plugin/tree/read-compressed-log This one was built on top of workflow-support-2.16.

            Do you think it is a good idea to submit a pull request? Looks like there are some works being done related to the logging.

            Show
            fajran Fajran Rusadi added a comment - Hi, I would also like to have this feature implemented. It would save me a ton of storage space if I could compress the log files and still able to read them. I did try to change the LogActionImpl class and it worked on my local test – https://github.com/fajran/workflow-support-plugin/tree/read-compressed-log This one was built on top of workflow-support-2.16. Do you think it is a good idea to submit a pull request? Looks like there are some works being done related to the logging.
            Hide
            jglick Jesse Glick added a comment -
            Show
            jglick Jesse Glick added a comment - See JENKINS-38381 .

              People

              • Assignee:
                Unassigned
                Reporter:
                marksyms Mark Syms
              • Votes:
                2 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: