Details

    • Type: Improvement
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: other
    • Labels:
    • Environment:
      Platform: All, OS: All
    • Similar Issues:

      Description

      Old builds can take up a significant amount of space on disk, even though they
      often take up very little space when tarred and zipped. For example, I have a
      build setup where the archived build output takes up 164MB, but can be tarred
      and zipped down to 8MB. This problem is most prevalent when there are lots of
      small files being archived, as with JUnit and Javadoc output.

      In the mailing list, the argument against doing this is:

      https://hudson.dev.java.net/servlets/ReadMsg?list=users&msgNo=10279

      "In most of the situations, I don't think it makes much sense for Hudson
      to do such desperate disk conservation measure. It's much cheaper to fix
      the problem by throwing more HDDs at the problem, and on modern file
      systems, you can have file-system level compression, too."

      Throwing more HDDs is a good solution in many cases, but here it can often be a
      20X win to compress the output. That means a 20X difference in the number of
      HDDs required.

      The problem with file-system level compression is that it will typically also
      apply to the workspace folder, which can have a nontrivial impact on build
      performance.

        Attachments

          Issue Links

            Activity

            Hide
            danielbeck Daniel Beck added a comment -

            I wrote a plugin that basically does what is requested here:

            https://github.com/daniel-beck/compress-buildlog-plugin

            It'll gzip the build log after the build completes if a checkbox is enabled on the job config page.

            It's very bare bones, but for some of you this might be sufficient.

            Show
            danielbeck Daniel Beck added a comment - I wrote a plugin that basically does what is requested here: https://github.com/daniel-beck/compress-buildlog-plugin It'll gzip the build log after the build completes if a checkbox is enabled on the job config page. It's very bare bones, but for some of you this might be sufficient.
            Hide
            jglick Jesse Glick added a comment -

            Sounds like that would suffice. Nice work!

            Show
            jglick Jesse Glick added a comment - Sounds like that would suffice. Nice work!
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            The plugin has not been released yet

            Show
            oleg_nenashev Oleg Nenashev added a comment - The plugin has not been released yet
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Forked and released Daniel's plugin

            Show
            oleg_nenashev Oleg Nenashev added a comment - Forked and released Daniel's plugin
            Hide
            marc321 Mr Cinquero added a comment - - edited

            The plugin disables the console tail functionality. And it does not compress any leftover logs. And you need to enable it by hand for each single job.

            Best solution so far (on Linux boxes): set up a daily script job:

            set -Eex
            cd "$JENKINS_HOME/jobs"
            find * -regex '^\([^/]+/builds/[0-9]+/log\|[^/]+/modules/[^/]+/builds/[0-9]+/log\)$' -cmin +600 -exec gzip -9v '{}' \;
            

            This compresses all uncompressed build logs which haven't been touched in the last 10 hours (to avoid processing active logs).

            Show
            marc321 Mr Cinquero added a comment - - edited The plugin disables the console tail functionality. And it does not compress any leftover logs. And you need to enable it by hand for each single job. Best solution so far (on Linux boxes): set up a daily script job: set -Eex cd "$JENKINS_HOME/jobs" find * -regex '^\([^/]+/builds/[0-9]+/log\|[^/]+/modules/[^/]+/builds/[0-9]+/log\)$' -cmin +600 -exec gzip -9v '{}' \; This compresses all uncompressed build logs which haven't been touched in the last 10 hours (to avoid processing active logs).

              People

              • Assignee:
                Unassigned
                Reporter:
                plinehan plinehan
              • Votes:
                21 Vote for this issue
                Watchers:
                19 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: