Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48344

Log files generated by Jenkins pipeline scripts are bloated

    Details

    • Similar Issues:
    • Released As:
      1.9

      Description

      We recently had one of our production Jenkins masters nearly run out of disk space due to an over-allocation of log data by just a couple of jobs running on our farm. In fact, 2 such builds consumed nearly a half a terabyte for their build logs alone! Further, closer examination of those log files revealed that the data therein was being excessively bloated by what appears to be some sort of markup or metadata attached to each line in the build log that looks something like this:

       

      ^[[8mha:////4OsXDMICFQEJVGu5QN07bZyJAnhssncpc0tH8m8uvnrSAAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA^[[0m[Pipeline] node

       

      To illustrate the problem I created a super trivial pipeline script as follows:

      node {
        stage('first') {
          echo "hello world"
        }
      }
      

      This simple example produced 5kb worth of build logs! To make matters worse, it appears as though the raw text (ie: excluding the markup) is also duplicated among several other .log files in the same folder as the main build log, causing even further bloat.

      I am creating this issue in the hopes that (a) someone can explain what this extra log metadata is in the main build log of a pipeline build and (b) someone can suggest some way to either eliminate this bloat and superfluous duplication, or to at least offer some way to detect large build logs and perhaps truncate, purge or even prevent them by causing the build to fail. Really anything other than filling up the Jenkins home folder would be preferable.

        Attachments

          Issue Links

            Activity

            Hide
            medianick Nick Jones added a comment - - edited

            Running the custom Timestamper plugin build in Jesse Glick's previous comment, on our test Jenkins instance, with a timestamps block in a Scripted Pipeline job but with the global "Enabled for all Pipeline builds" checkbox checked, I'm seeing 6681 "deprecated*.log" files – each the same size as the "log" file (2328MB) – created on our master node by a single build, all within a few minutes of each other. This is consuming 14GB+ of space for this one build, which would be catastrophic for disk space if we applied the same updates to our production Jenkins instance (where there are hundreds of jobs building hundreds of times per day).

            This timestamps block is still in place because our production Jenkins system hasn't had its plugins changed, and uses the same Jenkinsfile for this particular job, and with the hundreds of jobs we have in place, coordinating the removal of a timestamps block in GitHub across all these Jenkinsfiles with the upgrade of the Timestamper (and/or other plugins) would be exceptionally difficult. I'm assuming (perhaps wrongly) that these "deprecated" files are being generated by some interaction between this block and the newer plugin(s) in this system, but perhaps it's something else.

            Is the only way forward to remove all the timestamps blocks or options from all our Jenkinsfiles first, then upgrading these plugins? Or is something else going on?

            Show
            medianick Nick Jones added a comment - - edited Running the custom Timestamper plugin build in Jesse Glick 's previous comment, on our test Jenkins instance, with a timestamps block in a Scripted Pipeline job but with the global "Enabled for all Pipeline builds" checkbox checked, I'm seeing 6681 "deprecated*.log" files – each the same size as the "log" file (2328MB) – created on our master node by a single build , all within a few minutes of each other. This is consuming 14GB+ of space for this one build, which would be catastrophic for disk space if we applied the same updates to our production Jenkins instance (where there are hundreds of jobs building hundreds of times per day). This timestamps block is still in place because our production Jenkins system hasn't had its plugins changed, and uses the same Jenkinsfile for this particular job, and with the hundreds of jobs we have in place, coordinating the removal of a timestamps block in GitHub across all these Jenkinsfiles with the upgrade of the Timestamper (and/or other plugins) would be exceptionally difficult. I'm assuming (perhaps wrongly) that these "deprecated" files are being generated by some interaction between this block and the newer plugin(s) in this system, but perhaps it's something else. Is the only way forward to remove all the timestamps blocks or options from all our Jenkinsfiles first, then upgrading these plugins? Or is something else going on?
            Hide
            jglick Jesse Glick added a comment -

            Nick Jones no, with the global setting enabled the timestamps step/option does nothing besides print a warning that it is doing nothing. Your issue is some instance of JENKINS-54128, with the exact cause visible by checking the stack trace in your system log.

            Show
            jglick Jesse Glick added a comment - Nick Jones no, with the global setting enabled the timestamps step/option does nothing besides print a warning that it is doing nothing. Your issue is some instance of JENKINS-54128 , with the exact cause visible by checking the stack trace in your system log.
            Hide
            drulli Ulli Hafner added a comment -

            Jesse Glick I just came around this issue due to JENKINS-56484. Before Timestamper 1.9 I used ConsoleNote::removeNotes to get rid of additional console decorations when parsing warnings. This does obviously not work anymore. Is there a general way to remove the timestamper plugin output from the console log?

            Show
            drulli Ulli Hafner added a comment - Jesse Glick I just came around this issue due to JENKINS-56484 . Before Timestamper 1.9 I used ConsoleNote::removeNotes to get rid of additional console decorations when parsing warnings. This does obviously not work anymore. Is there a general way to remove the timestamper plugin output from the console log?
            Hide
            jglick Jesse Glick added a comment -

            Better to discuss there.

            Show
            jglick Jesse Glick added a comment - Better to discuss there.
            Hide
            timur87 Tim Oz added a comment -

            Please, help

            I have the same issue on Jenkins ver. 2.176.1, Timestamper 1.10 and AnsiColor 0.6.2 plugins.

            Show
            timur87 Tim Oz added a comment - Please, help I have the same issue on Jenkins ver. 2.176.1, Timestamper 1.10 and AnsiColor 0.6.2 plugins.

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                leedega Kevin Phillips
              • Votes:
                11 Vote for this issue
                Watchers:
                29 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: