Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48344

Log files generated by Jenkins pipeline scripts are bloated

    Details

    • Similar Issues:
    • Released As:
      1.9

      Description

      We recently had one of our production Jenkins masters nearly run out of disk space due to an over-allocation of log data by just a couple of jobs running on our farm. In fact, 2 such builds consumed nearly a half a terabyte for their build logs alone! Further, closer examination of those log files revealed that the data therein was being excessively bloated by what appears to be some sort of markup or metadata attached to each line in the build log that looks something like this:

       

      ^[[8mha:////4OsXDMICFQEJVGu5QN07bZyJAnhssncpc0tH8m8uvnrSAAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA^[[0m[Pipeline] node

       

      To illustrate the problem I created a super trivial pipeline script as follows:

      node {
        stage('first') {
          echo "hello world"
        }
      }
      

      This simple example produced 5kb worth of build logs! To make matters worse, it appears as though the raw text (ie: excluding the markup) is also duplicated among several other .log files in the same folder as the main build log, causing even further bloat.

      I am creating this issue in the hopes that (a) someone can explain what this extra log metadata is in the main build log of a pipeline build and (b) someone can suggest some way to either eliminate this bloat and superfluous duplication, or to at least offer some way to detect large build logs and perhaps truncate, purge or even prevent them by causing the build to fail. Really anything other than filling up the Jenkins home folder would be preferable.

        Attachments

          Issue Links

            Activity

            leedega Kevin Phillips created issue -
            leedega Kevin Phillips made changes -
            Field Original Value New Value
            Comment [ Note: I did find the build log compression plugin found [here|https://plugins.jenkins.io/compress-buildlog] which sounds encouraging, but given that it changes the actual format of the log file itself it risks breaking other plugins that use the log. I haven't tested it out with our current production system but it seems overly fragile / risky to me. Also, it doesn't address the duplication of the independent .log files either apparently.  ]
            leedega Kevin Phillips made changes -
            Environment jenkins core 2.46.3
            pipeline v2.6
            leedega Kevin Phillips made changes -
            Environment jenkins core 2.46.3
            pipeline v2.6
            jenkins core 2.60.2
            pipeline v2.6
            leedega Kevin Phillips made changes -
            Environment jenkins core 2.60.2
            pipeline v2.6
            jenkins core v2.43.3, v2.60.2, v2.89.2
            pipeline v2.6
            leedega Kevin Phillips made changes -
            Attachment log [ 40709 ]
            leedega Kevin Phillips made changes -
            Attachment support_2017-12-15_14.zip [ 40710 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-38381 [ JENKINS-38381 ]
            jbochenski Jakub Bochenski made changes -
            Link This issue is blocked by JENKINS-44195 [ JENKINS-44195 ]
            jglick Jesse Glick made changes -
            Issue Type Improvement [ 4 ] Bug [ 1 ]
            Component/s timestamper-plugin [ 15749 ]
            Component/s pipeline [ 21692 ]
            Labels performance pipeline
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue relates to JENKINS-51229 [ JENKINS-51229 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-45693 [ JENKINS-45693 ]
            jglick Jesse Glick made changes -
            Assignee Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 25 (Web Link)" [ 21848 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-54081 [ JENKINS-54081 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-54133 [ JENKINS-54133 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-55257 [ JENKINS-55257 ]
            jglick Jesse Glick made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Released As 1.9
            jglick Jesse Glick made changes -
            Link This issue is blocking INFRA-2028 [ INFRA-2028 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-45661 [ JENKINS-45661 ]
            drulli Ulli Hafner made changes -
            Link This issue causes JENKINS-56484 [ JENKINS-56484 ]
            basil Basil Crow made changes -
            Link This issue relates to JENKINS-58102 [ JENKINS-58102 ]

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                leedega Kevin Phillips
              • Votes:
                11 Vote for this issue
                Watchers:
                29 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: