Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45057

"too many files open": file handles leak, job output file not closed

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Jenkins seems to keep a open file handle to the log file (job output) for every single build, even those who have been discarded by the "Discard old build policy".

       

      This is a sample of the lsof output (whole file attached)

      java 8870 jenkins 941w REG 252,0 1840 1332171 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/.50063/log (deleted)
      java 8870 jenkins 942w REG 252,0 2023 402006 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/.50044/log (deleted)
      java 8870 jenkins 943w REG 252,0 2193 1332217 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/50101/log
      java 8870 jenkins 944w REG 252,0 2512 1332247 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/50106/log
      java 8870 jenkins 945w REG 252,0 1840 1703994 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/.50067/log (deleted)
      java 8870 jenkins 946w REG 252,0 2350 1332230 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/.50092/log (deleted)
      java 8870 jenkins 947w REG 252,0 1840 402034 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/.50049/log (deleted)
      java 8870 jenkins 948w REG 252,0 1840 927855 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/.50080/log (deleted)
      java 8870 jenkins 949w REG 252,0 2195 1332245 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/.50095/log (deleted)
      java 8870 jenkins 950w REG 252,0 2326 1332249 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/50107/log
      java 8870 jenkins 952w REG 252,0 2195 1332227 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/50102/log
      java 8870 jenkins 953w REG 252,0 2154 1332254 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/50109/log
      java 8870 jenkins 954w REG 252,0 2356 1332282 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/50105/log
      

       

        Attachments

          Issue Links

            Activity

            bbonacci Bruno Bonacci created issue -
            jonasatwork Jonas Jonsson made changes -
            Field Original Value New Value
            Link This issue relates to JENKINS-42934 [ JENKINS-42934 ]
            jonasatwork Jonas Jonsson made changes -
            Component/s groovy-plugin [ 15549 ]
            adamleggo Adam Leggo made changes -
            Assignee Bruno Bonacci [ bbonacci ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-43199 [ JENKINS-43199 ]
            andreasmandel Andreas Mandel made changes -
            Labels lts
            olivergondza Oliver Gondža made changes -
            Labels lts lts-candidate
            stephenconnolly Stephen Connolly made changes -
            Attachment jenkins-45057.patch [ 39073 ]
            stephenconnolly Stephen Connolly made changes -
            stephenconnolly Stephen Connolly made changes -
            Link This issue is related to JENKINS-45903 [ JENKINS-45903 ]
            stephenconnolly Stephen Connolly made changes -
            Link This issue duplicates JENKINS-43199 [ JENKINS-43199 ]
            stephenconnolly Stephen Connolly made changes -
            Link This issue is related to JENKINS-43199 [ JENKINS-43199 ]
            oleg_nenashev Oleg Nenashev made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            oleg_nenashev Oleg Nenashev made changes -
            Assignee Bruno Bonacci [ bbonacci ] Jesse Glick [ jglick ]
            oleg_nenashev Oleg Nenashev made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is duplicated by JENKINS-43199 [ JENKINS-43199 ]
            jglick Jesse Glick made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            oleg_nenashev Oleg Nenashev made changes -
            Labels lts-candidate 2.60.3-rejected lts-candidate
            stevenatcisco Steven Christenson made changes -
            Attachment Jenkins_FileHandleLeak.png [ 39331 ]
            stevenatcisco Steven Christenson made changes -
            Comment [

            Above is the change in file handle usage after upgrading to CloudBees Jenkins Enterprise 2.60.2.2-rolling. Our workaround until the core version is released is to set _ulimit -n_ very large and reboot at least weekly.

            If a better interim solution is known, we'd love to hear it. ]
            stevenatcisco Steven Christenson made changes -
            stevenatcisco Steven Christenson made changes -
            Attachment Jenkins_FileHandleLeak.png [ 39331 ]
            danielbeck Daniel Beck made changes -
            Labels 2.60.3-rejected lts-candidate 2.60.3-rejected
            stevenatcisco Steven Christenson made changes -
            Attachment image-2017-10-06-14-57-46-970.png [ 39959 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-48280 [ JENKINS-48280 ]

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                bbonacci Bruno Bonacci
              • Votes:
                13 Vote for this issue
                Watchers:
                29 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: