Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48280

File handle leak

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: core
    • Labels:
      None
    • Environment:
      Jenkins 2.73.3
    • Similar Issues:

      Description

      After upgrade to 2.73.3 we noticed that after a few days Jenkins becomes unresponsive and gives "too many open files" error. The only way to overcome the issue is to restart the instance.

      The File Leak Detector Plugin does show some open handles for completed jobs but after a few hours those are pruned by themselves from the list. The total amount of open file handles never exceeds 2000 according to the plugin.

      However when we run 'lsof' we see the increase of open file handles over time (see the screenshot attached: the big drop on Nov 27 is due to the manual Jenkins restart).

      Here's the summary of 2 lsof runs with 1 day between them. The list of top files:

      Nov-17:

      100632 slave.log
      32294 log
      7685 timestamps
      4193 random
      3635 urandom

      Nov-18:

      708532 log
      297707 timestamps
      98280 slave.log
      90675 Common.groovy
      85995 BobHelper.groovy
      

      Common.groovy and BobHelper.groovy are used by some jobs.

        Attachments

          Issue Links

            Activity

            Hide
            wheleph Volodymyr Sobotovych added a comment -
            Show
            wheleph Volodymyr Sobotovych added a comment - Jesse Glick , Oleg Nenashev this looks related to https://issues.jenkins-ci.org/browse/JENKINS-45057 . Can you have a look?
            Hide
            dshvedchenko Denis Shvedchenko added a comment -

            Using it in systemgroovystep to collect info about builds, 

            va 7 pentaho 1042r REG 259,0 2296 1085803614 /var/jenkins_home/jobs/EIS/jobs/pgsf_stage/jobs/test_b4596/jobs/TEST02/builds/648/log
            COMMAND  PID  TID    USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME
            java 7 8557 pentaho 14w REG 259,0 139604708 3227706987 /var/log/jenkins/jenkins.log
            > java 7 8557 pentaho 104r REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
            > java 7 8557 pentaho 669r REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
            > java 7 8557 pentaho 779r REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
            > java 7 8557 pentaho 888r REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
            > java 7 8558 pentaho mem REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
            > java 7 8558 pentaho mem REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
            > java 7 8558 pentaho mem REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
            > java 7 8558 pentaho mem REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
            > java 7 8558 pentaho 14w REG 259,0 139604708 3227706987 /var/log/jenkins/jenkins.log
            > java 7 8558 pentaho 104r REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
            > java 7 8558 pentaho 669r REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
            > java 7 8558 pentaho 779r REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
            > java 7 8558 pentaho 888r REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
            > java 7 8559 pentaho mem REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
            > java 7 8559 pentaho mem REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
            > java 7 8559 pentaho mem REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
            > java 7 8559 pentaho mem REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
            > java 7 8559 pentaho 14w REG 259,0 139604708 3227706987 /var/log/jenkins/jenkins.log
            > java 7 8559 pentaho 104r REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
            > java 7 8559 pentaho 669r REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
            > java 7 8559 pentaho 779r REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
            > java 7 8559 pentaho 888r REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
            > java 7 8560 pentaho mem REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
            > java 7 8560 pentaho mem REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
            > java 7 8560 pentaho mem REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
            > java 7 8560 pentaho mem REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
            > java 7 8560 pentaho 14w REG 259,0 139604708 3227706987 /var/log/jenkins/jenkins.log
            > java 7 8560 pentaho 104r REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
            > java 7 8560 pentaho 669r REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
            > java 7 8560 pentaho 779r REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
            > java 7 8560 pentaho 888r REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
            

             

            this is lsof output. every execution adds another TID left opened log and logging related jars

             

             

            Show
            dshvedchenko Denis Shvedchenko added a comment - Using it in systemgroovystep to collect info about builds,  va 7 pentaho 1042r REG 259,0 2296 1085803614 / var /jenkins_home/jobs/EIS/jobs/pgsf_stage/jobs/test_b4596/jobs/TEST02/builds/648/log COMMAND  PID  TID    USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME java 7 8557 pentaho 14w REG 259,0 139604708 3227706987 / var /log/jenkins/jenkins.log > java 7 8557 pentaho 104r REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8557 pentaho 669r REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8557 pentaho 779r REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8557 pentaho 888r REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8558 pentaho mem REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8558 pentaho mem REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8558 pentaho mem REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8558 pentaho mem REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8558 pentaho 14w REG 259,0 139604708 3227706987 / var /log/jenkins/jenkins.log > java 7 8558 pentaho 104r REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8558 pentaho 669r REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8558 pentaho 779r REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8558 pentaho 888r REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8559 pentaho mem REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8559 pentaho mem REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8559 pentaho mem REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8559 pentaho mem REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8559 pentaho 14w REG 259,0 139604708 3227706987 / var /log/jenkins/jenkins.log > java 7 8559 pentaho 104r REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8559 pentaho 669r REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8559 pentaho 779r REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8559 pentaho 888r REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8560 pentaho mem REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8560 pentaho mem REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8560 pentaho mem REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8560 pentaho mem REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8560 pentaho 14w REG 259,0 139604708 3227706987 / var /log/jenkins/jenkins.log > java 7 8560 pentaho 104r REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8560 pentaho 669r REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8560 pentaho 779r REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8560 pentaho 888r REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar   this is lsof output. every execution adds another TID left opened log and logging related jars    
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I am not sure whether it is an issue in the Jenkins core or on Groovy plugin

            Show
            oleg_nenashev Oleg Nenashev added a comment - I am not sure whether it is an issue in the Jenkins core or on Groovy plugin
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Is there a File Leak detector output with allocation traces?

            Show
            oleg_nenashev Oleg Nenashev added a comment - Is there a File Leak detector output with allocation traces?
            Hide
            dshvedchenko Denis Shvedchenko added a comment -

            Oleg Nenashev

            no, File Leak detector has no references to that files

            Show
            dshvedchenko Denis Shvedchenko added a comment - Oleg Nenashev no, File Leak detector has no references to that files

              People

              • Assignee:
                Unassigned
                Reporter:
                wheleph Volodymyr Sobotovych
              • Votes:
                2 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: