Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49386

Windows Jenkins process locks the Log folder. Can't delete builds

    Details

    • Similar Issues:

      Description

      Running Jenkins as a scheduled task on Windows 2008 server. Recently upgraded to Jenkins v2.100 and since then I have problems deleting builds. Getting 'build in use error'

      Checking for open file handles on the Windows server confirms that Jenkins is keeping open file handles on the log folders

      This is true for several jobs. Some common plugins in use

      [conditional-buildstep@1.3.5]
      [groovy@1.30]
      [envinject@1.93.1]
      [scriptler@2.9]
      [ssh@2.5]

        Attachments

          Activity

          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Please provide a dump from http://file-leak-detector.kohsuke.org/

          Show
          oleg_nenashev Oleg Nenashev added a comment - Please provide a dump from http://file-leak-detector.kohsuke.org/
          Hide
          ioannis Ioannis Moutsatsos added a comment -

          Unfortunately, I can't get this to run successfully on my Windows server. Trying to bind to the Windows Jenkins PID I get and error

          com.sun.tools.attach.AttachNotSupportedException: Insufficient memory or insufficient privileges to attach

          Also trying to run with the http option I get no errors but the JVM starts and then immediately dies. Anyway, will this give me more information than the Windows Process Explorer? For example if some of my embedded groovy code or scriptlets are leaving the file handles open will I be able to detect it with the file-leak-detector? Thanks for the feedback

          Show
          ioannis Ioannis Moutsatsos added a comment - Unfortunately, I can't get this to run successfully on my Windows server. Trying to bind to the Windows Jenkins PID I get and error com.sun.tools.attach.AttachNotSupportedException: Insufficient memory or insufficient privileges to attach Also trying to run with the http option I get no errors but the JVM starts and then immediately dies. Anyway, will this give me more information than the Windows Process Explorer? For example if some of my embedded groovy code or scriptlets are leaving the file handles open will I be able to detect it with the file-leak-detector? Thanks for the feedback
          Hide
          kevin_randino Kevin Randino added a comment -

          I'd like to confirm that I am having the same issues as reported by Ioannis on Jenkins 2.107.2.  This is causing a major problem for us as our artifacts are not getting cleaned up do to the job being "in use" when the LogRotator runs and we are constantly running out of disk space because of it.  When the LogRotator runs I get the following error in my log:

           

          Failed to rotate log java.io.IOException: M:\jenkins-jobs\Build-CheckoutInstall\builds\2415 is in use at hudson.model.Run.delete(Run.java:1505) at hudson.tasks.LogRotator.perform(LogRotator.java:145) at hudson.model.Job.logRotate(Job.java:472) at hudson.model.Run.execute(Run.java:1791) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429)

           

          This does not appear to happen to all of our projects, but I can't find the common thread between the jobs it is happening on. It also does not appear to hold logs from builds that fail, only builds that succeed.  I attempted to run the file leak detector Oleg provided, but I am getting the same error that Ioannis is.  This is nearly a show stopper for us, as our Jenkins system runs out of disk space constantly while we can't clean artifacts.

          Attached is a screenshot of some of the very many log files being held by Jenkins right now.

          If there's anything I can do to help track this down, or if you can tell me what I'm doing wrong with the file leak detector, please let me know!

          Show
          kevin_randino Kevin Randino added a comment - I'd like to confirm that I am having the same issues as reported by Ioannis on Jenkins 2.107.2.  This is causing a major problem for us as our artifacts are not getting cleaned up do to the job being "in use" when the LogRotator runs and we are constantly running out of disk space because of it.  When the LogRotator runs I get the following error in my log:   Failed to rotate log java.io.IOException: M:\jenkins-jobs\Build-CheckoutInstall\builds\2415 is in use at hudson.model.Run.delete(Run.java:1505) at hudson.tasks.LogRotator.perform(LogRotator.java:145) at hudson.model.Job.logRotate(Job.java:472) at hudson.model.Run.execute(Run.java:1791) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429)   This does not appear to happen to all of our projects, but I can't find the common thread between the jobs it is happening on. It also does not appear to hold logs from builds that fail, only builds that succeed.  I attempted to run the file leak detector Oleg provided, but I am getting the same error that Ioannis is.  This is nearly a show stopper for us, as our Jenkins system runs out of disk space constantly while we can't clean artifacts. Attached is a screenshot of some of the very many log files being held by Jenkins right now. If there's anything I can do to help track this down, or if you can tell me what I'm doing wrong with the file leak detector, please let me know!
          Hide
          kevin_randino Kevin Randino added a comment -

          Good news!  I figured out the issue on my end, and it was all of my own doing.  In the past when we used Jenkins 2.32.3, we never encountered this issue, it only started when we upgrade to v2.107.2.  It turns out we had a job that would open the log file of other jobs using the following groovy code:

          reader = new BufferedReader(build.getLogReader())
          

          The issue was we never did a reader.close() in the code, causing a handler for the log file to never be released.  The thing I find odd is this problem never popped up with Jenkins 2.32.3, so I still do believe there's some sort of change going on here, but knowing that this is not directly built into Jenkins, but rather something we did to ourselves is excellent news.  I've since started closing the BufferedReader, and now it looks like the handles are not being removed.  Sorry for the confusion on this one, but I hope this has helped others!

          Ioannis Moutsatsos, I've got a bit of code for you that you can use to try to find any jobs that may be using a log reader like I was.  If you go to your http://<jenkinsurl>/script web page, you can put this in there and it will scan all your jobs for the word "getLogReader" it may help you track down the same thing if it's happening on your system.  From there you just have to make sure they're getting closed.  Good luck!

          // Find a specific string within all jenkins jobs.
          
          search = "getLogReader()"
          
          jenkins = Jenkins.instance.getInstance()
          
          jenkins.getItems().each()
          {
            if (it.getConfigFile().asString().contains(search))
            {
              println(jenkins.getRootUrl() + "job/" + it.getDisplayName() + "/configure")
            }
          }
          
          Show
          kevin_randino Kevin Randino added a comment - Good news!  I figured out the issue on my end, and it was all of my own doing.  In the past when we used Jenkins 2.32.3, we never encountered this issue, it only started when we upgrade to v2.107.2.  It turns out we had a job that would open the log file of other jobs using the following groovy code: reader = new BufferedReader(build.getLogReader()) The issue was we never did a reader.close() in the code, causing a handler for the log file to never be released.  The thing I find odd is this problem never popped up with Jenkins 2.32.3, so I still do believe there's some sort of change going on here, but knowing that this is not directly built into Jenkins, but rather something we did to ourselves is excellent news.  I've since started closing the BufferedReader, and now it looks like the handles are not being removed.  Sorry for the confusion on this one, but I hope this has helped others! Ioannis Moutsatsos , I've got a bit of code for you that you can use to try to find any jobs that may be using a log reader like I was.  If you go to your http://<jenkinsurl>/script web page, you can put this in there and it will scan all your jobs for the word "getLogReader" it may help you track down the same thing if it's happening on your system.  From there you just have to make sure they're getting closed.  Good luck! // Find a specific string within all jenkins jobs. search = "getLogReader()" jenkins = Jenkins.instance.getInstance() jenkins.getItems().each() {   if (it.getConfigFile().asString().contains(search))   {     println(jenkins.getRootUrl() + "job/" + it.getDisplayName() + "/configure" )   } }

            People

            • Assignee:
              Unassigned
              Reporter:
              ioannis Ioannis Moutsatsos
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: