Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22088

Disk space leak with multiple copies of winstone-XXXX.jar in TEMP folder

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
    • Environment:
      Windows 8.1 x64, JDK 1.7.0_51, Jenkins 1.553
    • Similar Issues:

      Description

      Continued operation of Jenkins creates multiple copies of "winstone.jar" with a numeric suffix in TEMP folder (winstone12343543634634.jar). We had about 800Gb of these accumulated, slowing down the host system.

      We are currently running Jenkins directly (`java -jar jenkins.jar`) via Windows scheduled task.

        Attachments

          Activity

          t0yv0 Anton Tayanovskyy created issue -
          Hide
          mbshick Matt Hicks added a comment -

          This is a serious problem. I just discovered that my Jenkins installation, which I don't even use, has been filling my drive with these files for at least 2 months, taking up several hundred GB of space. This is on Mac OS X 10.9.

          Show
          mbshick Matt Hicks added a comment - This is a serious problem. I just discovered that my Jenkins installation, which I don't even use, has been filling my drive with these files for at least 2 months, taking up several hundred GB of space. This is on Mac OS X 10.9.
          Hide
          seand Sean Dwyer added a comment -

          I had this issue also. In my case it was caused by the Jenkins service starting and failing constantly and creating a winstone*.jar file in the temp folder each time. This was happening because Jenkins was already running as a process when the service was trying to start. I fixed it by killing the java process for Jenkins in Task Manager, then starting the Jenkins Service.

          Show
          seand Sean Dwyer added a comment - I had this issue also. In my case it was caused by the Jenkins service starting and failing constantly and creating a winstone*.jar file in the temp folder each time. This was happening because Jenkins was already running as a process when the service was trying to start. I fixed it by killing the java process for Jenkins in Task Manager, then starting the Jenkins Service.
          Hide
          amandaxu Amanda Xu added a comment -

          Hello There. I am running into the same problem. Is there a workaround? My Jenkins version is 1.542. Thanks in advanced.

          Show
          amandaxu Amanda Xu added a comment - Hello There. I am running into the same problem. Is there a workaround? My Jenkins version is 1.542. Thanks in advanced.
          Hide
          plnech PL Nech added a comment -

          Hi there,

          I have the same problem with Jenkins ver. 1.612 on Windows 7 64bits.

          Show
          plnech PL Nech added a comment - Hi there, I have the same problem with Jenkins ver. 1.612 on Windows 7 64bits.
          Hide
          gsxjay Jay Beeckman added a comment -

          Same here on 1.627 on Windows 2008 R2 64 Bit. I'm going to manually clear it out, but this should be fixed.

          Show
          gsxjay Jay Beeckman added a comment - Same here on 1.627 on Windows 2008 R2 64 Bit. I'm going to manually clear it out, but this should be fixed.
          Hide
          aarondmarasco_vsi Aaron D. Marasco added a comment -

          This seems to happen on non-Windows as well; I have a Linux server and have a bunch in /tmp.

          Show
          aarondmarasco_vsi Aaron D. Marasco added a comment - This seems to happen on non-Windows as well; I have a Linux server and have a bunch in /tmp .
          Hide
          deepakkoirala Deepak Koirala added a comment - - edited

          Is there a workaround yet? Do we need to manually delete all the winstoneXXXXXXXXXXXX.jar? Deleting these jars won't affect anything, right? We have Jenkins ver. 1.613, installed in windows server and facing the same issue. Almost 70 GB space is full within 3 months. And I saw most of the space is taken by winstone.jar in temp folder.

          Show
          deepakkoirala Deepak Koirala added a comment - - edited Is there a workaround yet? Do we need to manually delete all the winstoneXXXXXXXXXXXX.jar? Deleting these jars won't affect anything, right? We have Jenkins ver. 1.613, installed in windows server and facing the same issue. Almost 70 GB space is full within 3 months. And I saw most of the space is taken by winstone.jar in temp folder.
          Hide
          bardelotnzl Noël Bardelot added a comment -

          Same here on Windows 2008 Server.

          Show
          bardelotnzl Noël Bardelot added a comment - Same here on Windows 2008 Server.
          Hide
          dan Daniel added a comment -

          I just stumbled upon this Issue and used https://wiki.jenkins-ci.org/display/JENKINS/Tmp+Cleaner+Plugin as a workaround.
          Had to set all mentioned parameters with system properties when starting Jenkins but result was a cleaned /var/tmp after that.
          YMMV, but might be a valid workaround. HTH, Daniel

          Show
          dan Daniel added a comment - I just stumbled upon this Issue and used https://wiki.jenkins-ci.org/display/JENKINS/Tmp+Cleaner+Plugin as a workaround. Had to set all mentioned parameters with system properties when starting Jenkins but result was a cleaned /var/tmp after that. YMMV, but might be a valid workaround. HTH, Daniel
          rtyler R. Tyler Croy made changes -
          Field Original Value New Value
          Workflow JNJira [ 154167 ] JNJira + In-Review [ 178712 ]
          Hide
          johey Johan Smolinski added a comment - - edited

          I was also affected by this on Windows Server 2008, caused by an old jenkins process lying around while the service started another. A new winstone-XXXX.jar of 2 MB was created once every two seconds. I killed all java processes, deleted all files and restarted the service. Now there's only one of these jar files.

          I don't know the inners of jenkins, but there should be some sanity check before creating a new winstone-XXXX.jar.

          Show
          johey Johan Smolinski added a comment - - edited I was also affected by this on Windows Server 2008, caused by an old jenkins process lying around while the service started another. A new winstone-XXXX.jar of 2 MB was created once every two seconds. I killed all java processes, deleted all files and restarted the service. Now there's only one of these jar files. I don't know the inners of jenkins, but there should be some sanity check before creating a new winstone-XXXX.jar.
          Hide
          erwin_ntt Erwin Müller added a comment -

          Hello. Just run in this issue on our build server.
          Jenkins ver. 2.32.1.
          Windows Server 2008 R2 Standard 64 Bit.
          winstone-XXXX.jar was created and filled up disk space on C:. Solution was to kill all Jenkins/Java processes and restart Jenkins.

          Show
          erwin_ntt Erwin Müller added a comment - Hello. Just run in this issue on our build server. Jenkins ver. 2.32.1. Windows Server 2008 R2 Standard 64 Bit. winstone-XXXX.jar was created and filled up disk space on C:. Solution was to kill all Jenkins/Java processes and restart Jenkins.
          Hide
          juristr Juri Strumpflohner added a comment - - edited

          same here. We just upgraded to 2.32.3 a couple of days ago and now I have this issue. Didn't have it before, at least I did not notice anything.

          The question is, can I delete them safely?

          Show
          juristr Juri Strumpflohner added a comment - - edited same here. We just upgraded to 2.32.3 a couple of days ago and now I have this issue. Didn't have it before, at least I did not notice anything. The question is, can I delete them safely?
          Hide
          johey Johan Smolinski added a comment -

          Juri Strumpflohner, you probably have an old process running from the previous version. Close all jenkins processes including any old ones, remove all of the temp files and start it up. That's how I solved it and I'm pretty sure it's safe.

          Show
          johey Johan Smolinski added a comment - Juri Strumpflohner , you probably have an old process running from the previous version. Close all jenkins processes including any old ones, remove all of the temp files and start it up. That's how I solved it and I'm pretty sure it's safe.
          Hide
          acerqu Joseph Qu added a comment -

          I can reproduce this error. Set jenkins as a windows service. then stop it. Install another application which is using 8080 port. restart the jenkins service. it will keep generating winstone.jar

          Show
          acerqu Joseph Qu added a comment - I can reproduce this error. Set jenkins as a windows service. then stop it. Install another application which is using 8080 port. restart the jenkins service. it will keep generating winstone.jar
          Hide
          plitog Peter Goins added a comment -

          We've seen this same issue running version 2.59.  In our case it is creating a new file each time we update Jenkins. Only the most recent file was locked by the running service - all other copies could be deleted.

          Show
          plitog Peter Goins added a comment - We've seen this same issue running version 2.59.  In our case it is creating a new file each time we update Jenkins. Only the most recent file was locked by the running service - all other copies could be deleted.
          Hide
          djtjwillia Taylor Williams added a comment -

          Our Jenkins instance on version 2.46.3 is also having this issue. It has been rapidly creating these files. For our instance, it seems to happen with more than just  restarting the service, as since April, 2017 we have gotten 35,000 copies of this file in our temp folder. 

          Show
          djtjwillia Taylor Williams added a comment - Our Jenkins instance on version 2.46.3 is also having this issue. It has been rapidly creating these files. For our instance, it seems to happen with more than just  restarting the service, as since April, 2017 we have gotten 35,000 copies of this file in our temp folder. 
          Hide
          mrichar2 Mark R added a comment -

          We are experiencing this issue as well.

          Show
          mrichar2 Mark R added a comment - We are experiencing this issue as well.
          ifernandezcalvo Ivan Fernandez Calvo made changes -
          Assignee Ivan Fernandez Calvo [ ifernandezcalvo ]
          ifernandezcalvo Ivan Fernandez Calvo made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          ifernandezcalvo Ivan Fernandez Calvo made changes -
          Status In Progress [ 3 ] In Review [ 10005 ]
          Hide
          ifernandezcalvo Ivan Fernandez Calvo added a comment -

          the jna-* folders are also not deleted

          Show
          ifernandezcalvo Ivan Fernandez Calvo added a comment - the jna-* folders are also not deleted
          Hide
          jglick Jesse Glick added a comment -

          Please remember to add a link to your PR when you put one up for review. Otherwise it would be very hard to find the PR starting from JIRA.

          Show
          jglick Jesse Glick added a comment - Please remember to add a link to your PR when you put one up for review. Otherwise it would be very hard to find the PR starting from JIRA.
          jglick Jesse Glick made changes -
          Remote Link This issue links to "extras-executable-war PR 12 (Web Link)" [ 17818 ]
          jglick Jesse Glick made changes -
          Component/s core [ 15593 ]
          Component/s build-environment-plugin [ 17667 ]
          jglick Jesse Glick made changes -
          Labels jenkins winstone winstone
          Hide
          jglick Jesse Glick added a comment -

          Since extras-executable-war already supports the --extractedFilesFolder option perhaps we could just clear the contents of this folder on startup if it was explicitly specified? Then using this option would be the solution, and we would just want to modify the native packages to specify it.

          Show
          jglick Jesse Glick added a comment - Since extras-executable-war already supports the --extractedFilesFolder option perhaps we could just clear the contents of this folder on startup if it was explicitly specified? Then using this option would be the solution, and we would just want to modify the native packages to specify it.
          Hide
          ifernandezcalvo Ivan Fernandez Calvo added a comment -

          I added the check around the cleanup, now only do a cleanup if you set the --extractedFilesFolder=FOLDER property.

          Show
          ifernandezcalvo Ivan Fernandez Calvo added a comment - I added the check around the cleanup, now only do a cleanup if you set the --extractedFilesFolder=FOLDER property.
          Hide
          jngrb Jan Grieb added a comment -

          I experience this issue on a Win7 system which runs the Jenkins in the user space (not as service). 4 slaves are connected. Roughly every 5 minutes, a new winstone* file is created in the TEMP folder and never deleted.

          Show
          jngrb Jan Grieb added a comment - I experience this issue on a Win7 system which runs the Jenkins in the user space (not as service). 4 slaves are connected. Roughly every 5 minutes, a new winstone* file is created in the TEMP folder and never deleted.
          jamesdumay James Dumay made changes -
          Remote Link This issue links to "CloudBees Internal OSS-2489 (Web Link)" [ 18276 ]
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Ivan Fernandez Calvo
          Path:
          src/main/java/Main.java
          http://jenkins-ci.org/commit/extras-executable-war/fe7acc205a251aa84dfb2bc50bccad1bc225426b
          Log:
          JENKINS-22088 Disk space leak with multiple copies of winstone-XXXX.jar in TEMP folder

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Ivan Fernandez Calvo Path: src/main/java/Main.java http://jenkins-ci.org/commit/extras-executable-war/fe7acc205a251aa84dfb2bc50bccad1bc225426b Log: JENKINS-22088 Disk space leak with multiple copies of winstone-XXXX.jar in TEMP folder
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Ivan Fernandez Calvo
          Path:
          http://jenkins-ci.org/commit/extras-executable-war/ebb0c7fa2998c5cb450eeabc40621424beb1ebf7
          Log:
          Merge branch 'JENKINS-22088' of github.com:kuisathaverat/extras-executable-war into JENKINS-22088

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Ivan Fernandez Calvo Path: http://jenkins-ci.org/commit/extras-executable-war/ebb0c7fa2998c5cb450eeabc40621424beb1ebf7 Log: Merge branch ' JENKINS-22088 ' of github.com:kuisathaverat/extras-executable-war into JENKINS-22088
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          src/main/java/Main.java
          http://jenkins-ci.org/commit/extras-executable-war/3aa664c3880534e98aac9dd496cf67afeeef1f16
          Log:
          Merge pull request #12 from kuisathaverat/JENKINS-22088

          JENKINS-22088 Disk space leak

          Compare: https://github.com/jenkinsci/extras-executable-war/compare/a8f310ff07f6...3aa664c38805

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/Main.java http://jenkins-ci.org/commit/extras-executable-war/3aa664c3880534e98aac9dd496cf67afeeef1f16 Log: Merge pull request #12 from kuisathaverat/ JENKINS-22088 JENKINS-22088 Disk space leak Compare: https://github.com/jenkinsci/extras-executable-war/compare/a8f310ff07f6...3aa664c38805
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          war/pom.xml
          http://jenkins-ci.org/commit/jenkins/54c5872d1b27d244b250d518d6846351be13f503
          Log:
          JENKINS-22088 - Upgrade Extras Executable War from 1.36 to 1.37

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: war/pom.xml http://jenkins-ci.org/commit/jenkins/54c5872d1b27d244b250d518d6846351be13f503 Log: JENKINS-22088 - Upgrade Extras Executable War from 1.36 to 1.37
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          war/pom.xml
          http://jenkins-ci.org/commit/jenkins/bff2d2a399018655e3b54e1883f5efff176f3f8c
          Log:
          Merge pull request #3223 from oleg-nenashev/extras-executable-war/1.37

          JENKINS-22088 - Upgrade Extras Executable War from 1.36 to 1.37

          Compare: https://github.com/jenkinsci/jenkins/compare/2a6fc653ee7b...bff2d2a39901

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: war/pom.xml http://jenkins-ci.org/commit/jenkins/bff2d2a399018655e3b54e1883f5efff176f3f8c Log: Merge pull request #3223 from oleg-nenashev/extras-executable-war/1.37 JENKINS-22088 - Upgrade Extras Executable War from 1.36 to 1.37 Compare: https://github.com/jenkinsci/jenkins/compare/2a6fc653ee7b...bff2d2a39901
          ifernandezcalvo Ivan Fernandez Calvo made changes -
          Status In Review [ 10005 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          ptha Peter Hannon added a comment - - edited

          I am still seeing this issue with Jenkins v 2.121.3. From jenkins release notes: https://jenkins.io/changelog-stable/#v2.107.1 I thought this had been fixed in v2.107.1

          I noticed recently that there were several winstone jars in my /tmp. I manually removed all but the most recent.  I then did a safeRestart of Jenkins and there were then 2 winstone jars in the /tmp directory of the master Jenkins node.

          ls -lrt winstone*.jar
          -rw-r--r-- 1 jenkins users 2146767 2019-03-04 12:23 winstone9180695101677339973.jar
          -rw-r--r-- 1 jenkins users 2146767 2019-04-04 11:21 winstone8639977864837551161.jar
          

          We are running the Jenkins war directly via a shell script

          #!/bin/bash
          nohup /home/jenkins/jdk8/bin/java -jar /home/jenkins/jenkins.war --httpPort=8081 > /home/jenkins/jenkins.log 2>&1 &
          

           

          I'm not sure if it's related but I am also seeing some empty jetty directories with similar timestamps:

          ls -ldrt jetty*
          drwxr-xr-x 2 jenkins users 4096 2019-03-04 12:23 jetty-0.0.0.0-8081-war-_-any-2428298004054280574.dir
          drwxr-xr-x 2 jenkins users 4096 2019-04-04 11:21 jetty-0.0.0.0-8081-war-_-any-7220745071344149958.dir
          

           

           

          Show
          ptha Peter Hannon added a comment - - edited I am still seeing this issue with Jenkins v 2.121.3. From jenkins release notes:  https://jenkins.io/changelog-stable/#v2.107.1  I thought this had been fixed in v2.107.1 I noticed recently that there were several winstone jars in my /tmp. I manually removed all but the most recent.  I then did a safeRestart of Jenkins and there were then 2 winstone jars in the /tmp directory of the master Jenkins node. ls -lrt winstone*.jar -rw-r--r-- 1 jenkins users 2146767 2019-03-04 12:23 winstone9180695101677339973.jar -rw-r--r-- 1 jenkins users 2146767 2019-04-04 11:21 winstone8639977864837551161.jar We are running the Jenkins war directly via a shell script #!/bin/bash nohup /home/jenkins/jdk8/bin/java -jar /home/jenkins/jenkins.war --httpPort=8081 > /home/jenkins/jenkins.log 2>&1 &   I'm not sure if it's related but I am also seeing some empty jetty directories with similar timestamps: ls -ldrt jetty* drwxr-xr-x 2 jenkins users 4096 2019-03-04 12:23 jetty-0.0.0.0-8081-war-_-any-2428298004054280574.dir drwxr-xr-x 2 jenkins users 4096 2019-04-04 11:21 jetty-0.0.0.0-8081-war-_-any-7220745071344149958.dir    
          Hide
          ptha Peter Hannon added a comment -

          Ivan Fernandez Calvo thanks for the reply, if you see my comment (the shell script), I'm running this in linux, so the windows service change is not applicable for me.

          Are you saying for a linux environment, this generated winstone jar file needs to be cleaned manually?

          Show
          ptha Peter Hannon added a comment - Ivan Fernandez Calvo thanks for the reply, if you see my comment (the shell script), I'm running this in linux, so the windows service change is not applicable for me. Are you saying for a linux environment, this generated winstone jar file needs to be cleaned manually?
          ifernandezcalvo Ivan Fernandez Calvo made changes -
          Comment [ ~if you only upgrade the Jenkins instance, the issue would be not resolved. The change is in the windows service that launches the Agent, so to resolve the issue you have to uninstall and install again the windows service if not, you would not change anything~ ]
          Hide
          ifernandezcalvo Ivan Fernandez Calvo added a comment -

          Peter Hannon sorry, I've replied something unrelated.

          You have to pass an argument to enable the cleanup `--extractedFilesFolder=FOLDER_PATH`

          https://github.com/kuisathaverat/extras-executable-war/blob/8f00f05c22cb4953ed0a034e0d7ef35c0076f99f/src/main/java/Main.java#L229-L232

          Show
          ifernandezcalvo Ivan Fernandez Calvo added a comment - Peter Hannon sorry, I've replied something unrelated. You have to pass an argument to enable the cleanup `--extractedFilesFolder=FOLDER_PATH` https://github.com/kuisathaverat/extras-executable-war/blob/8f00f05c22cb4953ed0a034e0d7ef35c0076f99f/src/main/java/Main.java#L229-L232
          Hide
          ptha Peter Hannon added a comment -

          Ivan Fernandez Calvo thanks again!

          As per your suggestion, I updated my start shell script to:

           

          #!/bin/bash
          nohup /home/jenkins/jdk8/bin/java -jar /home/jenkins/jenkins.war --httpPort=8081 --extractedFilesFolder=/tmp > /home/jenkins/jenkins.log 2>&1 &
          

          Now the extra winstone jars are cleaned up on startup, I also tested a safeRestart and again there was only 1 winstone jar file (for the running Jenkins).

          So it's solved for me, cheers!

           

          Show
          ptha Peter Hannon added a comment - Ivan Fernandez Calvo  thanks again! As per your suggestion, I updated my start shell script to:   #!/bin/bash nohup /home/jenkins/jdk8/bin/java -jar /home/jenkins/jenkins.war --httpPort=8081 --extractedFilesFolder=/tmp > /home/jenkins/jenkins.log 2>&1 & Now the extra winstone jars are cleaned up on startup, I also tested a safeRestart and again there was only 1 winstone jar file (for the running Jenkins). So it's solved for me, cheers!  
          Hide
          shashankkulsh Shashank Kulshrestha added a comment -

          This issue is happening with my current Jenkins build 2.156 on windows. Is there any solution for windows for this. 

          Disk is getting full, due to so many winstone*.jar in C:\...\AppData\Local\Temp folder

          Show
          shashankkulsh Shashank Kulshrestha added a comment - This issue is happening with my current Jenkins build 2.156 on windows. Is there any solution for windows for this.  Disk is getting full, due to so many winstone*.jar in C:\...\AppData\Local\Temp folder
          Hide
          ifernandezcalvo Ivan Fernandez Calvo added a comment -

          Shashank Kulshrestha Did you read the comment before yours? there you have the solution *--extractedFilesFolder=c:\TMP-FOLDER-PATH*

          Show
          ifernandezcalvo Ivan Fernandez Calvo added a comment - Shashank Kulshrestha Did you read the comment before yours? there you have the solution * --extractedFilesFolder=c:\TMP-FOLDER-PATH *
          Hide
          barna10 Matt Barna added a comment -

          Is there any plan to fix this? Seems wonky that Jenkins knows where to write the jar, but not where to go to delete it.

          Show
          barna10 Matt Barna added a comment - Is there any plan to fix this? Seems wonky that Jenkins knows where to write the jar, but not where to go to delete it.

            People

            • Assignee:
              ifernandezcalvo Ivan Fernandez Calvo
              Reporter:
              t0yv0 Anton Tayanovskyy
            • Votes:
              20 Vote for this issue
              Watchers:
              33 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: