Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-11366

Jenkins takes up too much space in /var/run

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Environment:
      Ubuntu 11.10 Linode vps

      Description

      As on newer debian systems /run or /usr/run is now tmpvfs based and allocated to 10% of memory the jenkins startup fails on low-memory systems (512MB ram).

      [Winstone 2011/10/16 13:38:21] - Beginning extraction from war file
      [Winstone 2011/10/16 13:38:23] - Error initializing web application: prefix []
      java.io.IOException: No space left on device
      at java.io.FileOutputStream.writeBytes(Native Method)
      at java.io.FileOutputStream.write(FileOutputStream.java:282)
      at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273)
      at winstone.HostConfiguration.<init>(HostConfiguration.java:73)
      at winstone.HostGroup.initHost(HostGroup.java:85)
      at winstone.HostGroup.<init>(HostGroup.java:45)
      at winstone.Launcher.<init>(Launcher.java:197)
      at winstone.Launcher.main(Launcher.java:398)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at Main._main(Main.java:268)
      at Main.main(Main.java:96)

      The mount command returns this:
      none on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)

      The df command returns this:
      none 50M 50M 0 100% /run

      I'm not sure about the best solution but jenkins seems by far the biggest user of /run, all other services together use about 10MB. Maybe jenkins could use a different folder to store it's extracted war?

      As a temporary fix I change /etc/default/jenkins this way:
      #JENKINS_ARGS="--webroot=/var/run/jenkins/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
      JENKINS_ARGS="--webroot=$JENKINS_HOME/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"

        Issue Links

          Activity

          francisdb francisdb created issue -
          francisdb francisdb made changes -
          Field Original Value New Value
          Description As on newer debian systems /opt is now tmpvfs based and allocated to 10% of memory the jenkins startup fails on low-memory systems (512MB ram).

          [Winstone 2011/10/16 13:38:21] - Beginning extraction from war file
          [Winstone 2011/10/16 13:38:23] - Error initializing web application: prefix []
          java.io.IOException: No space left on device
          at java.io.FileOutputStream.writeBytes(Native Method)
          at java.io.FileOutputStream.write(FileOutputStream.java:282)
          at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273)
          at winstone.HostConfiguration.<init>(HostConfiguration.java:73)
          at winstone.HostGroup.initHost(HostGroup.java:85)
          at winstone.HostGroup.<init>(HostGroup.java:45)
          at winstone.Launcher.<init>(Launcher.java:197)
          at winstone.Launcher.main(Launcher.java:398)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at Main._main(Main.java:268)
          at Main.main(Main.java:96)


          The mount command returns this:
          none on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)

          The df command returns this:
          none 50M 50M 0 100% /run

          I'm not sure about the best solution but jenkins seems by far the biggest user of /opt, all other services together use about 10MB. Maybe jenkins could use a different folder to store it's extracted war?

          A temporary fix for this would revert to the previous ubuntu behavior of keeping /opt on disk but I have not found how to do this yet.
          As on newer debian systems /opt is now tmpvfs based and allocated to 10% of memory the jenkins startup fails on low-memory systems (512MB ram).

          [Winstone 2011/10/16 13:38:21] - Beginning extraction from war file
          [Winstone 2011/10/16 13:38:23] - Error initializing web application: prefix []
          java.io.IOException: No space left on device
          at java.io.FileOutputStream.writeBytes(Native Method)
          at java.io.FileOutputStream.write(FileOutputStream.java:282)
          at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273)
          at winstone.HostConfiguration.<init>(HostConfiguration.java:73)
          at winstone.HostGroup.initHost(HostGroup.java:85)
          at winstone.HostGroup.<init>(HostGroup.java:45)
          at winstone.Launcher.<init>(Launcher.java:197)
          at winstone.Launcher.main(Launcher.java:398)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at Main._main(Main.java:268)
          at Main.main(Main.java:96)


          The mount command returns this:
          none on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)

          The df command returns this:
          none 50M 50M 0 100% /run

          I'm not sure about the best solution but jenkins seems by far the biggest user of /opt, all other services together use about 10MB. Maybe jenkins could use a different folder to store it's extracted war?

          As a temporary fix I change /etc/default/jenkins this way:
          #JENKINS_ARGS="--webroot=/var/run/jenkins/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
          JENKINS_ARGS="--webroot=$JENKINS_HOME/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
          Hide
          oldelvet Richard Mortimer added a comment -

          Your description of the problem is very confusing. You mention /opt as being mounted using memory backed storage but I rather suspect you mean /run (which historically includes /var/run too).

          That said I can understand the issue that you raise and can see how it could be a problem on low-memory systems. I don't know what the appropriate location would be regarding Debian policy but I'm guessing it would likely be /var/cache/jenkins or something like that.

          Basically jenkins currently unpacks the war into /var/run/jenkins and fills it up.

          Show
          oldelvet Richard Mortimer added a comment - Your description of the problem is very confusing. You mention /opt as being mounted using memory backed storage but I rather suspect you mean /run (which historically includes /var/run too). That said I can understand the issue that you raise and can see how it could be a problem on low-memory systems. I don't know what the appropriate location would be regarding Debian policy but I'm guessing it would likely be /var/cache/jenkins or something like that. Basically jenkins currently unpacks the war into /var/run/jenkins and fills it up.
          francisdb francisdb made changes -
          Description As on newer debian systems /opt is now tmpvfs based and allocated to 10% of memory the jenkins startup fails on low-memory systems (512MB ram).

          [Winstone 2011/10/16 13:38:21] - Beginning extraction from war file
          [Winstone 2011/10/16 13:38:23] - Error initializing web application: prefix []
          java.io.IOException: No space left on device
          at java.io.FileOutputStream.writeBytes(Native Method)
          at java.io.FileOutputStream.write(FileOutputStream.java:282)
          at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273)
          at winstone.HostConfiguration.<init>(HostConfiguration.java:73)
          at winstone.HostGroup.initHost(HostGroup.java:85)
          at winstone.HostGroup.<init>(HostGroup.java:45)
          at winstone.Launcher.<init>(Launcher.java:197)
          at winstone.Launcher.main(Launcher.java:398)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at Main._main(Main.java:268)
          at Main.main(Main.java:96)


          The mount command returns this:
          none on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)

          The df command returns this:
          none 50M 50M 0 100% /run

          I'm not sure about the best solution but jenkins seems by far the biggest user of /opt, all other services together use about 10MB. Maybe jenkins could use a different folder to store it's extracted war?

          As a temporary fix I change /etc/default/jenkins this way:
          #JENKINS_ARGS="--webroot=/var/run/jenkins/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
          JENKINS_ARGS="--webroot=$JENKINS_HOME/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
          As on newer debian systems /run or /usr/run is now tmpvfs based and allocated to 10% of memory the jenkins startup fails on low-memory systems (512MB ram).

          [Winstone 2011/10/16 13:38:21] - Beginning extraction from war file
          [Winstone 2011/10/16 13:38:23] - Error initializing web application: prefix []
          java.io.IOException: No space left on device
          at java.io.FileOutputStream.writeBytes(Native Method)
          at java.io.FileOutputStream.write(FileOutputStream.java:282)
          at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273)
          at winstone.HostConfiguration.<init>(HostConfiguration.java:73)
          at winstone.HostGroup.initHost(HostGroup.java:85)
          at winstone.HostGroup.<init>(HostGroup.java:45)
          at winstone.Launcher.<init>(Launcher.java:197)
          at winstone.Launcher.main(Launcher.java:398)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at Main._main(Main.java:268)
          at Main.main(Main.java:96)


          The mount command returns this:
          none on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)

          The df command returns this:
          none 50M 50M 0 100% /run

          I'm not sure about the best solution but jenkins seems by far the biggest user of /opt, all other services together use about 10MB. Maybe jenkins could use a different folder to store it's extracted war?

          As a temporary fix I change /etc/default/jenkins this way:
          #JENKINS_ARGS="--webroot=/var/run/jenkins/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
          JENKINS_ARGS="--webroot=$JENKINS_HOME/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
          francisdb francisdb made changes -
          Description As on newer debian systems /run or /usr/run is now tmpvfs based and allocated to 10% of memory the jenkins startup fails on low-memory systems (512MB ram).

          [Winstone 2011/10/16 13:38:21] - Beginning extraction from war file
          [Winstone 2011/10/16 13:38:23] - Error initializing web application: prefix []
          java.io.IOException: No space left on device
          at java.io.FileOutputStream.writeBytes(Native Method)
          at java.io.FileOutputStream.write(FileOutputStream.java:282)
          at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273)
          at winstone.HostConfiguration.<init>(HostConfiguration.java:73)
          at winstone.HostGroup.initHost(HostGroup.java:85)
          at winstone.HostGroup.<init>(HostGroup.java:45)
          at winstone.Launcher.<init>(Launcher.java:197)
          at winstone.Launcher.main(Launcher.java:398)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at Main._main(Main.java:268)
          at Main.main(Main.java:96)


          The mount command returns this:
          none on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)

          The df command returns this:
          none 50M 50M 0 100% /run

          I'm not sure about the best solution but jenkins seems by far the biggest user of /opt, all other services together use about 10MB. Maybe jenkins could use a different folder to store it's extracted war?

          As a temporary fix I change /etc/default/jenkins this way:
          #JENKINS_ARGS="--webroot=/var/run/jenkins/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
          JENKINS_ARGS="--webroot=$JENKINS_HOME/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
          As on newer debian systems /run or /usr/run is now tmpvfs based and allocated to 10% of memory the jenkins startup fails on low-memory systems (512MB ram).

          [Winstone 2011/10/16 13:38:21] - Beginning extraction from war file
          [Winstone 2011/10/16 13:38:23] - Error initializing web application: prefix []
          java.io.IOException: No space left on device
          at java.io.FileOutputStream.writeBytes(Native Method)
          at java.io.FileOutputStream.write(FileOutputStream.java:282)
          at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273)
          at winstone.HostConfiguration.<init>(HostConfiguration.java:73)
          at winstone.HostGroup.initHost(HostGroup.java:85)
          at winstone.HostGroup.<init>(HostGroup.java:45)
          at winstone.Launcher.<init>(Launcher.java:197)
          at winstone.Launcher.main(Launcher.java:398)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at Main._main(Main.java:268)
          at Main.main(Main.java:96)


          The mount command returns this:
          none on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)

          The df command returns this:
          none 50M 50M 0 100% /run

          I'm not sure about the best solution but jenkins seems by far the biggest user of /run, all other services together use about 10MB. Maybe jenkins could use a different folder to store it's extracted war?

          As a temporary fix I change /etc/default/jenkins this way:
          #JENKINS_ARGS="--webroot=/var/run/jenkins/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
          JENKINS_ARGS="--webroot=$JENKINS_HOME/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
          Hide
          francisdb francisdb added a comment -

          My excuses for the confusing problem description, I fixed my errors. The problem is indeed that /var/run/jenkins which is a link to /run/jenkins fills up because on a 512MB memory system /run is only ~50MB

          Show
          francisdb francisdb added a comment - My excuses for the confusing problem description, I fixed my errors. The problem is indeed that /var/run/jenkins which is a link to /run/jenkins fills up because on a 512MB memory system /run is only ~50MB
          francisdb francisdb made changes -
          Summary Low memory debian systems cause IOException: No space left on device Jenkins takes up too much space in /usr/run
          francisdb francisdb made changes -
          Link This issue is duplicated by JENKINS-11364 [ JENKINS-11364 ]
          francisdb francisdb made changes -
          Link This issue is duplicated by JENKINS-11365 [ JENKINS-11365 ]
          Hide
          tisch tischler added a comment -

          I'm getting the same error message on a 512 MB RAM Debian Squeeze system.

          Here's the log:

          Running from: /usr/share/jenkins/jenkins.war
          [Winstone 2011/11/15 12:26:25] - Beginning extraction from war file
          [Winstone 2011/11/15 12:28:46] - Error initializing web application: prefix []
          java.io.IOException: No space left on device
          at java.io.FileOutputStream.writeBytes(Native Method)
          at java.io.FileOutputStream.write(FileOutputStream.java:297)
          at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273)
          at winstone.HostConfiguration.<init>(HostConfiguration.java:73)
          at winstone.HostGroup.initHost(HostGroup.java:85)
          at winstone.HostGroup.<init>(HostGroup.java:45)
          at winstone.Launcher.<init>(Launcher.java:197)
          at winstone.Launcher.main(Launcher.java:398)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:616)
          at Main._main(Main.java:268)
          at Main.main(Main.java:96)

          Show
          tisch tischler added a comment - I'm getting the same error message on a 512 MB RAM Debian Squeeze system. Here's the log: Running from: /usr/share/jenkins/jenkins.war [Winstone 2011/11/15 12:26:25] - Beginning extraction from war file [Winstone 2011/11/15 12:28:46] - Error initializing web application: prefix [] java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:297) at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273) at winstone.HostConfiguration.<init>(HostConfiguration.java:73) at winstone.HostGroup.initHost(HostGroup.java:85) at winstone.HostGroup.<init>(HostGroup.java:45) at winstone.Launcher.<init>(Launcher.java:197) at winstone.Launcher.main(Launcher.java:398) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at Main._main(Main.java:268) at Main.main(Main.java:96)
          Hide
          oldelvet Richard Mortimer added a comment -

          I've been doing a bit more research and /var/cache/jenkins looks like the proper place for unpacking the war file. Tomcat6 on Ubuntu (and Debian) puts its Work directory in /var/cache.

          I should be able to sort out a patch/pull request in the next day or so.

          Show
          oldelvet Richard Mortimer added a comment - I've been doing a bit more research and /var/cache/jenkins looks like the proper place for unpacking the war file. Tomcat6 on Ubuntu (and Debian) puts its Work directory in /var/cache. I should be able to sort out a patch/pull request in the next day or so.
          oldelvet Richard Mortimer made changes -
          Assignee Richard Mortimer [ oldelvet ]
          oldelvet Richard Mortimer made changes -
          Summary Jenkins takes up too much space in /usr/run Jenkins takes up too much space in /var/run
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Richard Mortimer
          Path:
          changelog.html
          debian/debian/dirs
          debian/debian/jenkins.default
          debian/debian/jenkins.postinst
          debian/debian/jenkins.postrm
          http://jenkins-ci.org/commit/jenkins/9714917685e2b50f59b4feda9bc4e561870a944f
          Log:
          [FIXED JENKINS-11366] Jenkins takes up too much space in /var/run

          Use /var/cache/jenkins to hold the unpacked jenkins.war file at runtime.

          Compare: https://github.com/jenkinsci/jenkins/compare/a1714c1...9714917

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Richard Mortimer Path: changelog.html debian/debian/dirs debian/debian/jenkins.default debian/debian/jenkins.postinst debian/debian/jenkins.postrm http://jenkins-ci.org/commit/jenkins/9714917685e2b50f59b4feda9bc4e561870a944f Log: [FIXED JENKINS-11366] Jenkins takes up too much space in /var/run Use /var/cache/jenkins to hold the unpacked jenkins.war file at runtime. Compare: https://github.com/jenkinsci/jenkins/compare/a1714c1...9714917
          scm_issue_link SCM/JIRA link daemon made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          dogfood dogfood added a comment -

          Integrated in jenkins_main_trunk #1331
          [FIXED JENKINS-11366] Jenkins takes up too much space in /var/run

          Kohsuke Kawaguchi : 9714917685e2b50f59b4feda9bc4e561870a944f
          Files :

          • debian/debian/jenkins.postrm
          • debian/debian/jenkins.default
          • debian/debian/dirs
          • changelog.html
          • debian/debian/jenkins.postinst
          Show
          dogfood dogfood added a comment - Integrated in jenkins_main_trunk #1331 [FIXED JENKINS-11366] Jenkins takes up too much space in /var/run Kohsuke Kawaguchi : 9714917685e2b50f59b4feda9bc4e561870a944f Files : debian/debian/jenkins.postrm debian/debian/jenkins.default debian/debian/dirs changelog.html debian/debian/jenkins.postinst
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Richard Mortimer
          Path:
          debian/dirs
          debian/jenkins.default
          debian/jenkins.postinst
          debian/jenkins.postrm
          http://jenkins-ci.org/commit/packaging/b98f3a38238b70d4ab2fd9183a7e8b6b9db39aff
          Log:
          [FIXED JENKINS-11366] Jenkins takes up too much space in /var/run

          Use /var/cache/jenkins to hold the unpacked jenkins.war file at runtime.

          Originally-From: jenkins-ci.org/commit/core/9714917685e2b50f59b4feda9bc4e561870a944f

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Richard Mortimer Path: debian/dirs debian/jenkins.default debian/jenkins.postinst debian/jenkins.postrm http://jenkins-ci.org/commit/packaging/b98f3a38238b70d4ab2fd9183a7e8b6b9db39aff Log: [FIXED JENKINS-11366] Jenkins takes up too much space in /var/run Use /var/cache/jenkins to hold the unpacked jenkins.war file at runtime. Originally-From: jenkins-ci.org/commit/core/9714917685e2b50f59b4feda9bc4e561870a944f

            People

            • Assignee:
              oldelvet Richard Mortimer
              Reporter:
              francisdb francisdb
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: