Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-11366

Jenkins takes up too much space in /var/run

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Environment:
      Ubuntu 11.10 Linode vps
    • Similar Issues:

      Description

      As on newer debian systems /run or /usr/run is now tmpvfs based and allocated to 10% of memory the jenkins startup fails on low-memory systems (512MB ram).

      [Winstone 2011/10/16 13:38:21] - Beginning extraction from war file
      [Winstone 2011/10/16 13:38:23] - Error initializing web application: prefix []
      java.io.IOException: No space left on device
      at java.io.FileOutputStream.writeBytes(Native Method)
      at java.io.FileOutputStream.write(FileOutputStream.java:282)
      at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273)
      at winstone.HostConfiguration.<init>(HostConfiguration.java:73)
      at winstone.HostGroup.initHost(HostGroup.java:85)
      at winstone.HostGroup.<init>(HostGroup.java:45)
      at winstone.Launcher.<init>(Launcher.java:197)
      at winstone.Launcher.main(Launcher.java:398)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at Main._main(Main.java:268)
      at Main.main(Main.java:96)

      The mount command returns this:
      none on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)

      The df command returns this:
      none 50M 50M 0 100% /run

      I'm not sure about the best solution but jenkins seems by far the biggest user of /run, all other services together use about 10MB. Maybe jenkins could use a different folder to store it's extracted war?

      As a temporary fix I change /etc/default/jenkins this way:
      #JENKINS_ARGS="--webroot=/var/run/jenkins/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
      JENKINS_ARGS="--webroot=$JENKINS_HOME/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"

        Attachments

          Issue Links

            Activity

            francisdb francisdb created issue -
            francisdb francisdb made changes -
            Field Original Value New Value
            Description As on newer debian systems /opt is now tmpvfs based and allocated to 10% of memory the jenkins startup fails on low-memory systems (512MB ram).

            [Winstone 2011/10/16 13:38:21] - Beginning extraction from war file
            [Winstone 2011/10/16 13:38:23] - Error initializing web application: prefix []
            java.io.IOException: No space left on device
            at java.io.FileOutputStream.writeBytes(Native Method)
            at java.io.FileOutputStream.write(FileOutputStream.java:282)
            at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273)
            at winstone.HostConfiguration.<init>(HostConfiguration.java:73)
            at winstone.HostGroup.initHost(HostGroup.java:85)
            at winstone.HostGroup.<init>(HostGroup.java:45)
            at winstone.Launcher.<init>(Launcher.java:197)
            at winstone.Launcher.main(Launcher.java:398)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at Main._main(Main.java:268)
            at Main.main(Main.java:96)


            The mount command returns this:
            none on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)

            The df command returns this:
            none 50M 50M 0 100% /run

            I'm not sure about the best solution but jenkins seems by far the biggest user of /opt, all other services together use about 10MB. Maybe jenkins could use a different folder to store it's extracted war?

            A temporary fix for this would revert to the previous ubuntu behavior of keeping /opt on disk but I have not found how to do this yet.
            As on newer debian systems /opt is now tmpvfs based and allocated to 10% of memory the jenkins startup fails on low-memory systems (512MB ram).

            [Winstone 2011/10/16 13:38:21] - Beginning extraction from war file
            [Winstone 2011/10/16 13:38:23] - Error initializing web application: prefix []
            java.io.IOException: No space left on device
            at java.io.FileOutputStream.writeBytes(Native Method)
            at java.io.FileOutputStream.write(FileOutputStream.java:282)
            at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273)
            at winstone.HostConfiguration.<init>(HostConfiguration.java:73)
            at winstone.HostGroup.initHost(HostGroup.java:85)
            at winstone.HostGroup.<init>(HostGroup.java:45)
            at winstone.Launcher.<init>(Launcher.java:197)
            at winstone.Launcher.main(Launcher.java:398)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at Main._main(Main.java:268)
            at Main.main(Main.java:96)


            The mount command returns this:
            none on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)

            The df command returns this:
            none 50M 50M 0 100% /run

            I'm not sure about the best solution but jenkins seems by far the biggest user of /opt, all other services together use about 10MB. Maybe jenkins could use a different folder to store it's extracted war?

            As a temporary fix I change /etc/default/jenkins this way:
            #JENKINS_ARGS="--webroot=/var/run/jenkins/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
            JENKINS_ARGS="--webroot=$JENKINS_HOME/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
            Hide
            oldelvet Richard Mortimer added a comment -

            Your description of the problem is very confusing. You mention /opt as being mounted using memory backed storage but I rather suspect you mean /run (which historically includes /var/run too).

            That said I can understand the issue that you raise and can see how it could be a problem on low-memory systems. I don't know what the appropriate location would be regarding Debian policy but I'm guessing it would likely be /var/cache/jenkins or something like that.

            Basically jenkins currently unpacks the war into /var/run/jenkins and fills it up.

            Show
            oldelvet Richard Mortimer added a comment - Your description of the problem is very confusing. You mention /opt as being mounted using memory backed storage but I rather suspect you mean /run (which historically includes /var/run too). That said I can understand the issue that you raise and can see how it could be a problem on low-memory systems. I don't know what the appropriate location would be regarding Debian policy but I'm guessing it would likely be /var/cache/jenkins or something like that. Basically jenkins currently unpacks the war into /var/run/jenkins and fills it up.
            francisdb francisdb made changes -
            Description As on newer debian systems /opt is now tmpvfs based and allocated to 10% of memory the jenkins startup fails on low-memory systems (512MB ram).

            [Winstone 2011/10/16 13:38:21] - Beginning extraction from war file
            [Winstone 2011/10/16 13:38:23] - Error initializing web application: prefix []
            java.io.IOException: No space left on device
            at java.io.FileOutputStream.writeBytes(Native Method)
            at java.io.FileOutputStream.write(FileOutputStream.java:282)
            at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273)
            at winstone.HostConfiguration.<init>(HostConfiguration.java:73)
            at winstone.HostGroup.initHost(HostGroup.java:85)
            at winstone.HostGroup.<init>(HostGroup.java:45)
            at winstone.Launcher.<init>(Launcher.java:197)
            at winstone.Launcher.main(Launcher.java:398)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at Main._main(Main.java:268)
            at Main.main(Main.java:96)


            The mount command returns this:
            none on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)

            The df command returns this:
            none 50M 50M 0 100% /run

            I'm not sure about the best solution but jenkins seems by far the biggest user of /opt, all other services together use about 10MB. Maybe jenkins could use a different folder to store it's extracted war?

            As a temporary fix I change /etc/default/jenkins this way:
            #JENKINS_ARGS="--webroot=/var/run/jenkins/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
            JENKINS_ARGS="--webroot=$JENKINS_HOME/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
            As on newer debian systems /run or /usr/run is now tmpvfs based and allocated to 10% of memory the jenkins startup fails on low-memory systems (512MB ram).

            [Winstone 2011/10/16 13:38:21] - Beginning extraction from war file
            [Winstone 2011/10/16 13:38:23] - Error initializing web application: prefix []
            java.io.IOException: No space left on device
            at java.io.FileOutputStream.writeBytes(Native Method)
            at java.io.FileOutputStream.write(FileOutputStream.java:282)
            at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273)
            at winstone.HostConfiguration.<init>(HostConfiguration.java:73)
            at winstone.HostGroup.initHost(HostGroup.java:85)
            at winstone.HostGroup.<init>(HostGroup.java:45)
            at winstone.Launcher.<init>(Launcher.java:197)
            at winstone.Launcher.main(Launcher.java:398)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at Main._main(Main.java:268)
            at Main.main(Main.java:96)


            The mount command returns this:
            none on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)

            The df command returns this:
            none 50M 50M 0 100% /run

            I'm not sure about the best solution but jenkins seems by far the biggest user of /opt, all other services together use about 10MB. Maybe jenkins could use a different folder to store it's extracted war?

            As a temporary fix I change /etc/default/jenkins this way:
            #JENKINS_ARGS="--webroot=/var/run/jenkins/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
            JENKINS_ARGS="--webroot=$JENKINS_HOME/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
            francisdb francisdb made changes -
            Description As on newer debian systems /run or /usr/run is now tmpvfs based and allocated to 10% of memory the jenkins startup fails on low-memory systems (512MB ram).

            [Winstone 2011/10/16 13:38:21] - Beginning extraction from war file
            [Winstone 2011/10/16 13:38:23] - Error initializing web application: prefix []
            java.io.IOException: No space left on device
            at java.io.FileOutputStream.writeBytes(Native Method)
            at java.io.FileOutputStream.write(FileOutputStream.java:282)
            at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273)
            at winstone.HostConfiguration.<init>(HostConfiguration.java:73)
            at winstone.HostGroup.initHost(HostGroup.java:85)
            at winstone.HostGroup.<init>(HostGroup.java:45)
            at winstone.Launcher.<init>(Launcher.java:197)
            at winstone.Launcher.main(Launcher.java:398)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at Main._main(Main.java:268)
            at Main.main(Main.java:96)


            The mount command returns this:
            none on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)

            The df command returns this:
            none 50M 50M 0 100% /run

            I'm not sure about the best solution but jenkins seems by far the biggest user of /opt, all other services together use about 10MB. Maybe jenkins could use a different folder to store it's extracted war?

            As a temporary fix I change /etc/default/jenkins this way:
            #JENKINS_ARGS="--webroot=/var/run/jenkins/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
            JENKINS_ARGS="--webroot=$JENKINS_HOME/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
            As on newer debian systems /run or /usr/run is now tmpvfs based and allocated to 10% of memory the jenkins startup fails on low-memory systems (512MB ram).

            [Winstone 2011/10/16 13:38:21] - Beginning extraction from war file
            [Winstone 2011/10/16 13:38:23] - Error initializing web application: prefix []
            java.io.IOException: No space left on device
            at java.io.FileOutputStream.writeBytes(Native Method)
            at java.io.FileOutputStream.write(FileOutputStream.java:282)
            at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273)
            at winstone.HostConfiguration.<init>(HostConfiguration.java:73)
            at winstone.HostGroup.initHost(HostGroup.java:85)
            at winstone.HostGroup.<init>(HostGroup.java:45)
            at winstone.Launcher.<init>(Launcher.java:197)
            at winstone.Launcher.main(Launcher.java:398)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at Main._main(Main.java:268)
            at Main.main(Main.java:96)


            The mount command returns this:
            none on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)

            The df command returns this:
            none 50M 50M 0 100% /run

            I'm not sure about the best solution but jenkins seems by far the biggest user of /run, all other services together use about 10MB. Maybe jenkins could use a different folder to store it's extracted war?

            As a temporary fix I change /etc/default/jenkins this way:
            #JENKINS_ARGS="--webroot=/var/run/jenkins/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
            JENKINS_ARGS="--webroot=$JENKINS_HOME/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
            Hide
            francisdb francisdb added a comment -

            My excuses for the confusing problem description, I fixed my errors. The problem is indeed that /var/run/jenkins which is a link to /run/jenkins fills up because on a 512MB memory system /run is only ~50MB

            Show
            francisdb francisdb added a comment - My excuses for the confusing problem description, I fixed my errors. The problem is indeed that /var/run/jenkins which is a link to /run/jenkins fills up because on a 512MB memory system /run is only ~50MB
            francisdb francisdb made changes -
            Summary Low memory debian systems cause IOException: No space left on device Jenkins takes up too much space in /usr/run
            francisdb francisdb made changes -
            Link This issue is duplicated by JENKINS-11364 [ JENKINS-11364 ]
            francisdb francisdb made changes -
            Link This issue is duplicated by JENKINS-11365 [ JENKINS-11365 ]
            Hide
            tisch tischler added a comment -

            I'm getting the same error message on a 512 MB RAM Debian Squeeze system.

            Here's the log:

            Running from: /usr/share/jenkins/jenkins.war
            [Winstone 2011/11/15 12:26:25] - Beginning extraction from war file
            [Winstone 2011/11/15 12:28:46] - Error initializing web application: prefix []
            java.io.IOException: No space left on device
            at java.io.FileOutputStream.writeBytes(Native Method)
            at java.io.FileOutputStream.write(FileOutputStream.java:297)
            at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273)
            at winstone.HostConfiguration.<init>(HostConfiguration.java:73)
            at winstone.HostGroup.initHost(HostGroup.java:85)
            at winstone.HostGroup.<init>(HostGroup.java:45)
            at winstone.Launcher.<init>(Launcher.java:197)
            at winstone.Launcher.main(Launcher.java:398)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:616)
            at Main._main(Main.java:268)
            at Main.main(Main.java:96)

            Show
            tisch tischler added a comment - I'm getting the same error message on a 512 MB RAM Debian Squeeze system. Here's the log: Running from: /usr/share/jenkins/jenkins.war [Winstone 2011/11/15 12:26:25] - Beginning extraction from war file [Winstone 2011/11/15 12:28:46] - Error initializing web application: prefix [] java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:297) at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:273) at winstone.HostConfiguration.<init>(HostConfiguration.java:73) at winstone.HostGroup.initHost(HostGroup.java:85) at winstone.HostGroup.<init>(HostGroup.java:45) at winstone.Launcher.<init>(Launcher.java:197) at winstone.Launcher.main(Launcher.java:398) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at Main._main(Main.java:268) at Main.main(Main.java:96)
            Hide
            oldelvet Richard Mortimer added a comment -

            I've been doing a bit more research and /var/cache/jenkins looks like the proper place for unpacking the war file. Tomcat6 on Ubuntu (and Debian) puts its Work directory in /var/cache.

            I should be able to sort out a patch/pull request in the next day or so.

            Show
            oldelvet Richard Mortimer added a comment - I've been doing a bit more research and /var/cache/jenkins looks like the proper place for unpacking the war file. Tomcat6 on Ubuntu (and Debian) puts its Work directory in /var/cache. I should be able to sort out a patch/pull request in the next day or so.
            oldelvet Richard Mortimer made changes -
            Assignee Richard Mortimer [ oldelvet ]
            oldelvet Richard Mortimer made changes -
            Summary Jenkins takes up too much space in /usr/run Jenkins takes up too much space in /var/run
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Richard Mortimer
            Path:
            changelog.html
            debian/debian/dirs
            debian/debian/jenkins.default
            debian/debian/jenkins.postinst
            debian/debian/jenkins.postrm
            http://jenkins-ci.org/commit/jenkins/9714917685e2b50f59b4feda9bc4e561870a944f
            Log:
            [FIXED JENKINS-11366] Jenkins takes up too much space in /var/run

            Use /var/cache/jenkins to hold the unpacked jenkins.war file at runtime.

            Compare: https://github.com/jenkinsci/jenkins/compare/a1714c1...9714917

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Richard Mortimer Path: changelog.html debian/debian/dirs debian/debian/jenkins.default debian/debian/jenkins.postinst debian/debian/jenkins.postrm http://jenkins-ci.org/commit/jenkins/9714917685e2b50f59b4feda9bc4e561870a944f Log: [FIXED JENKINS-11366] Jenkins takes up too much space in /var/run Use /var/cache/jenkins to hold the unpacked jenkins.war file at runtime. Compare: https://github.com/jenkinsci/jenkins/compare/a1714c1...9714917
            scm_issue_link SCM/JIRA link daemon made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #1331
            [FIXED JENKINS-11366] Jenkins takes up too much space in /var/run

            Kohsuke Kawaguchi : 9714917685e2b50f59b4feda9bc4e561870a944f
            Files :

            • debian/debian/jenkins.postrm
            • debian/debian/jenkins.default
            • debian/debian/dirs
            • changelog.html
            • debian/debian/jenkins.postinst
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #1331 [FIXED JENKINS-11366] Jenkins takes up too much space in /var/run Kohsuke Kawaguchi : 9714917685e2b50f59b4feda9bc4e561870a944f Files : debian/debian/jenkins.postrm debian/debian/jenkins.default debian/debian/dirs changelog.html debian/debian/jenkins.postinst
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Richard Mortimer
            Path:
            debian/dirs
            debian/jenkins.default
            debian/jenkins.postinst
            debian/jenkins.postrm
            http://jenkins-ci.org/commit/packaging/b98f3a38238b70d4ab2fd9183a7e8b6b9db39aff
            Log:
            [FIXED JENKINS-11366] Jenkins takes up too much space in /var/run

            Use /var/cache/jenkins to hold the unpacked jenkins.war file at runtime.

            Originally-From: jenkins-ci.org/commit/core/9714917685e2b50f59b4feda9bc4e561870a944f

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Richard Mortimer Path: debian/dirs debian/jenkins.default debian/jenkins.postinst debian/jenkins.postrm http://jenkins-ci.org/commit/packaging/b98f3a38238b70d4ab2fd9183a7e8b6b9db39aff Log: [FIXED JENKINS-11366] Jenkins takes up too much space in /var/run Use /var/cache/jenkins to hold the unpacked jenkins.war file at runtime. Originally-From: jenkins-ci.org/commit/core/9714917685e2b50f59b4feda9bc4e561870a944f
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 141592 ] JNJira + In-Review [ 189705 ]

              People

              • Assignee:
                oldelvet Richard Mortimer
                Reporter:
                francisdb francisdb
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: