Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
    • Environment:
      Jenkins 1.537 installed on CentOS 6.2 as RPM and running as a service.
    • Similar Issues:

      Description

      Images, such as build icons, start disappearing sporadically from the Jenkins UI. The UI instead displays a "broken image" icon. When accessing the image's URL, such as e.g. http://<jenkins-host>:6080/static/cc7624e2/images/32x32/health-00to19.png, a 404 error is returned.

      Analysis:
      The images appear to be served from the /tmp/jetty-0.0.0.0-6080-jenkins.war-any/webapp/images/ folder. By default, CentOS runs a daily cron of tmpwatch, which deletes files under /tmp that haven't been accessed in 10 days. So image files are deleted if Jenkins is running for more than 10 days, and an image hasn't been accessed (for example, the health-00to19.png won't be accessed if there were no builds with <20% health lately).

      Workaround:
      I was able to resolve the problem by editing /etc/sysconfig/jenkins and adding " -Djava.io.tmpdir=/var/tmp" to JENKINS_JAVA_OPTIONS. However, it would be better if Jenkins came correctly configured out-of-the-box.

        Attachments

          Issue Links

            Activity

            eliac Eli Acherkan created issue -
            kraxel Sven Keller made changes -
            Field Original Value New Value
            Attachment screenshot-1 jenkins on IE.jpg [ 24764 ]
            Hide
            kraxel Sven Keller added a comment - - edited

            We also experience a similar issue. could it be the same?
            see screenshots
            on IE anf FF
            We use 1.541

            Show
            kraxel Sven Keller added a comment - - edited We also experience a similar issue. could it be the same? see screenshots on IE anf FF We use 1.541
            Hide
            kraxel Sven Keller added a comment -

            missing icons only in manage jenkins view

            Show
            kraxel Sven Keller added a comment - missing icons only in manage jenkins view
            kraxel Sven Keller made changes -
            Attachment screenshot-2 firefox.jpg [ 24765 ]
            Hide
            belpk K P added a comment -

            Yes, mostly this issue is caused by people clearing out the TEMP where jetty extracted the war; while jenkins is still running.

            This is the reason I suggested yesterday on this issue: JENKINS-17526 , where people mention missing css and png files (of course, not only images are served by jetty from the TEMP, but also the .css files)

            Show
            belpk K P added a comment - Yes, mostly this issue is caused by people clearing out the TEMP where jetty extracted the war; while jenkins is still running. This is the reason I suggested yesterday on this issue: JENKINS-17526 , where people mention missing css and png files (of course, not only images are served by jetty from the TEMP, but also the .css files)
            belpk K P made changes -
            Link This issue is related to JENKINS-17526 [ JENKINS-17526 ]
            Hide
            belpk K P added a comment -

            I guess for most people clearing $TEMP may be the cause for missing png and css files (error 404 by jetty)

            Show
            belpk K P added a comment - I guess for most people clearing $TEMP may be the cause for missing png and css files (error 404 by jetty)
            belpk K P made changes -
            Link This issue duplicates JENKINS-17526 [ JENKINS-17526 ]
            belpk K P made changes -
            Link This issue duplicates JENKINS-17526 [ JENKINS-17526 ]
            Hide
            jameshowe James Howe added a comment -

            The same problem seems to also result in class files being deleted if not used for a while, which as you can imagine leads to a lot of problems when you do try to use them.

            Show
            jameshowe James Howe added a comment - The same problem seems to also result in class files being deleted if not used for a while, which as you can imagine leads to a lot of problems when you do try to use them.
            Hide
            kraxel Sven Keller added a comment -

            We currently face this issue again (jenkins running <24h)
            We don't use the rpm version. what is the TEMP dir jenkins uses then?

            Show
            kraxel Sven Keller added a comment - We currently face this issue again (jenkins running <24h) We don't use the rpm version. what is the TEMP dir jenkins uses then?
            Hide
            samxiao Sam Xiao added a comment -

            still happening on 1.539

            Show
            samxiao Sam Xiao added a comment - still happening on 1.539
            samxiao Sam Xiao made changes -
            Attachment z.png [ 24773 ]
            Hide
            kraxel Sven Keller added a comment -

            it is indeed a problem with deleted files in the $TEMP
            since when was it introduced to store jetty files in $TEMP?
            Why was this done?

            Show
            kraxel Sven Keller added a comment - it is indeed a problem with deleted files in the $TEMP since when was it introduced to store jetty files in $TEMP? Why was this done?
            gaige Gaige Paulsen made changes -
            Link This issue is related to JENKINS-20682 [ JENKINS-20682 ]
            Hide
            gaige Gaige Paulsen added a comment -

            These seem to be rooted in the same issue

            Show
            gaige Gaige Paulsen added a comment - These seem to be rooted in the same issue
            swiniak Andrzej Pasterczyk made changes -
            Link This issue is related to JENKINS-21133 [ JENKINS-21133 ]
            Hide
            oldelvet Richard Mortimer added a comment -

            I'm pretty certain that is is due to the underlying JENKINS-20682 problem. That moves the folder where the jenkins.war is exploded into a location chosen by Jetty (TEMP Dir). However the old location pointed to by --webroot (/var/cache/jenkins/war in my Ubuntu system) is not cleared out and contains the last exploded version of the jenkins.war file.

            Jenkins still seems to be using the --webroot version of the exploded war to serve up images rather than using the most recently exploded war in the TEMP dir. Thus any added/changed images are not picked up and either result in a 404 not-found or the old version being served.

            I worked around this by shutting down my Jenkins and moving the old exploded war file out of the way

            sudo mv /var/cache/jenkins/war /var/cache/jenkins/war.gone

            Note you can remove the /var/cache/jenkins/war directory but I played safe for now just in case I needed to put it back.

            When I restarted Jenkins it fell back to using the images from the exploded WAR in the TEMP dir and all is happy.

            The underlying issue obviously needs resolving but this workaround should help people to get the missing images back.

            Show
            oldelvet Richard Mortimer added a comment - I'm pretty certain that is is due to the underlying JENKINS-20682 problem. That moves the folder where the jenkins.war is exploded into a location chosen by Jetty (TEMP Dir). However the old location pointed to by --webroot (/var/cache/jenkins/war in my Ubuntu system) is not cleared out and contains the last exploded version of the jenkins.war file. Jenkins still seems to be using the --webroot version of the exploded war to serve up images rather than using the most recently exploded war in the TEMP dir. Thus any added/changed images are not picked up and either result in a 404 not-found or the old version being served. I worked around this by shutting down my Jenkins and moving the old exploded war file out of the way sudo mv /var/cache/jenkins/war /var/cache/jenkins/war.gone Note you can remove the /var/cache/jenkins/war directory but I played safe for now just in case I needed to put it back. When I restarted Jenkins it fell back to using the images from the exploded WAR in the TEMP dir and all is happy. The underlying issue obviously needs resolving but this workaround should help people to get the missing images back.
            Hide
            franco_salas Franco Salas added a comment - - edited

            Still occurs in version 1.549 (installed as a windows 7 service)

            Edit:

            We found out the problem was that we had a bot periodically cleaning the TEMP folder and so it would destroy all the Jenkins images and CSS files. Why are all images and CSS files stored in the TEMP folder? It seems to be counter productive to put all these files in a temporary folder which is shared by other applications.

            Show
            franco_salas Franco Salas added a comment - - edited Still occurs in version 1.549 (installed as a windows 7 service) Edit: We found out the problem was that we had a bot periodically cleaning the TEMP folder and so it would destroy all the Jenkins images and CSS files. Why are all images and CSS files stored in the TEMP folder? It seems to be counter productive to put all these files in a temporary folder which is shared by other applications.
            Hide
            t0yv0 Anton Tayanovskyy added a comment -

            Still having this issue for 1.553 running via scheduled task on Windows 8.1.

            Workaround with a custom temporary folder DID NOT WORK. Starts ok, but after a while the images are missing, and indeed files are deleted from the custom tmp folder. Is JVM itself or parts of Jenkins doing the cleanup of tmp?

            Show
            t0yv0 Anton Tayanovskyy added a comment - Still having this issue for 1.553 running via scheduled task on Windows 8.1. Workaround with a custom temporary folder DID NOT WORK. Starts ok, but after a while the images are missing, and indeed files are deleted from the custom tmp folder. Is JVM itself or parts of Jenkins doing the cleanup of tmp?
            Hide
            jameshowe James Howe added a comment -

            It's is almost certainly not Jenkins nor the JVM that deletes the files.
            Whatever on your system is cleaning temp folders is also looking at your custom one, whatever that is.

            Show
            jameshowe James Howe added a comment - It's is almost certainly not Jenkins nor the JVM that deletes the files. Whatever on your system is cleaning temp folders is also looking at your custom one, whatever that is.
            Hide
            t0yv0 Anton Tayanovskyy added a comment -

            @James, well, how is it possible if this keeps happening when I pick the folder at random? Sounds like the only system aware that this folder is "temporary" is JVM/Jenkins.

            Show
            t0yv0 Anton Tayanovskyy added a comment - @James, well, how is it possible if this keeps happening when I pick the folder at random? Sounds like the only system aware that this folder is "temporary" is JVM/Jenkins.
            Hide
            randroid Roberto Andrade added a comment -
            Show
            randroid Roberto Andrade added a comment - This seems to have been resolved on Jenkins 1.555: https://github.com/jenkinsci/jenkins/commit/7c51cec55649b0e46047cb5e3198b66c3bdd94d5
            Hide
            oldelvet Richard Mortimer added a comment -

            See JENKINS-22442
            It should fix the issue reported here without requiring the tmpdir hack in 1.555

            Show
            oldelvet Richard Mortimer added a comment - See JENKINS-22442 It should fix the issue reported here without requiring the tmpdir hack in 1.555
            Hide
            swf Yves Schumann added a comment -

            We're facing the same problem with 1.557. Any news here?

            Show
            swf Yves Schumann added a comment - We're facing the same problem with 1.557. Any news here?
            Hide
            oldelvet Richard Mortimer added a comment -

            The fix went into 1.558. I have that running here and it is now unpacking the WAR file into /var/cache/jenkins/war as per the --webroot setting.

            Show
            oldelvet Richard Mortimer added a comment - The fix went into 1.558. I have that running here and it is now unpacking the WAR file into /var/cache/jenkins/war as per the --webroot setting.
            danielbeck Daniel Beck made changes -
            Link This issue duplicates JENKINS-22442 [ JENKINS-22442 ]
            Hide
            danielbeck Daniel Beck added a comment -

            Duplicate of JENKINS-22442 which was fixed in 1.558 and 1.554.1.

            Show
            danielbeck Daniel Beck added a comment - Duplicate of JENKINS-22442 which was fixed in 1.558 and 1.554.1.
            danielbeck Daniel Beck made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Duplicate [ 3 ]
            Hide
            intel352 Jonathan Langevin added a comment - - edited

            In 1.576, seeing a similar issue where plugin images for the left nav menu are broken. Inspecting the images in browser shows URL of

            //plugin/config-rotator/images/rotate.png

            Jenkins is installed via Ubuntu APT, running as a service.

            Show
            intel352 Jonathan Langevin added a comment - - edited In 1.576, seeing a similar issue where plugin images for the left nav menu are broken. Inspecting the images in browser shows URL of //plugin/config-rotator/images/rotate.png Jenkins is installed via Ubuntu APT, running as a service.
            intel352 Jonathan Langevin made changes -
            Resolution Duplicate [ 3 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            Assignee Richard Mortimer [ oldelvet ]
            Hide
            oldelvet Richard Mortimer added a comment -

            This doesn't look like the same problem. The URL you included looks to be wrong there should be something between the "//" at the start of the URL. This needs logging as a separate issue probably against the config-rotator plugin.

            Show
            oldelvet Richard Mortimer added a comment - This doesn't look like the same problem. The URL you included looks to be wrong there should be something between the " // " at the start of the URL. This needs logging as a separate issue probably against the config-rotator plugin.
            Hide
            intel352 Jonathan Langevin added a comment - - edited

            This issue isn't specific to config-rotator, as there are multiple broken plugin images in the menu.

            I'll get a new bug report created later this morning

            Show
            intel352 Jonathan Langevin added a comment - - edited This issue isn't specific to config-rotator, as there are multiple broken plugin images in the menu. I'll get a new bug report created later this morning
            intel352 Jonathan Langevin made changes -
            Attachment Screenshot 2014-08-19 10.55.02.png [ 26611 ]
            Hide
            jameshowe James Howe added a comment -

            Already been done: JENKINS-24316

            Show
            jameshowe James Howe added a comment - Already been done: JENKINS-24316
            Hide
            intel352 Jonathan Langevin added a comment -

            Thanks for the link James

            Show
            intel352 Jonathan Langevin added a comment - Thanks for the link James
            intel352 Jonathan Langevin made changes -
            Status Reopened [ 4 ] Closed [ 6 ]
            Assignee Richard Mortimer [ oldelvet ]
            Resolution Fixed [ 1 ]
            Hide
            danielbeck Daniel Beck added a comment -

            Just in case someone visits:

            This is not the issue that occurs after upgrade to 1.576. Visit JENKINS-24316

            Show
            danielbeck Daniel Beck added a comment - Just in case someone visits: This is not the issue that occurs after upgrade to 1.576. Visit JENKINS-24316
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 152669 ] JNJira + In-Review [ 207224 ]

              People

              • Assignee:
                Unassigned
                Reporter:
                eliac Eli Acherkan
              • Votes:
                16 Vote for this issue
                Watchers:
                20 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: