Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42637

Jenkins stuck UI does not open at all

    Details

    • Similar Issues:

      Description

      Jenkins 2.49 Don't know what is actually the problem. Memory dump available for in-house usage via Dropbox etc. if needed. Please send me an email.

       

      Seems to be related to https://issues.apache.org/jira/browse/GROOVY-8067 Groovy bug. Don't know when Jenkins groovy will be update from 2.4.8 to 2.4.9 version .

        Attachments

          Issue Links

            Activity

            Hide
            svanoort Sam Van Oort added a comment -

            Heikki Simperi  We understand this is a critical issue for you, and in order to solve it, can you provide an isolated example that will reproduce this without depending on your infra?    

            I think the reason we're hesitant to update Groovy more is that every update seems to introduce a new issue of this sort, which generally requires a significant time investment from someone deeply specialized in this area. 

             

            Show
            svanoort Sam Van Oort added a comment - Heikki Simperi   We understand this is a critical issue for you, and in order to solve it, can you provide an isolated example that will reproduce this without depending on your infra?     I think the reason we're hesitant to update Groovy more is that every update seems to introduce a new issue of this sort, which generally requires a significant time investment from someone deeply specialized in this area.   
            Hide
            heikkisi Heikki Simperi added a comment -

            Sam Van Oort I will look into this. Groovy bug has java class to reproduce the problem in groovy but I understand that it is not quite relevant.

            Is there way to give bundled groovy command line parameter as suggested as fast fix?

             

            -Dgroovy.use.classvalue=true 

            Show
            heikkisi Heikki Simperi added a comment - Sam Van Oort I will look into this. Groovy bug has java class to reproduce the problem in groovy but I understand that it is not quite relevant. Is there way to give bundled groovy command line parameter as suggested as fast fix?   -Dgroovy.use.classvalue=true 
            Hide
            grayaii Alex Gray added a comment -

             

            Not sure if this will help, but in our /etc/sysconfig/jenkins you can added:

            JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Dgroovy.use.classvalue=true -Xms4096m -Xmx4096m -XX:MaxPermSize=1024m -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -Dhudson.model.ParametersAction.keepUndefinedParameters=true"
            

            We also run some crazy groovy code in some of jenkins jobs and we tend to run out of memory too, so we've installed this plugin to help us track java resources:

             

            https://wiki.jenkins-ci.org/display/JENKINS/Monitoring

            And finally, we periodically (once per hour i think) run this groovy script to clean things up:

            import net.bull.javamelody.*;
            
            before = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();
            System.gc(); 
            after = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();
            println I18N.getFormattedString("ramasse_miette_execute", Math.round((before - after) / 1024));
            
            

            And we launch it like this:

            java -jar jenkins-cli.jar -noCertificateCheck -i id_rsa -s JENKINS_URL groovy my_groovy_script.groovy'
            

            You can probably put that groovy script as a jenkins job have it run periodically.

            Ever since we we run that System.gc() command, we've never run out of memory.  We run 100s of jobs a day a AWS t2.medium without any downtime for months at a time.  Before I did these things, we were running on a huge instance and we had to restart it every week or so.

            Hope this helps!

             

            Show
            grayaii Alex Gray added a comment -   Not sure if this will help, but in our /etc/sysconfig/jenkins you can added: JENKINS_JAVA_OPTIONS= "-Djava.awt.headless= true -Dgroovy.use.classvalue= true -Xms4096m -Xmx4096m -XX:MaxPermSize=1024m -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -Dhudson.model.ParametersAction.keepUndefinedParameters= true " We also run some crazy groovy code in some of jenkins jobs and we tend to run out of memory too, so we've installed this plugin to help us track java resources:   https://wiki.jenkins-ci.org/display/JENKINS/Monitoring And finally, we periodically (once per hour i think) run this groovy script to clean things up: import net.bull.javamelody.*; before = Runtime .getRuntime().totalMemory() - Runtime .getRuntime().freeMemory(); System .gc(); after = Runtime .getRuntime().totalMemory() - Runtime .getRuntime().freeMemory(); println I18N.getFormattedString( "ramasse_miette_execute" , Math .round((before - after) / 1024)); And we launch it like this: java -jar jenkins-cli.jar -noCertificateCheck -i id_rsa -s JENKINS_URL groovy my_groovy_script.groovy' You can probably put that groovy script as a jenkins job have it run periodically. Ever since we we run that System.gc() command, we've never run out of memory.  We run 100s of jobs a day a AWS t2.medium without any downtime for months at a time.  Before I did these things, we were running on a huge instance and we had to restart it every week or so. Hope this helps!  
            Hide
            heikkisi Heikki Simperi added a comment -

            Hello!

            We could not make reproducible test case with small effort.

            However, the parameter tuning from Groovy ticket seem to resolve the state. Now Jenkins has been working for one whole week without problems!

             

            Show
            heikkisi Heikki Simperi added a comment - Hello! We could not make reproducible test case with small effort. However, the parameter tuning from Groovy ticket seem to resolve the state. Now Jenkins has been working for one whole week without problems!  
            Hide
            danekan Dane Kantner added a comment -

            From another bug where it was mentioned,  -Dgroovy.use.classvalue=true  seems to have fixed this for us as well. We actually don't use anything crazy at all in Groovy (I use it to read load a 3 line text file into a parameter list and there are try/catch blocks around that read operation even to fail to a default "" response), nonetheless the bugs in this version brought the entire platform to a state of frequent unavailability.  There are other bugs open related to this but it's hard to picture how this is "resolved" when it's such a major flaw that has already been fixed in groovy itself, though Jenkins still needs to upgrade to using that newer version internally.

            Show
            danekan Dane Kantner added a comment - From another bug where it was mentioned,  -Dgroovy.use.classvalue=true  seems to have fixed this for us as well. We actually don't use anything crazy at all in Groovy (I use it to read load a 3 line text file into a parameter list and there are try/catch blocks around that read operation even to fail to a default "" response), nonetheless the bugs in this version brought the entire platform to a state of frequent unavailability.  There are other bugs open related to this but it's hard to picture how this is "resolved" when it's such a major flaw that has already been fixed in groovy itself, though Jenkins still needs to upgrade to using that newer version internally.

              People

              • Assignee:
                Unassigned
                Reporter:
                heikkisi Heikki Simperi
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: