Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38503

Jenkins memory leak (Metaspace increases over the period)

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Blocker
    • Resolution: Duplicate
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      Every week we have to restart jenkins due to OOM. Continuously Metaspace increases and the number of classes loaded are also increasing.

      Earlier we had this problem with both master and windows slaves. But we figured out that disabling the JNLP3 protocol on windows slave and not running groovy script on slaves solved the problem. But. it is still a problem on master and restarting a master is not anyone is expecting.

      We recently upgraded our jenkins environment to 2.7.2 and took all the latest plugins.
      We have moved all our flow jobs to pipeline except one.

        Attachments

        1. heap.jpg
          heap.jpg
          349 kB
        2. metaspace.jpg
          metaspace.jpg
          246 kB

          Issue Links

            Activity

            Hide
            subbu_7518 subbarao p added a comment -

            increase your jenkins memory size

            Show
            subbu_7518 subbarao p added a comment - increase your jenkins memory size
            Hide
            varun_shrivastava_1987 varun shrivastava added a comment -

            you can refer to attachment see how metaspace is increasing

            Show
            varun_shrivastava_1987 varun shrivastava added a comment - you can refer to attachment see how metaspace is increasing
            Hide
            varun_shrivastava_1987 varun shrivastava added a comment -

            i have given 4GB heap

            Show
            varun_shrivastava_1987 varun shrivastava added a comment - i have given 4GB heap
            Hide
            danielbeck Daniel Beck added a comment -

            Duplicate of JENKINS-33358?

            Show
            danielbeck Daniel Beck added a comment - Duplicate of JENKINS-33358 ?
            Hide
            varun_shrivastava_1987 varun shrivastava added a comment - - edited

            After generating the heap dump and analyzing those as given in link https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+OutOfMemoryError

            I found the below instances and a lot of them
            this - value: groovy.lang.GroovyClassLoader #28
            <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #28
            <- <classLoader> - class: get_oas_releases$_run_closure5, value: groovy.lang.GroovyClassLoader$InnerLoader #50
            <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: get_oas_releases$_run_closure5 class get_oas_releases$_run_closure5
            <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #2204
            <- [10548] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2198
            <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #563500
            <- [9] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #7
            <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1
            <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1
            <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1
            <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo
            <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667
            <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14

            this - value: groovy.lang.GroovyClassLoader #39
            <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #39
            <- <classLoader> - class: Script1, value: groovy.lang.GroovyClassLoader$InnerLoader #61
            <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: Script1 class Script1
            <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #987
            <- [1] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2129
            <- [6906] - class: java.lang.Object[], value: java.lang.Object[] #825722
            <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #563500
            <- [9] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #7
            <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1
            <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1
            <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1
            <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo
            <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667
            <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14

            this - value: groovy.lang.GroovyClassLoader #106
            <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #106
            <- <classLoader> - class: get_oas_releases$_run_closure2, value: groovy.lang.GroovyClassLoader$InnerLoader #128
            <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: get_oas_releases$_run_closure2 class get_oas_releases$_run_closure2
            <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #2146
            <- [14122] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2113
            <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #577746
            <- [8] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #8
            <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1
            <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1
            <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1
            <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo
            <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667
            <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14

            this - value: groovy.lang.GroovyClassLoader #145
            <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #145
            <- <classLoader> - class: Script1, value: groovy.lang.GroovyClassLoader$InnerLoader #167
            <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: Script1 class Script1
            <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #1099
            <- [5219] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #883
            <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #557473
            <- [15] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #1
            <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1
            <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1
            <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1
            <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo
            <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667
            <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14

            Show
            varun_shrivastava_1987 varun shrivastava added a comment - - edited After generating the heap dump and analyzing those as given in link https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+OutOfMemoryError I found the below instances and a lot of them this - value: groovy.lang.GroovyClassLoader #28 <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #28 <- <classLoader> - class: get_oas_releases$_run_closure5, value: groovy.lang.GroovyClassLoader$InnerLoader #50 <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: get_oas_releases$_run_closure5 class get_oas_releases$_run_closure5 <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #2204 <- [10548] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2198 <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #563500 <- [9] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #7 <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1 <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1 <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1 <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667 <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14 this - value: groovy.lang.GroovyClassLoader #39 <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #39 <- <classLoader> - class: Script1, value: groovy.lang.GroovyClassLoader$InnerLoader #61 <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: Script1 class Script1 <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #987 <- [1] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2129 <- [6906] - class: java.lang.Object[], value: java.lang.Object[] #825722 <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #563500 <- [9] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #7 <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1 <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1 <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1 <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667 <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14 this - value: groovy.lang.GroovyClassLoader #106 <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #106 <- <classLoader> - class: get_oas_releases$_run_closure2, value: groovy.lang.GroovyClassLoader$InnerLoader #128 <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: get_oas_releases$_run_closure2 class get_oas_releases$_run_closure2 <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #2146 <- [14122] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2113 <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #577746 <- [8] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #8 <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1 <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1 <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1 <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667 <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14 this - value: groovy.lang.GroovyClassLoader #145 <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #145 <- <classLoader> - class: Script1, value: groovy.lang.GroovyClassLoader$InnerLoader #167 <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: Script1 class Script1 <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #1099 <- [5219] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #883 <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #557473 <- [15] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #1 <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1 <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1 <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1 <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667 <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14
            Hide
            danielbeck Daniel Beck added a comment -

            Looks like JENKINS-33358.

            Show
            danielbeck Daniel Beck added a comment - Looks like JENKINS-33358 .

              People

              • Assignee:
                vjuranek vjuranek
                Reporter:
                varun_shrivastava_1987 varun shrivastava
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: