Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45010

Cron thread throw an exception java.lang.OutOfMemoryError: GC overhead limit exceeded

    Details

    • Similar Issues:

      Description

      Command to start Jenkins:

      /usr/bin/java -Xmx1024m -Xms1024m -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.rmi.port=1099 -Djava.rmi.server.hostname=10.0.0.66 -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080 --ajp13Port=-1

       

      Log error:

      Jun 20, 2017 9:06:48 AM org.eclipse.jetty.server.session.Session beginInvalidate
      INFO: Session node0o10k68itvv7217ppw6c9hmqa841 already being invalidated
      Jun 20, 2017 9:34:09 AM com.cloudbees.hudson.plugins.folder.computed.FolderCron doRun
      WARNING: Cron thread throw an exception
      java.lang.OutOfMemoryError: GC overhead limit exceeded
      at java.util.regex.Matcher.<init>(Matcher.java:225)
      at java.util.regex.Pattern.matcher(Pattern.java:1093)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$2.perform(RoleMap.java:329)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.walk(RoleMap.java:387)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.<init>(RoleMap.java:376)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$2.<init>(RoleMap.java:327)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.getMatchingRoles(RoleMap.java:327)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.newMatchingRoleMap(RoleMap.java:283)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:124)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:141)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:136)
      at hudson.model.Job.getACL(Job.java:1599)
      at hudson.model.AbstractItem.hasPermission(AbstractItem.java:506)
      at com.cloudbees.hudson.plugins.folder.AbstractFolder.getItems(AbstractFolder.java:1146)
      at hudson.model.Items.getAllItems(Items.java:407)
      at hudson.model.Items.getAllItems(Items.java:417)
      at hudson.model.Items.getAllItems(Items.java:403)
      at jenkins.model.Jenkins.getAllItems(Jenkins.java:1757)
      at com.cloudbees.hudson.plugins.folder.computed.FolderCron.checkTriggers(FolderCron.java:117)
      at com.cloudbees.hudson.plugins.folder.computed.FolderCron.doRun(FolderCron.java:99)
      at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
      at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)

      Jun 20, 2017 9:34:20 AM hudson.triggers.Trigger$Cron doRun
      WARNING: Cron thread throw an exception
      java.lang.OutOfMemoryError: GC overhead limit exceeded
      at com.google.common.cache.AbstractCache$SimpleStatsCounter.<init>(AbstractCache.java:210)
      at com.google.common.cache.CacheBuilder$2.get(CacheBuilder.java:170)
      at com.google.common.cache.CacheBuilder$2.get(CacheBuilder.java:167)
      at com.google.common.cache.LocalCache.<init>(LocalCache.java:296)
      at com.google.common.cache.LocalCache$LocalManualCache.<init>(LocalCache.java:4750)
      at com.google.common.cache.LocalCache$LocalManualCache.<init>(LocalCache.java:4745)
      at com.google.common.cache.CacheBuilder.build(CacheBuilder.java:757)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.<init>(RoleMap.java:86)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.newMatchingRoleMap(RoleMap.java:288)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:124)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:141)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:136)
      at hudson.model.Job.getACL(Job.java:1599)
      at hudson.model.AbstractItem.hasPermission(AbstractItem.java:506)
      at com.cloudbees.hudson.plugins.folder.AbstractFolder.getItems(AbstractFolder.java:1146)
      at hudson.model.Items$AllItemsIterable$AllItemsIterator.hasNext(Items.java:583)
      at hudson.triggers.Trigger.checkTriggers(Trigger.java:269)
      at hudson.triggers.Trigger$Cron.doRun(Trigger.java:226)
      at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
      at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)

      java.lang.OutOfMemoryError: GC overhead limit exceeded
      at java.util.regex.Matcher.<init>(Matcher.java:225)
      at java.util.regex.Pattern.matcher(Pattern.java:1093)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$2.perform(RoleMap.java:329)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.walk(RoleMap.java:387)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.<init>(RoleMap.java:376)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$2.<init>(RoleMap.java:327)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.getMatchingRoles(RoleMap.java:327)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.newMatchingRoleMap(RoleMap.java:283)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:124)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:141)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:136)
      at hudson.model.Job.getACL(Job.java:1599)
      at hudson.model.AbstractItem.hasPermission(AbstractItem.java:506)
      at com.cloudbees.hudson.plugins.folder.AbstractFolder.getItems(AbstractFolder.java:1146)
      at hudson.model.Items.getAllItems(Items.java:407)
      at hudson.model.Items.getAllItems(Items.java:417)
      at hudson.model.Items.getAllItems(Items.java:403)
      at jenkins.model.Jenkins.getAllItems(Jenkins.java:1757)
      at com.cloudbees.hudson.plugins.folder.computed.FolderCron.checkTriggers(FolderCron.java:117)
      at com.cloudbees.hudson.plugins.folder.computed.FolderCron.doRun(FolderCron.java:99)
      at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
      at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)

      Jun 20, 2017 9:34:32 AM net.bull.javamelody.JavaLogger warn
      WARNING: exception while collecting data: java.lang.OutOfMemoryError: GC overhead limit exceeded
      java.lang.OutOfMemoryError: GC overhead limit exceeded
      at net.bull.javamelody.CounterRequest.clone(CounterRequest.java:420)
      at net.bull.javamelody.Collector.collectCounterRequestData(Collector.java:813)
      at net.bull.javamelody.Collector.collectCounterRequestsAndErrorsData(Collector.java:759)
      at net.bull.javamelody.Collector.collectCounterData(Collector.java:747)
      at net.bull.javamelody.Collector.collect(Collector.java:380)
      at net.bull.javamelody.Collector.collectWithoutErrors(Collector.java:352)
      at net.bull.javamelody.Collector.collectLocalContextWithoutErrors(Collector.java:341)
      at net.bull.javamelody.FilterContext$CollectTimerTask.run(FilterContext.java:64)
      at java.util.TimerThread.mainLoop(Timer.java:555)
      at java.util.TimerThread.run(Timer.java:505)

      Jun 20, 2017 9:34:42 AM net.bull.javamelody.JavaLogger warn
      WARNING: exception while collecting data
      java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
      at hudson.remoting.Channel$2.adapt(Channel.java:862)
      at hudson.remoting.Channel$2.adapt(Channel.java:857)
      at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
      at net.bull.javamelody.RemoteCallHelper.collectDataByNodeName(RemoteCallHelper.java:172)
      at net.bull.javamelody.RemoteCallHelper.collectJavaInformationsListByName(RemoteCallHelper.java:182)
      at net.bull.javamelody.NodesCollector.collectWithoutErrorsNow(NodesCollector.java:153)
      at net.bull.javamelody.NodesCollector.collectWithoutErrors(NodesCollector.java:142)
      at net.bull.javamelody.NodesCollector$1.run(NodesCollector.java:86)
      at java.util.TimerThread.mainLoop(Timer.java:555)
      at java.util.TimerThread.run(Timer.java:505)
      Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
      at java.lang.Class.getDeclaredMethods0(Native Method)
      at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
      at java.lang.Class.getDeclaredMethod(Class.java:2128)
      at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475)
      at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72)
      at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498)
      at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
      at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
      at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598)
      at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)
      at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
      at hudson.remoting.UserRequest.deserialize(UserRequest.java:217)
      at hudson.remoting.UserResponse.retrieve(UserRequest.java:250)
      at hudson.remoting.Channel$2.adapt(Channel.java:860)
      ... 9 more

        Attachments

        1. jenkins_VisualVm_2017.06.27_0441pm.png
          jenkins_VisualVm_2017.06.27_0441pm.png
          89 kB
        2. jenkins.log
          3.06 MB
        3. jenkins.log.1
          5.65 MB
        4. Printscreen_with_error_on_VisualVM.png
          Printscreen_with_error_on_VisualVM.png
          264 kB
        5. screenshot-1.png
          screenshot-1.png
          117 kB
        6. threaddump-2017.06.21_0234pm.tdump
          211 kB
        7. threaddump-2017.06.27_0519pm.tdump
          106 kB

          Activity

          Hide
          giselasqa Gisela Nogueira added a comment -

          Oleg Nenashev,

          The performance problem with Jenkins occurred again today. All jobs was stopped at the peak moment. I sow my Jenkins and didn't have anyone information about de problem, anything Exception and anything error.

          My VisualVm image error:

          Show
          giselasqa Gisela Nogueira added a comment - Oleg Nenashev , The performance problem with Jenkins occurred again today. All jobs was stopped at the peak moment. I sow my Jenkins and didn't have anyone information about de problem, anything Exception and anything error. My VisualVm image error:
          Hide
          giselasqa Gisela Nogueira added a comment -

          I had updated my environment to the last Jenkins version (2.66) and the max memory heap to 2048M.

          Show
          giselasqa Gisela Nogueira added a comment - I had updated my environment to the last Jenkins version (2.66) and the max memory heap to 2048M.
          Hide
          giselasqa Gisela Nogueira added a comment -

          Hi Oleg Nenashev,

          I've changed my JVM parameters. I increased JVM memory from default to 2024 MBs and configured the garbage collector.

          Jenkins log information:

          The problema was started at 04:41 pm but in the jenkins.log haven't any information about the performance problems.

          Part of jenkins.log

          Jun 27, 2017 4:35:39 PM hudson.model.Run execute
          INFO: Immobile/IMMOBILE-FINANCEIRO #3 main build action completed: FAILURE
          Jun 27, 2017 4:46:45 PM org.eclipse.jetty.server.session.Session beginInvalidate
          INFO: Session node01pbn3vxmf3g60180w3ga3638yl24339 already being invalidated

           

          Attached files:

          I'll attach more files to help you.

          • jenkins_VisualVm_2017.06.27_0441pm.png: Print screen of VisualVm at the problem moment.
          • threaddump-2017.06.27_0519pm.tdump: Thread dump of Jenkins service

           

          Follows my JVM arguments:

          -Xmx2048m
          -Xms1024m
          -Djava.awt.headless=true
          -Dcom.sun.management.jmxremote
          -Dcom.sun.management.jmxremote.authenticate=false
          -Dcom.sun.management.jmxremote.ssl=false
          -Dcom.sun.management.jmxremote.port=1099
          -Dcom.sun.management.jmxremote.rmi.port=1099
          -Djava.rmi.server.hostname=10.0.0.66
          -verbose:gc
          -XX:+HeapDumpOnOutOfMemoryError
          -XX:HeapDumpPath=/var/log/jenkins/
          Xloggc:$/var/log/jenkins/gc%t.log
          -XX:NumberOfGCLogFiles=5
          -XX:+UseGCLogFileRotation
          -XX:GCLogFileSize=20m
          -XX:+PrintGC
          -XX:+PrintGCDateStamps
          -XX:+PrintGCDetails
          -XX:+PrintHeapAtGC
          -XX:+PrintGCCause
          -XX:+PrintTenuringDistribution
          -XX:+PrintReferenceGC
          -XX:+PrintAdaptiveSizePolicy
          -XX:+UseConcMarkSweepGC
          -XX:+CMSParallelRemarkEnabled
          -XX:+ParallelRefProcEnabled
          -XX:+CMSClassUnloadingEnabled
          -XX:+ScavengeBeforeFullGC
          -XX:+CMSScavengeBeforeRemark
          -XX:NewSize=512m
          -XX:MaxNewSize=512m
          -XX:NewRatio=2

           

           

          If you need for more information, you can ask to me.

          Show
          giselasqa Gisela Nogueira added a comment - Hi Oleg Nenashev , I've changed my JVM parameters. I increased JVM memory from default to 2024 MBs and configured the garbage collector. Jenkins log information : The problema was started at 04:41 pm but in the jenkins.log haven't any information about the performance problems. Part of jenkins.log Jun 27, 2017 4:35:39 PM hudson.model.Run execute INFO: Immobile/IMMOBILE-FINANCEIRO #3 main build action completed: FAILURE Jun 27, 2017 4:46:45 PM org.eclipse.jetty.server.session.Session beginInvalidate INFO: Session node01pbn3vxmf3g60180w3ga3638yl24339 already being invalidated   Attached files : I'll attach more files to help you. jenkins_VisualVm_2017.06.27_0441pm.png: Print screen of VisualVm at the problem moment. threaddump-2017.06.27_0519pm.tdump: Thread dump of Jenkins service   Follows my JVM arguments : -Xmx2048m -Xms1024m -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.rmi.port=1099 -Djava.rmi.server.hostname=10.0.0.66 -verbose:gc -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/jenkins/ Xloggc:$/var/log/jenkins/gc %t.log -XX:NumberOfGCLogFiles=5 -XX:+UseGCLogFileRotation -XX:GCLogFileSize=20m -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCCause -XX:+PrintTenuringDistribution -XX:+PrintReferenceGC -XX:+PrintAdaptiveSizePolicy -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:+CMSClassUnloadingEnabled -XX:+ScavengeBeforeFullGC -XX:+CMSScavengeBeforeRemark -XX:NewSize=512m -XX:MaxNewSize=512m -XX:NewRatio=2     If you need for more information, you can ask to me.
          Hide
          giselasqa Gisela Nogueira added a comment -

          We identified the problem and solved them.

          Show
          giselasqa Gisela Nogueira added a comment - We identified the problem and solved them.
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Gisela Nogueira so what was the root cause?

          Show
          oleg_nenashev Oleg Nenashev added a comment - Gisela Nogueira so what was the root cause?

            People

            • Assignee:
              Unassigned
              Reporter:
              giselasqa Gisela Nogueira
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: