Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43197

Deadlock when many jobs running

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Cannot Reproduce
    • Component/s: core
    • Labels:
      None
    • Environment:
      Jenkins 2.48
      xUnit plugin 1.102
      JUnit Plugin 1.20
      Plot Plugin 1.9
      RHEL6.3, Java 1.8.0_92
    • Similar Issues:

      Description

      After update of Jenkins from 2.8 to 2.48 nodes get stuck from time to time.
      Thread dump shows:

      "Executor #0 for my_node : executing my_job #510" #266539 daemon prio=5 os_prio=0 tid=0x00007f5fc1268800 nid=0x4885 waiting for monitor entry [0x00007f5f471f3000]
         java.lang.Thread.State: BLOCKED (on object monitor)
          at org.codehaus.groovy.reflection.ClassInfo$GlobalClassSet.add(ClassInfo.java:477)
          - waiting to lock <0x00007f621754d9c0> (a org.codehaus.groovy.util.ManagedLinkedList)
          at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:83)
          at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:79)
          at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue.<init>(GroovyClassValuePreJava7.java:37)
          at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:64)
          at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:55)
          at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.put(AbstractConcurrentMap.java:157)
          at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.getOrPut(AbstractConcurrentMap.java:100)
          at org.codehaus.groovy.util.AbstractConcurrentMap.getOrPut(AbstractConcurrentMap.java:38)
          at org.codehaus.groovy.reflection.GroovyClassValuePreJava7.get(GroovyClassValuePreJava7.java:94)
          at org.codehaus.groovy.reflection.ClassInfo.getClassInfo(ClassInfo.java:144)
          at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:258)
          at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:883)
          at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:34)
          at groovy.lang.Script.<init>(Script.java:42)
      :
      

      In changelog we found that with Jenkins version 2.47 Groovy was updated to 2.4.8 "to address memory leak issue".

      Possibly this issue is related to

      https://issues.apache.org/jira/browse/GROOVY-8067: Possible deadlock when creating new ClassInfo entries in the cache

      which is in Groovy 2.4.8 and solved in 2.4.9

       

       

        Attachments

          Issue Links

            Activity

            Hide
            danekan Dane Kantner added a comment - - edited

            I think I'm having the same issue after upgrading past 2.47.  I'm having constant deadlocks on monitoring jobs to the point I'm about to do a restore from backup or rebuild of the full system if not resolved shortly. The jobs themselves use Groovy to read a file that is used in GroovyAxis, just a small list of branch names.  Additionally I use the Extensible choice plug-in to read almost the same list from a txt file via groovy.  These text files are dynamically updated every few minutes but I have turned the updating off to rule out that file being locked at the time/etc. This has worked well for at least a year. After the updates, it seems to be all plug-ins that use Groovy that are of issue, though. When this happens I cannot load any jobs that use the extensible choice groovy plug-in either. The only real resolution seems to be a service restart. I've tried re-configuring threads to only 1/slave, and the opposite as well, adding slaves, etc. and it doesn't seem to be an issue of shortage of resources.

            The deadlocks seem to be a combination of jobs that are getting stuck in waiting on org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment@XYZ and then in turn that is causing other jobs to be stuck waiting on a list that job seems to hold hostage.  waiting on org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment@f3b6160   – that part is a big unexplainable as there's no real need it needs to be any sort of shared object. They are groovy scripts loading a groovyaxis of 3-5 branch names to that is then ran on an axis.

             

            Executor #-1 for dkxyz-8 : executing Monitor-ClientXYZ-VS2012 #90622

            "Executor #-1 for dkxyz-8 : executing Monitor-ClientXYZ-VS2012 #90622" Id=11917 Group=main BLOCKED on org.codehaus.groovy.util.ManagedLinkedList@53114cc9 owned by "Executor #-1 for dxyz-8 : executing Monitor-JMS-VS2012 #118161" Id=11908 at org.codehaus.groovy.reflection.ClassInfo$GlobalClassSet.add(ClassInfo.java:477) - blocked on org.codehaus.groovy.util.ManagedLinkedList@53114cc9 at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:83) at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:79) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue.<init>(GroovyClassValuePreJava7.java:37) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:64) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:55) at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.put(AbstractConcurrentMap.java:120) at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.getOrPut(AbstractConcurrentMap.java:100) at org.codehaus.groovy.util.AbstractConcurrentMap.getOrPut(AbstractConcurrentMap.java:38) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7.get(GroovyClassValuePreJava7.java:94) at org.codehaus.groovy.reflection.ClassInfo.getClassInfo(ClassInfo.java:144) at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:258) at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:883) at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:34) at groovy.lang.Script.<init>(Script.java:42) at Script1.<init>(Script1.groovy) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:431) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:584) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:623) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:594) at org.jenkinsci.plugins.GroovyAxis.evaluateGroovy(GroovyAxis.java:52) at org.jenkinsci.plugins.GroovyAxis.rebuild(GroovyAxis.java:40) at hudson.matrix.MatrixProject.rebuildConfigurations(MatrixProject.java:641) at hudson.matrix.MatrixProject.getRunConfiguration(MatrixProject.java:692) at hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:359) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534) at hudson.model.Run.execute(Run.java:1735) at hudson.matrix.MatrixBuild.run(MatrixBuild.java:313) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:405) Number of locked synchronizers = 1 - java.util.concurrent.locks.ReentrantLock$NonfairSync@dbed655

            above reference is waiting on this job:

            Executor #-1 for dkxyz-8 : executing MonitorJMS-VS2012 #118161

             

            "Executor #-1 for dkxyz-8 : executing MonitorJMS-VS2012 #118161" Id=11908 Group=main WAITING on org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment@f3b6160 at sun.misc.Unsafe.park(Native Method) - waiting on org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment@f3b6160 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source) at org.codehaus.groovy.util.LockableObject.lock(LockableObject.java:37) at org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment.removeEntry(AbstractConcurrentMapBase.java:173) at org.codehaus.groovy.util.ManagedConcurrentMap$Entry.finalizeReference(ManagedConcurrentMap.java:81) at org.codehaus.groovy.util.ManagedConcurrentMap$EntryWithValue.finalizeReference(ManagedConcurrentMap.java:115) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue.finalizeReference(GroovyClassValuePreJava7.java:51) at org.codehaus.groovy.util.ReferenceManager$CallBackedManager.removeStallEntries0(ReferenceManager.java:108) at org.codehaus.groovy.util.ReferenceManager$CallBackedManager.removeStallEntries(ReferenceManager.java:93) at org.codehaus.groovy.util.ReferenceManager$CallBackedManager.afterReferenceCreation(ReferenceManager.java:117) at org.codehaus.groovy.util.ReferenceManager$1.afterReferenceCreation(ReferenceManager.java:135) at org.codehaus.groovy.util.ManagedReference.<init>(ManagedReference.java:36) at org.codehaus.groovy.util.ManagedReference.<init>(ManagedReference.java:40) at org.codehaus.groovy.util.ManagedLinkedList$Element.<init>(ManagedLinkedList.java:40) at org.codehaus.groovy.util.ManagedLinkedList.add(ManagedLinkedList.java:102) at org.codehaus.groovy.reflection.ClassInfo$GlobalClassSet.add(ClassInfo.java:478) - locked org.codehaus.groovy.util.ManagedLinkedList@53114cc9 at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:83) at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:79) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue.<init>(GroovyClassValuePreJava7.java:37) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:64) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:55) at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.put(AbstractConcurrentMap.java:120) at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.getOrPut(AbstractConcurrentMap.java:100) at org.codehaus.groovy.util.AbstractConcurrentMap.getOrPut(AbstractConcurrentMap.java:38) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7.get(GroovyClassValuePreJava7.java:94) at org.codehaus.groovy.reflection.ClassInfo.getClassInfo(ClassInfo.java:144) at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:258) at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:883) at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:34) at groovy.lang.Script.<init>(Script.java:42) at Script1.<init>(Script1.groovy) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:431) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:584) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:623) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:594) at org.jenkinsci.plugins.GroovyAxis.evaluateGroovy(GroovyAxis.java:52) at org.jenkinsci.plugins.GroovyAxis.rebuild(GroovyAxis.java:40) at hudson.matrix.MatrixProject.rebuildConfigurations(MatrixProject.java:641) at hudson.matrix.MatrixProject.getRunConfiguration(MatrixProject.java:692) at hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:359) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534) at hudson.model.Run.execute(Run.java:1735) at hudson.matrix.MatrixBuild.run(MatrixBuild.java:313) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:405) Number of locked synchronizers = 1 - java.util.concurrent.locks.ReentrantLock$NonfairSync@744c1dc2

             

            Also at same time getting a lot of these GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment type errors:

            Executor #-1 for asc5-1 : executing Monitor-Portal-VS2012 #98314

            "Executor #-1 for asc5-1 : executing Monitor-Portal-VS2012 #98314" Id=11919 Group=main WAITING on org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment@18aa4da4 at sun.misc.Unsafe.park(Native Method) - waiting on org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment@18aa4da4 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source) at org.codehaus.groovy.util.LockableObject.lock(LockableObject.java:37) at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.put(AbstractConcurrentMap.java:104) at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.getOrPut(AbstractConcurrentMap.java:100) at org.codehaus.groovy.util.AbstractConcurrentMap.getOrPut(AbstractConcurrentMap.java:38) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7.get(GroovyClassValuePreJava7.java:94) at org.codehaus.groovy.reflection.ClassInfo.getClassInfo(ClassInfo.java:144) at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:258) at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:883) at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:34) at groovy.lang.Script.<init>(Script.java:42) at Script1.<init>(Script1.groovy) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:431) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:584) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:623) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:594) at org.jenkinsci.plugins.GroovyAxis.evaluateGroovy(GroovyAxis.java:52) at org.jenkinsci.plugins.GroovyAxis.rebuild(GroovyAxis.java:40) at hudson.matrix.MatrixProject.rebuildConfigurations(MatrixProject.java:641) at hudson.matrix.MatrixProject.getRunConfiguration(MatrixProject.java:692) at hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:359) at hudson.model

            Show
            danekan Dane Kantner added a comment - - edited I think I'm having the same issue after upgrading past 2.47.  I'm having constant deadlocks on monitoring jobs to the point I'm about to do a restore from backup or rebuild of the full system if not resolved shortly. The jobs themselves use Groovy to read a file that is used in GroovyAxis, just a small list of branch names.  Additionally I use the Extensible choice plug-in to read almost the same list from a txt file via groovy.  These text files are dynamically updated every few minutes but I have turned the updating off to rule out that file being locked at the time/etc. This has worked well for at least a year. After the updates, it seems to be all plug-ins that use Groovy that are of issue, though. When this happens I cannot load any jobs that use the extensible choice groovy plug-in either. The only real resolution seems to be a service restart. I've tried re-configuring threads to only 1/slave, and the opposite as well, adding slaves, etc. and it doesn't seem to be an issue of shortage of resources. The deadlocks seem to be a combination of jobs that are getting stuck in waiting on org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment@XYZ and then in turn that is causing other jobs to be stuck waiting on a list that job seems to hold hostage.  waiting on org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment@f3b6160   – that part is a big unexplainable as there's no real need it needs to be any sort of shared object. They are groovy scripts loading a groovyaxis of 3-5 branch names to that is then ran on an axis.   Executor #-1 for dkxyz-8 : executing Monitor-ClientXYZ-VS2012 #90622 "Executor #-1 for dkxyz-8 : executing Monitor-ClientXYZ-VS2012 #90622" Id=11917 Group=main BLOCKED on org.codehaus.groovy.util.ManagedLinkedList@53114cc9 owned by "Executor #-1 for dxyz-8 : executing Monitor-JMS-VS2012 #118161" Id=11908 at org.codehaus.groovy.reflection.ClassInfo$GlobalClassSet.add(ClassInfo.java:477) - blocked on org.codehaus.groovy.util.ManagedLinkedList@53114cc9 at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:83) at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:79) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue.<init>(GroovyClassValuePreJava7.java:37) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:64) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:55) at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.put(AbstractConcurrentMap.java:120) at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.getOrPut(AbstractConcurrentMap.java:100) at org.codehaus.groovy.util.AbstractConcurrentMap.getOrPut(AbstractConcurrentMap.java:38) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7.get(GroovyClassValuePreJava7.java:94) at org.codehaus.groovy.reflection.ClassInfo.getClassInfo(ClassInfo.java:144) at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:258) at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:883) at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:34) at groovy.lang.Script.<init>(Script.java:42) at Script1.<init>(Script1.groovy) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:431) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:584) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:623) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:594) at org.jenkinsci.plugins.GroovyAxis.evaluateGroovy(GroovyAxis.java:52) at org.jenkinsci.plugins.GroovyAxis.rebuild(GroovyAxis.java:40) at hudson.matrix.MatrixProject.rebuildConfigurations(MatrixProject.java:641) at hudson.matrix.MatrixProject.getRunConfiguration(MatrixProject.java:692) at hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:359) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534) at hudson.model.Run.execute(Run.java:1735) at hudson.matrix.MatrixBuild.run(MatrixBuild.java:313) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:405) Number of locked synchronizers = 1 - java.util.concurrent.locks.ReentrantLock$NonfairSync@dbed655 above reference is waiting on this job: Executor #-1 for dkxyz-8 : executing MonitorJMS-VS2012 #118161   "Executor #-1 for dkxyz-8 : executing MonitorJMS-VS2012 #118161" Id=11908 Group=main WAITING on org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment@f3b6160 at sun.misc.Unsafe.park(Native Method) - waiting on org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment@f3b6160 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source) at org.codehaus.groovy.util.LockableObject.lock(LockableObject.java:37) at org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment.removeEntry(AbstractConcurrentMapBase.java:173) at org.codehaus.groovy.util.ManagedConcurrentMap$Entry.finalizeReference(ManagedConcurrentMap.java:81) at org.codehaus.groovy.util.ManagedConcurrentMap$EntryWithValue.finalizeReference(ManagedConcurrentMap.java:115) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue.finalizeReference(GroovyClassValuePreJava7.java:51) at org.codehaus.groovy.util.ReferenceManager$CallBackedManager.removeStallEntries0(ReferenceManager.java:108) at org.codehaus.groovy.util.ReferenceManager$CallBackedManager.removeStallEntries(ReferenceManager.java:93) at org.codehaus.groovy.util.ReferenceManager$CallBackedManager.afterReferenceCreation(ReferenceManager.java:117) at org.codehaus.groovy.util.ReferenceManager$1.afterReferenceCreation(ReferenceManager.java:135) at org.codehaus.groovy.util.ManagedReference.<init>(ManagedReference.java:36) at org.codehaus.groovy.util.ManagedReference.<init>(ManagedReference.java:40) at org.codehaus.groovy.util.ManagedLinkedList$Element.<init>(ManagedLinkedList.java:40) at org.codehaus.groovy.util.ManagedLinkedList.add(ManagedLinkedList.java:102) at org.codehaus.groovy.reflection.ClassInfo$GlobalClassSet.add(ClassInfo.java:478) - locked org.codehaus.groovy.util.ManagedLinkedList@53114cc9 at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:83) at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:79) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue.<init>(GroovyClassValuePreJava7.java:37) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:64) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:55) at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.put(AbstractConcurrentMap.java:120) at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.getOrPut(AbstractConcurrentMap.java:100) at org.codehaus.groovy.util.AbstractConcurrentMap.getOrPut(AbstractConcurrentMap.java:38) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7.get(GroovyClassValuePreJava7.java:94) at org.codehaus.groovy.reflection.ClassInfo.getClassInfo(ClassInfo.java:144) at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:258) at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:883) at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:34) at groovy.lang.Script.<init>(Script.java:42) at Script1.<init>(Script1.groovy) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:431) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:584) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:623) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:594) at org.jenkinsci.plugins.GroovyAxis.evaluateGroovy(GroovyAxis.java:52) at org.jenkinsci.plugins.GroovyAxis.rebuild(GroovyAxis.java:40) at hudson.matrix.MatrixProject.rebuildConfigurations(MatrixProject.java:641) at hudson.matrix.MatrixProject.getRunConfiguration(MatrixProject.java:692) at hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:359) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534) at hudson.model.Run.execute(Run.java:1735) at hudson.matrix.MatrixBuild.run(MatrixBuild.java:313) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:405) Number of locked synchronizers = 1 - java.util.concurrent.locks.ReentrantLock$NonfairSync@744c1dc2   Also at same time getting a lot of these GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment type errors: Executor #-1 for asc5-1 : executing Monitor-Portal-VS2012 #98314 "Executor #-1 for asc5-1 : executing Monitor-Portal-VS2012 #98314" Id=11919 Group=main WAITING on org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment@18aa4da4 at sun.misc.Unsafe.park(Native Method) - waiting on org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment@18aa4da4 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source) at org.codehaus.groovy.util.LockableObject.lock(LockableObject.java:37) at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.put(AbstractConcurrentMap.java:104) at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.getOrPut(AbstractConcurrentMap.java:100) at org.codehaus.groovy.util.AbstractConcurrentMap.getOrPut(AbstractConcurrentMap.java:38) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7.get(GroovyClassValuePreJava7.java:94) at org.codehaus.groovy.reflection.ClassInfo.getClassInfo(ClassInfo.java:144) at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:258) at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:883) at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:34) at groovy.lang.Script.<init>(Script.java:42) at Script1.<init>(Script1.groovy) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:431) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:584) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:623) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:594) at org.jenkinsci.plugins.GroovyAxis.evaluateGroovy(GroovyAxis.java:52) at org.jenkinsci.plugins.GroovyAxis.rebuild(GroovyAxis.java:40) at hudson.matrix.MatrixProject.rebuildConfigurations(MatrixProject.java:641) at hudson.matrix.MatrixProject.getRunConfiguration(MatrixProject.java:692) at hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:359) at hudson.model
            Hide
            hjhafner Hans-Juergen Hafner added a comment -

            The workaround to start Jenkins with -Dgroovy.use.classvalue=true proposed at  GROOVY-8067 improved situation a lot for us.

            Show
            hjhafner Hans-Juergen Hafner added a comment - The workaround to start Jenkins with -Dgroovy.use.classvalue=true proposed at  GROOVY-8067 improved situation a lot for us.
            Hide
            io39d io39d added a comment - - edited

            We experiencing the same issue. Today 2 Jobs stuck, after some time Jenkins was unresponsive.

            Threaddump -> https://pastebin.com/WTF9QUPu (JIRA upload failed)

            Jenkins 2.57
            OpenJDK 1.8.0_121
            Debian Jessie

            Trying the workaround, let's see if it occurs again

            Show
            io39d io39d added a comment - - edited We experiencing the same issue. Today 2 Jobs stuck, after some time Jenkins was unresponsive. Threaddump -> https://pastebin.com/WTF9QUPu  (JIRA upload failed) Jenkins 2.57 OpenJDK 1.8.0_121 Debian Jessie Trying the workaround, let's see if it occurs again
            Hide
            danekan Dane Kantner added a comment -

            Thanks Hans, I've added -Dgroovy.use.classvalue=true to jenkins.xml config for startup w/ that flag and it seems to have resolved the Groovy issues...

             

            Is there a process used to update to the latest groovy library to the newer version where they've fixed this to actually resolve the underlying issue? I'm assuming it's not just upgraded because of regression resources?

            Show
            danekan Dane Kantner added a comment - Thanks Hans, I've added -Dgroovy.use.classvalue=true to jenkins.xml config for startup w/ that flag and it seems to have resolved the Groovy issues...   Is there a process used to update to the latest groovy library to the newer version where they've fixed this to actually resolve the underlying issue? I'm assuming it's not just upgraded because of regression resources?
            Hide
            danielbeck Daniel Beck added a comment -

            Jenkins 2.61 upgraded to Groovy 2.4.11. This change was considered too risky for LTS inclusion, so we decided on 2.60 as the LTS baseline after 2.46.x.

            I doubt a Groovy backport qualifies for LTS. Still, summoning Oliver Gondža.

            In terms of this issue, if the Groovy upgrade fixes it, it should be gone in 2.61 and newer. Please report your results on these versions of Jenkins.

             

            Show
            danielbeck Daniel Beck added a comment - Jenkins 2.61 upgraded to Groovy 2.4.11. This change was considered too risky for LTS inclusion, so we decided on 2.60 as the LTS baseline after 2.46.x. I doubt a Groovy backport qualifies for LTS. Still, summoning Oliver Gondža . In terms of this issue, if the Groovy upgrade fixes it, it should be gone in 2.61 and newer. Please report your results on these versions of Jenkins.  
            Hide
            danielbeck Daniel Beck added a comment -

            The next LTS baseline will be Jenkins 2.73, so if the change in 2.61 resolved this issue, it will be obsolete then.

            Show
            danielbeck Daniel Beck added a comment - The next LTS baseline will be Jenkins 2.73, so if the change in 2.61 resolved this issue, it will be obsolete then.
            Hide
            danielbeck Daniel Beck added a comment -

            Responses to JENKINS-33358 indicate this has been fixed. After months without further responses I'll assume that is the case here as well.

            Show
            danielbeck Daniel Beck added a comment - Responses to JENKINS-33358 indicate this has been fixed. After months without further responses I'll assume that is the case here as well.

              People

              • Assignee:
                Unassigned
                Reporter:
                hjhafner Hans-Juergen Hafner
              • Votes:
                6 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: