Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48505

Lock contention in ExtensionList.getExtensionList

    Details

    • Similar Issues:

      Description

      Upgraded Jenkins from 2.66 to 2.89.1 and hit lock contention on ExtensionList.getExtensionList

      maybe caused by the change #24ede5

      
           @SuppressWarnings({"unchecked"})
           public <T> ExtensionList<T> getExtensionList(Class<T> extensionType) {
      -        return extensionLists.get(extensionType);
      +        return extensionLists.computeIfAbsent(extensionType, key -> ExtensionList.create(this, key));
           }
      
      

      Sample code:
      GlobalConfiguration.all().get(SplunkJenkinsInstallation.class);
      source: https://github.com/splunk/splunkforjenkins/blob/master/splunk-devops/src/main/java/com/splunk/splunkjenkins/SplunkJenkinsInstallation.java#L106

      stack trace

      "org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#793]" #190991 daemon prio=5 os_prio=0 tid=0x00007fb38f2f8800 nid=0xe54 waiting for monitor entry [0x00007fb3632dc000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
              - waiting to lock <0x0000000342d647c8> (a java.util.concurrent.ConcurrentHashMap$Node)
              at jenkins.model.Jenkins.getDescriptorList(Jenkins.java:2652)
              at jenkins.model.GlobalConfiguration.all(GlobalConfiguration.java:73)
              at com.splunk.splunkjenkins.SplunkJenkinsInstallation.get(SplunkJenkinsInstallation.java:105)
              at com.splunk.splunkjenkins.utils.LogFileCallable.send(LogFileCallable.java:90)
              at com.splunk.splunkjenkins.utils.LogFileCallable.sendFiles(LogFileCallable.java:57)
              at com.splunk.splunkjenkins.utils.LogEventHelper.sendFiles(LogEventHelper.java:282)
              at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:111)
              at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:92)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
              at hudson.security.ACL.impersonate(ACL.java:260)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      "org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#673]" #155842 daemon prio=5 os_prio=0 tid=0x00007fb38f3c6800 nid=0xe7d8 waiting for monitor entry [0x00007fb2ee566000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
              - locked <0x0000000342d647c8> (a java.util.concurrent.ConcurrentHashMap$Node)
              at jenkins.model.Jenkins.getDescriptorList(Jenkins.java:2652)
              at jenkins.model.GlobalConfiguration.all(GlobalConfiguration.java:73)
              at com.splunk.splunkjenkins.SplunkJenkinsInstallation.get(SplunkJenkinsInstallation.java:105)
              at com.splunk.splunkjenkins.utils.LogFileCallable.send(LogFileCallable.java:90)
              at com.splunk.splunkjenkins.utils.LogFileCallable.sendFiles(LogFileCallable.java:57)
              at com.splunk.splunkjenkins.utils.LogEventHelper.sendFiles(LogEventHelper.java:282)
              at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:111)
              at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:92)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
              at hudson.security.ACL.impersonate(ACL.java:260)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      grep "waiting to lock <0x0000000342d647c8>" jstack_dump.txt |wc -l
      139
      

      the lock 0x00007fb2ee566000 was not show up in jstack, maybe a C level lock
      Note: there is no dead lock, it is still functional but slow

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                johnou johnou
                Reporter:
                fengxx Ted
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: