Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48505

Lock contention in ExtensionList.getExtensionList

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Upgraded Jenkins from 2.66 to 2.89.1 and hit lock contention on ExtensionList.getExtensionList

      maybe caused by the change #24ede5

      
           @SuppressWarnings({"unchecked"})
           public <T> ExtensionList<T> getExtensionList(Class<T> extensionType) {
      -        return extensionLists.get(extensionType);
      +        return extensionLists.computeIfAbsent(extensionType, key -> ExtensionList.create(this, key));
           }
      
      

      Sample code:
      GlobalConfiguration.all().get(SplunkJenkinsInstallation.class);
      source: https://github.com/splunk/splunkforjenkins/blob/master/splunk-devops/src/main/java/com/splunk/splunkjenkins/SplunkJenkinsInstallation.java#L106

      stack trace

      "org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#793]" #190991 daemon prio=5 os_prio=0 tid=0x00007fb38f2f8800 nid=0xe54 waiting for monitor entry [0x00007fb3632dc000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
              - waiting to lock <0x0000000342d647c8> (a java.util.concurrent.ConcurrentHashMap$Node)
              at jenkins.model.Jenkins.getDescriptorList(Jenkins.java:2652)
              at jenkins.model.GlobalConfiguration.all(GlobalConfiguration.java:73)
              at com.splunk.splunkjenkins.SplunkJenkinsInstallation.get(SplunkJenkinsInstallation.java:105)
              at com.splunk.splunkjenkins.utils.LogFileCallable.send(LogFileCallable.java:90)
              at com.splunk.splunkjenkins.utils.LogFileCallable.sendFiles(LogFileCallable.java:57)
              at com.splunk.splunkjenkins.utils.LogEventHelper.sendFiles(LogEventHelper.java:282)
              at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:111)
              at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:92)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
              at hudson.security.ACL.impersonate(ACL.java:260)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      "org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#673]" #155842 daemon prio=5 os_prio=0 tid=0x00007fb38f3c6800 nid=0xe7d8 waiting for monitor entry [0x00007fb2ee566000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
              - locked <0x0000000342d647c8> (a java.util.concurrent.ConcurrentHashMap$Node)
              at jenkins.model.Jenkins.getDescriptorList(Jenkins.java:2652)
              at jenkins.model.GlobalConfiguration.all(GlobalConfiguration.java:73)
              at com.splunk.splunkjenkins.SplunkJenkinsInstallation.get(SplunkJenkinsInstallation.java:105)
              at com.splunk.splunkjenkins.utils.LogFileCallable.send(LogFileCallable.java:90)
              at com.splunk.splunkjenkins.utils.LogFileCallable.sendFiles(LogFileCallable.java:57)
              at com.splunk.splunkjenkins.utils.LogEventHelper.sendFiles(LogEventHelper.java:282)
              at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:111)
              at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:92)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
              at hudson.security.ACL.impersonate(ACL.java:260)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      grep "waiting to lock <0x0000000342d647c8>" jstack_dump.txt |wc -l
      139
      

      the lock 0x00007fb2ee566000 was not show up in jstack, maybe a C level lock
      Note: there is no dead lock, it is still functional but slow

        Attachments

          Issue Links

            Activity

            Hide
            fengxx Ted added a comment -

            maybe related to the JDK bug: JDK-8184907 : ConcurrentHashMap.computeIfAbsent acquires lock even when key present
            http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8184907
            It seems that it is only fixed in JDK9

            Show
            fengxx Ted added a comment - maybe related to the JDK bug: JDK-8184907 : ConcurrentHashMap.computeIfAbsent acquires lock even when key present http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8184907 It seems that it is only fixed in JDK9
            Hide
            oleg_nenashev Oleg Nenashev added a comment -
            Show
            oleg_nenashev Oleg Nenashev added a comment - CC Jesse Glick
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/src/main/java/jenkins/model/Jenkins.java
            http://jenkins-ci.org/commit/jenkins/3a529636cfc64cc6ea454d55ca88fd7a1f7981a5
            Log:
            Merge pull request #3198 from johnou/contention-JENKINS-48505

            JENKINS-48505 - Invoke optimistic get before computeIfAbsent to avoid contention.

            Compare: https://github.com/jenkinsci/jenkins/compare/09bcc5d6538b...3a529636cfc6

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/java/jenkins/model/Jenkins.java http://jenkins-ci.org/commit/jenkins/3a529636cfc64cc6ea454d55ca88fd7a1f7981a5 Log: Merge pull request #3198 from johnou/contention- JENKINS-48505 JENKINS-48505 - Invoke optimistic get before computeIfAbsent to avoid contention. Compare: https://github.com/jenkinsci/jenkins/compare/09bcc5d6538b...3a529636cfc6
            Hide
            danielbeck Daniel Beck added a comment -

            Merged towards 2.96.

            Show
            danielbeck Daniel Beck added a comment - Merged towards 2.96.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Ted Xiao
            Path:
            splunk-devops/src/main/java/com/splunk/splunkjenkins/SplunkJenkinsInstallation.java
            http://jenkins-ci.org/commit/splunk-devops-plugin/10c7ca8fd87a17f68c708e9b49e4b06501d96726
            Log:
            workaround JENKINS-48505 JDK-8184907

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Ted Xiao Path: splunk-devops/src/main/java/com/splunk/splunkjenkins/SplunkJenkinsInstallation.java http://jenkins-ci.org/commit/splunk-devops-plugin/10c7ca8fd87a17f68c708e9b49e4b06501d96726 Log: workaround JENKINS-48505 JDK-8184907

              People

              • Assignee:
                johnou johnou
                Reporter:
                fengxx Ted
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: