Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48505

Lock contention in ExtensionList.getExtensionList

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Upgraded Jenkins from 2.66 to 2.89.1 and hit lock contention on ExtensionList.getExtensionList

      maybe caused by the change #24ede5

      
           @SuppressWarnings({"unchecked"})
           public <T> ExtensionList<T> getExtensionList(Class<T> extensionType) {
      -        return extensionLists.get(extensionType);
      +        return extensionLists.computeIfAbsent(extensionType, key -> ExtensionList.create(this, key));
           }
      
      

      Sample code:
      GlobalConfiguration.all().get(SplunkJenkinsInstallation.class);
      source: https://github.com/splunk/splunkforjenkins/blob/master/splunk-devops/src/main/java/com/splunk/splunkjenkins/SplunkJenkinsInstallation.java#L106

      stack trace

      "org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#793]" #190991 daemon prio=5 os_prio=0 tid=0x00007fb38f2f8800 nid=0xe54 waiting for monitor entry [0x00007fb3632dc000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
              - waiting to lock <0x0000000342d647c8> (a java.util.concurrent.ConcurrentHashMap$Node)
              at jenkins.model.Jenkins.getDescriptorList(Jenkins.java:2652)
              at jenkins.model.GlobalConfiguration.all(GlobalConfiguration.java:73)
              at com.splunk.splunkjenkins.SplunkJenkinsInstallation.get(SplunkJenkinsInstallation.java:105)
              at com.splunk.splunkjenkins.utils.LogFileCallable.send(LogFileCallable.java:90)
              at com.splunk.splunkjenkins.utils.LogFileCallable.sendFiles(LogFileCallable.java:57)
              at com.splunk.splunkjenkins.utils.LogEventHelper.sendFiles(LogEventHelper.java:282)
              at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:111)
              at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:92)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
              at hudson.security.ACL.impersonate(ACL.java:260)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      "org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#673]" #155842 daemon prio=5 os_prio=0 tid=0x00007fb38f3c6800 nid=0xe7d8 waiting for monitor entry [0x00007fb2ee566000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
              - locked <0x0000000342d647c8> (a java.util.concurrent.ConcurrentHashMap$Node)
              at jenkins.model.Jenkins.getDescriptorList(Jenkins.java:2652)
              at jenkins.model.GlobalConfiguration.all(GlobalConfiguration.java:73)
              at com.splunk.splunkjenkins.SplunkJenkinsInstallation.get(SplunkJenkinsInstallation.java:105)
              at com.splunk.splunkjenkins.utils.LogFileCallable.send(LogFileCallable.java:90)
              at com.splunk.splunkjenkins.utils.LogFileCallable.sendFiles(LogFileCallable.java:57)
              at com.splunk.splunkjenkins.utils.LogEventHelper.sendFiles(LogEventHelper.java:282)
              at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:111)
              at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:92)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
              at hudson.security.ACL.impersonate(ACL.java:260)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      grep "waiting to lock <0x0000000342d647c8>" jstack_dump.txt |wc -l
      139
      

      the lock 0x00007fb2ee566000 was not show up in jstack, maybe a C level lock
      Note: there is no dead lock, it is still functional but slow

        Attachments

          Issue Links

            Activity

            fengxx Ted created issue -
            fengxx Ted made changes -
            Field Original Value New Value
            Description Upgraded Jenkins from 2.66 to 2.89.1 and hit lock contention on ExtensionList.getExtensionList

            maybe caused by the change #24ede5
            {code}

                 @SuppressWarnings({"unchecked"})
                 public <T> ExtensionList<T> getExtensionList(Class<T> extensionType) {
            - return extensionLists.get(extensionType);
            + return extensionLists.computeIfAbsent(extensionType, key -> ExtensionList.create(this, key));
                 }

            {code}

            Sample code:
            GlobalConfiguration.all().get(SplunkJenkinsInstallation.class);

            stack trace
            {code}
            "org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#793]" #190991 daemon prio=5 os_prio=0 tid=0x00007fb38f2f8800 nid=0xe54 waiting for monitor entry [0x00007fb3632dc000]
               java.lang.Thread.State: BLOCKED (on object monitor)
                    at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
                    - waiting to lock <0x0000000342d647c8> (a java.util.concurrent.ConcurrentHashMap$Node)
                    at jenkins.model.Jenkins.getDescriptorList(Jenkins.java:2652)
                    at jenkins.model.GlobalConfiguration.all(GlobalConfiguration.java:73)
                    at com.splunk.splunkjenkins.SplunkJenkinsInstallation.get(SplunkJenkinsInstallation.java:105)
                    at com.splunk.splunkjenkins.utils.LogFileCallable.send(LogFileCallable.java:90)
                    at com.splunk.splunkjenkins.utils.LogFileCallable.sendFiles(LogFileCallable.java:57)
                    at com.splunk.splunkjenkins.utils.LogEventHelper.sendFiles(LogEventHelper.java:282)
                    at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:111)
                    at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:92)
                    at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
                    at hudson.security.ACL.impersonate(ACL.java:260)
                    at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:745)

            "org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#673]" #155842 daemon prio=5 os_prio=0 tid=0x00007fb38f3c6800 nid=0xe7d8 waiting for monitor entry [0x00007fb2ee566000]
               java.lang.Thread.State: BLOCKED (on object monitor)
                    at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
                    - locked <0x0000000342d647c8> (a java.util.concurrent.ConcurrentHashMap$Node)
                    at jenkins.model.Jenkins.getDescriptorList(Jenkins.java:2652)
                    at jenkins.model.GlobalConfiguration.all(GlobalConfiguration.java:73)
                    at com.splunk.splunkjenkins.SplunkJenkinsInstallation.get(SplunkJenkinsInstallation.java:105)
                    at com.splunk.splunkjenkins.utils.LogFileCallable.send(LogFileCallable.java:90)
                    at com.splunk.splunkjenkins.utils.LogFileCallable.sendFiles(LogFileCallable.java:57)
                    at com.splunk.splunkjenkins.utils.LogEventHelper.sendFiles(LogEventHelper.java:282)
                    at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:111)
                    at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:92)
                    at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
                    at hudson.security.ACL.impersonate(ACL.java:260)
                    at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:745)

            grep "waiting to lock <0x0000000342d647c8>" jstack_dump.txt |wc -l
            139
            {code}

            the lock 0x00007fb2ee566000 was not show up in jstack, maybe a C level lock
            Note: there is no dead lock, it is still functional but slow

            fengxx Ted made changes -
            Description Upgraded Jenkins from 2.66 to 2.89.1 and hit lock contention on ExtensionList.getExtensionList

            maybe caused by the change #24ede5
            {code}

                 @SuppressWarnings({"unchecked"})
                 public <T> ExtensionList<T> getExtensionList(Class<T> extensionType) {
            - return extensionLists.get(extensionType);
            + return extensionLists.computeIfAbsent(extensionType, key -> ExtensionList.create(this, key));
                 }

            {code}

            Sample code:
            GlobalConfiguration.all().get(SplunkJenkinsInstallation.class);

            stack trace
            {code}
            "org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#793]" #190991 daemon prio=5 os_prio=0 tid=0x00007fb38f2f8800 nid=0xe54 waiting for monitor entry [0x00007fb3632dc000]
               java.lang.Thread.State: BLOCKED (on object monitor)
                    at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
                    - waiting to lock <0x0000000342d647c8> (a java.util.concurrent.ConcurrentHashMap$Node)
                    at jenkins.model.Jenkins.getDescriptorList(Jenkins.java:2652)
                    at jenkins.model.GlobalConfiguration.all(GlobalConfiguration.java:73)
                    at com.splunk.splunkjenkins.SplunkJenkinsInstallation.get(SplunkJenkinsInstallation.java:105)
                    at com.splunk.splunkjenkins.utils.LogFileCallable.send(LogFileCallable.java:90)
                    at com.splunk.splunkjenkins.utils.LogFileCallable.sendFiles(LogFileCallable.java:57)
                    at com.splunk.splunkjenkins.utils.LogEventHelper.sendFiles(LogEventHelper.java:282)
                    at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:111)
                    at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:92)
                    at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
                    at hudson.security.ACL.impersonate(ACL.java:260)
                    at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:745)

            "org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#673]" #155842 daemon prio=5 os_prio=0 tid=0x00007fb38f3c6800 nid=0xe7d8 waiting for monitor entry [0x00007fb2ee566000]
               java.lang.Thread.State: BLOCKED (on object monitor)
                    at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
                    - locked <0x0000000342d647c8> (a java.util.concurrent.ConcurrentHashMap$Node)
                    at jenkins.model.Jenkins.getDescriptorList(Jenkins.java:2652)
                    at jenkins.model.GlobalConfiguration.all(GlobalConfiguration.java:73)
                    at com.splunk.splunkjenkins.SplunkJenkinsInstallation.get(SplunkJenkinsInstallation.java:105)
                    at com.splunk.splunkjenkins.utils.LogFileCallable.send(LogFileCallable.java:90)
                    at com.splunk.splunkjenkins.utils.LogFileCallable.sendFiles(LogFileCallable.java:57)
                    at com.splunk.splunkjenkins.utils.LogEventHelper.sendFiles(LogEventHelper.java:282)
                    at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:111)
                    at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:92)
                    at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
                    at hudson.security.ACL.impersonate(ACL.java:260)
                    at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:745)

            grep "waiting to lock <0x0000000342d647c8>" jstack_dump.txt |wc -l
            139
            {code}

            the lock 0x00007fb2ee566000 was not show up in jstack, maybe a C level lock
            Note: there is no dead lock, it is still functional but slow

            Upgraded Jenkins from 2.66 to 2.89.1 and hit lock contention on ExtensionList.getExtensionList

            maybe caused by the change #24ede5
            {code}

                 @SuppressWarnings({"unchecked"})
                 public <T> ExtensionList<T> getExtensionList(Class<T> extensionType) {
            - return extensionLists.get(extensionType);
            + return extensionLists.computeIfAbsent(extensionType, key -> ExtensionList.create(this, key));
                 }

            {code}

            Sample code:
            GlobalConfiguration.all().get(SplunkJenkinsInstallation.class);
            source: https://github.com/splunk/splunkforjenkins/blob/master/splunk-devops/src/main/java/com/splunk/splunkjenkins/SplunkJenkinsInstallation.java#L106

            stack trace
            {code}
            "org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#793]" #190991 daemon prio=5 os_prio=0 tid=0x00007fb38f2f8800 nid=0xe54 waiting for monitor entry [0x00007fb3632dc000]
               java.lang.Thread.State: BLOCKED (on object monitor)
                    at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
                    - waiting to lock <0x0000000342d647c8> (a java.util.concurrent.ConcurrentHashMap$Node)
                    at jenkins.model.Jenkins.getDescriptorList(Jenkins.java:2652)
                    at jenkins.model.GlobalConfiguration.all(GlobalConfiguration.java:73)
                    at com.splunk.splunkjenkins.SplunkJenkinsInstallation.get(SplunkJenkinsInstallation.java:105)
                    at com.splunk.splunkjenkins.utils.LogFileCallable.send(LogFileCallable.java:90)
                    at com.splunk.splunkjenkins.utils.LogFileCallable.sendFiles(LogFileCallable.java:57)
                    at com.splunk.splunkjenkins.utils.LogEventHelper.sendFiles(LogEventHelper.java:282)
                    at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:111)
                    at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:92)
                    at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
                    at hudson.security.ACL.impersonate(ACL.java:260)
                    at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:745)

            "org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#673]" #155842 daemon prio=5 os_prio=0 tid=0x00007fb38f3c6800 nid=0xe7d8 waiting for monitor entry [0x00007fb2ee566000]
               java.lang.Thread.State: BLOCKED (on object monitor)
                    at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
                    - locked <0x0000000342d647c8> (a java.util.concurrent.ConcurrentHashMap$Node)
                    at jenkins.model.Jenkins.getDescriptorList(Jenkins.java:2652)
                    at jenkins.model.GlobalConfiguration.all(GlobalConfiguration.java:73)
                    at com.splunk.splunkjenkins.SplunkJenkinsInstallation.get(SplunkJenkinsInstallation.java:105)
                    at com.splunk.splunkjenkins.utils.LogFileCallable.send(LogFileCallable.java:90)
                    at com.splunk.splunkjenkins.utils.LogFileCallable.sendFiles(LogFileCallable.java:57)
                    at com.splunk.splunkjenkins.utils.LogEventHelper.sendFiles(LogEventHelper.java:282)
                    at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:111)
                    at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:92)
                    at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
                    at hudson.security.ACL.impersonate(ACL.java:260)
                    at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:745)

            grep "waiting to lock <0x0000000342d647c8>" jstack_dump.txt |wc -l
            139
            {code}

            the lock 0x00007fb2ee566000 was not show up in jstack, maybe a C level lock
            Note: there is no dead lock, it is still functional but slow

            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-48227 [ JENKINS-48227 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-48227 [ JENKINS-48227 ]
            jglick Jesse Glick made changes -
            Labels performance
            jglick Jesse Glick made changes -
            Remote Link This issue links to "JDK-8161372 (Web Link)" [ 19403 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 3198 (Web Link)" [ 19404 ]
            jglick Jesse Glick made changes -
            Labels performance lts-candidate performance regression
            jglick Jesse Glick made changes -
            Assignee johnou [ johnou ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            danielbeck Daniel Beck made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            olivergondza Oliver Gond┼ża made changes -
            Labels lts-candidate performance regression 2.89.3-fixed performance regression

              People

              • Assignee:
                johnou johnou
                Reporter:
                fengxx Ted
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: