Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59706

Threads blocked in /blue/rest/organizations/jenkins/computers/ due to loop in PipelineRunImpl.getCommitUrl

    Details

    • Similar Issues:

      Description

      ci.jenkins.io froze, and Daniel Beck found that hundreds of threads were blocked in

      "Handling GET /blue/rest/organizations/jenkins/computers/ from ... : ..." #... prio=5 os_prio=0 cpu=9992.49ms elapsed=2517.82s tid=... nid=... waiting for monitor entry  [...]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:370)
      	- waiting to lock <...> (a hudson.model.RunMap)
      	at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:228)
      	at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber(WorkflowJob.java:232)
      	at io.jenkins.blueocean.rest.impl.pipeline.PipelineRunImpl.getCommitUrl(PipelineRunImpl.java:197)
      	at java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(java.base@11.0.4/DirectMethodHandle$Holder)
      	at java.lang.invoke.LambdaForm$MH/....invoke(java.base@11.0.4/LambdaForm$MH)
      	at java.lang.invoke.LambdaForm$MH/....invoke_MT(java.base@11.0.4/LambdaForm$MH)
      	at io.jenkins.blueocean.commons.stapler.export.MethodProperty.getValue(MethodProperty.java:72)
      	at io.jenkins.blueocean.commons.stapler.export.ExportInterceptor$1.getValue(ExportInterceptor.java:46)
      	at io.jenkins.blueocean.commons.stapler.Export$BlueOceanExportInterceptor.getValue(Export.java:196)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:136)
      	at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154)
      	at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeBuffered(Property.java:175)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:235)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154)
      	at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeBuffered(Property.java:175)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:219)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154)
      	at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
      	at io.jenkins.blueocean.commons.stapler.export.Model.writeTo(Model.java:199)
      	at io.jenkins.blueocean.commons.stapler.Export.writeOne(Export.java:177)
      	at io.jenkins.blueocean.commons.stapler.Export.serveExposedBean(Export.java:168)
      	at io.jenkins.blueocean.commons.stapler.Export.doJson(Export.java:108)
      	at io.jenkins.blueocean.commons.stapler.TreeResponse$Processor$1.generateResponse(TreeResponse.java:48)
      	at ...
      

      Looking at the code, this seems like a classic lazy-loading violation: looping over possibly every historical build of a job, each load of which involves expensive disk I/O. Under no circumstances may a plugin do this.

      This seems to have been noted in JENKINS-56773 but the response was just to disable the blueocean-executor-info plugin. How are users who already have this plugin installed supposed to know this? Some throttling was added, but evidently not enough.

        Attachments

          Issue Links

            Activity

            jglick Jesse Glick created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Link This issue relates to JENKINS-56773 [ JENKINS-56773 ]
            jglick Jesse Glick made changes -
            Link This issue is caused by JENKINS-45011 [ JENKINS-45011 ]
            jglick Jesse Glick made changes -
            Assignee Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "blueocean #2029 (Web Link)" [ 23814 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            markewaite Mark Waite added a comment -

            As far as I can tell, https://github.com/jenkinsci/blueocean-plugin/releases/tag/blueocean-parent-1.21.0 includes this change. Thanks Jesse Glick for that pull request! I think this can be marked 'Closed' in release blue ocean 1.21.0

            Show
            markewaite Mark Waite added a comment - As far as I can tell, https://github.com/jenkinsci/blueocean-plugin/releases/tag/blueocean-parent-1.21.0 includes this change. Thanks Jesse Glick for that pull request! I think this can be marked 'Closed' in release blue ocean 1.21.0

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                jglick Jesse Glick
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: