Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46460

Performance issue in AsyncResourceDisposer.persist

    Details

    • Similar Issues:

      Description

      A thread dump of a Jenkins system suffering very poor performance showed many threads blocked in AsyncResourceDisposer.persist, here:

      "Computer.threadPoolForRemoting [#...]" ... waiting for monitor entry [...]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at com.thoughtworks.xstream.core.DefaultConverterLookup.lookupConverterForType(DefaultConverterLookup.java:49)
      	- waiting to lock <...> (a com.thoughtworks.xstream.core.DefaultConverterLookup)
      	at com.thoughtworks.xstream.XStream$1.lookupConverterForType(XStream.java:498)
      	at ...
      	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
      	at hudson.XmlFile.write(XmlFile.java:170)
      	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.persist(AsyncResourceDisposer.java:175)
      	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.access$300(AsyncResourceDisposer.java:65)
      	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer$WorkItem.run(AsyncResourceDisposer.java:262)
      	at ...
      

      This is written poorly and causing massive contention on a single lock in XStream; it should be coalescing calls to persist, and preferably even delaying them by a few seconds' grace period to try to batch up as much as possible.

      The workaround is to remove this plugin. It is typically used by ws-cleanup; better to, say, use the CleanBeforeCheckout option in Git.

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            Problem likely aggravated by locks from JENKINS-19561.

            Show
            jglick Jesse Glick added a comment - Problem likely aggravated by locks from  JENKINS-19561 .
            Hide
            jglick Jesse Glick added a comment -

            After some digging, found another case of a system with >4k threads blocked in this way.

            Show
            jglick Jesse Glick added a comment - After some digging, found another case of a system with >4k threads blocked in this way.
            Hide
            olivergondza Oliver Gondža added a comment -

            Another approach would be to avoid persisting when the lastState have not changed after dispose was called.

            Show
            olivergondza Oliver Gondža added a comment - Another approach would be to avoid persisting when the lastState have not changed after dispose was called.
            Hide
            olivergondza Oliver Gondža added a comment -

            After second though, I do not think the the benefit of having the lastState persisted reliably is worth the price. It was excluded from the persistence altogether (comment indicated I wanted to do that a long time ago) and every run of WorkItem does not initiate save any longer.

            Show
            olivergondza Oliver Gondža added a comment - After second though, I do not think the the benefit of having the lastState persisted reliably is worth the price. It was excluded from the persistence altogether (comment indicated I wanted to do that a long time ago) and every run of WorkItem does not initiate save any longer.
            Hide
            olivergondza Oliver Gondža added a comment -

            Released on 0.8.

            Show
            olivergondza Oliver Gondža added a comment - Released on 0.8.

              People

              • Assignee:
                olivergondza Oliver Gondža
                Reporter:
                jglick Jesse Glick
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: