Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-13154

Heavy thread congestion with FingerPrint.save

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Environment:
    • Similar Issues:

      Description

      We are seeing repeatable very heavy lock congestion with FingerPrint.save.

      Jenkins apparently slides down to a state where a lot of threads have locked / are competing for a lock on FingerPrint instance and are competing for a lock on AnnotationMapper (a singleton). Everything grings to a halt.

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            @edenman: separate issue please. (Can use JIRA’s “related” link if appropriate.) Your thread dump anyway does not show Jenkins being blocked, but rather excessively busy doing a couple different things: loading fingerprints; and loading historical build records (perhaps after having discarded them under memory pressure).

            Show
            jglick Jesse Glick added a comment - @edenman: separate issue please. (Can use JIRA’s “related” link if appropriate.) Your thread dump anyway does not show Jenkins being blocked, but rather excessively busy doing a couple different things: loading fingerprints; and loading historical build records (perhaps after having discarded them under memory pressure).
            Hide
            edenman Eric Denman added a comment -

            @jglick thanks! Filed JENKINS-17412

            Show
            edenman Eric Denman added a comment - @jglick thanks! Filed JENKINS-17412
            Hide
            mp3 Mikko Peltonen added a comment -

            We are still seeing lots of congestion with Jenkins 1.517 and our builds may still stay tens of minutes in "Waiting for Jenkins to finish collecting data" -phase.

            The place has changed though, now thread dumps show lots of these:

            java.lang.Thread.State: BLOCKED (on object monitor)
            at java.util.Collections$SynchronizedMap.get(Collections.java:2031)

            • waiting to lock <0x000000070c6dcfa8> (a java.util.Collections$SynchronizedMap)
              at com.thoughtworks.xstream.core.DefaultConverterLookup.lookupConverterForType(DefaultConverterLookup.java:49)
            Show
            mp3 Mikko Peltonen added a comment - We are still seeing lots of congestion with Jenkins 1.517 and our builds may still stay tens of minutes in "Waiting for Jenkins to finish collecting data" -phase. The place has changed though, now thread dumps show lots of these: java.lang.Thread.State: BLOCKED (on object monitor) at java.util.Collections$SynchronizedMap.get(Collections.java:2031) waiting to lock <0x000000070c6dcfa8> (a java.util.Collections$SynchronizedMap) at com.thoughtworks.xstream.core.DefaultConverterLookup.lookupConverterForType(DefaultConverterLookup.java:49)
            Hide
            mp3 Mikko Peltonen added a comment -

            Attached relevant parts of the whole thread dump.

            Show
            mp3 Mikko Peltonen added a comment - Attached relevant parts of the whole thread dump.
            Hide
            jglick Jesse Glick added a comment -

            @mp3 whatever you are seeing is a distinct bug. Please file separately and use JIRA’s “is related to” link as needed.

            (I would have suspected a regression from JENKINS-18775, but that is much newer than 1.517.)

            Show
            jglick Jesse Glick added a comment - @mp3 whatever you are seeing is a distinct bug. Please file separately and use JIRA’s “is related to” link as needed. (I would have suspected a regression from JENKINS-18775 , but that is much newer than 1.517.)

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                t_kurki Teppo Kurki
              • Votes:
                4 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: