Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19712

[Fingerprint] Excessive Heap consumption during "Waiting for Jenkins to finish collecting data"

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Fingerprinting has always been a bit pain in the ... that's why i thought we could improve performance of our maven build jobs by upgrading from 1.480.3 to 1.509.3 which promised some fixes regarding Fingerprinting.

      Before the upgrade (using 1.480.3) we experienced Fingerprinting to take time between 5min and 15min on average for builds of the size of between 5 to 30 modules.

      Actually it got way worse after upgrade to 1.509.3:
      i observed huge amounts of data being loaded into heap which hit into old gen space pretty much instantly because of the huge amount being loaded into heap. so we ended up with running full GCs as long as there were builds in the state of "Waiting for Jenkins to finish collecting data".

      we then downgraded to 1.509.2 which worked out pretty well so far.

      we observed these problems with:

      • jenkins 1.509.3
      • tomcat 6
      • jdk 6
      • jvm-options: -XX:MaxPermSize=512M -Xmx6G -Xms6G -XX:NewRatio=4 -XX:SurvivorRatio=6 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
      • centos 5.5 final

      we also observed many blocked threads during Fingerprinting-phases. unfortunately i didnt keep the whole thread dump log output, but the were blocked on hudson.model.Fingerprint.add(Fingerprint.java:717)

      I attached some screenshots of our jvm-monitoring where you can clearly see the difference between 1.509.3 (many Full GC runs) and 1.509.2 (zero Full GC runs)
      i also tuned the NewRatio jvm-option a bit when i downgraded to 1.509.2

      I have to say that this issue is kind of a big hit to us, we won't be able to upgrade to any jenkins version containing these Fingerprint code changes within 1.509.3 as long as fingerprinting is still a mandatory step!

        Attachments

          Issue Links

            Activity

            Hide
            seder Stefan Prietl added a comment -

            Upgrading to 1.509.4 seems to fix this issue also.

            Here is a comparison of 1.509.4 (12:00 - 14:00) and 1.509.3 (14:00-16:00):

            This was running on a server with 4 CPUs so a load of 40 is somewhat high.
            Btw. the data contained in the fingerprint folder in this test was about 1.4GB. The data was collected by our production Jenkins running on 1.509.3 during builds. So there was (or still is?) probably a bug with the cleanup task.

            However: IMHO using LTS 1.509.3 is not recommended.

            Show
            seder Stefan Prietl added a comment - Upgrading to 1.509.4 seems to fix this issue also. Here is a comparison of 1.509.4 (12:00 - 14:00) and 1.509.3 (14:00-16:00): This was running on a server with 4 CPUs so a load of 40 is somewhat high. Btw. the data contained in the fingerprint folder in this test was about 1.4GB. The data was collected by our production Jenkins running on 1.509.3 during builds. So there was (or still is?) probably a bug with the cleanup task. However: IMHO using LTS 1.509.3 is not recommended.
            Hide
            jglick Jesse Glick added a comment -

            JENKINS-18417 may have improved the situation in 1.509.4; if this is no longer as much of an issue, it could probably be closed as a duplicate. JENKINS-11333 may offer a workaround.

            Show
            jglick Jesse Glick added a comment - JENKINS-18417 may have improved the situation in 1.509.4; if this is no longer as much of an issue, it could probably be closed as a duplicate. JENKINS-11333 may offer a workaround.
            Hide
            danielbeck Daniel Beck added a comment -

            What's the situation on recent Jenkins versions? Is this issue still as severe? Jesse indicated two issues that seem to be able to improvement the situation in many environments.

            Show
            danielbeck Daniel Beck added a comment - What's the situation on recent Jenkins versions? Is this issue still as severe? Jesse indicated two issues that seem to be able to improvement the situation in many environments.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Both improvements are implemented, no other responses. I suppose we can close this issue

            Show
            oleg_nenashev Oleg Nenashev added a comment - Both improvements are implemented, no other responses. I suppose we can close this issue
            Hide
            dkoryavov Denis Koryavov added a comment -

            It looks like we faced up with the same situation after upgrade from 2.89 to 2.107.3 (please, see here https://issues.jenkins-ci.org/browse/JENKINS-52150). So, I believe that the bug is still not fixed.

            Show
            dkoryavov Denis Koryavov added a comment - It looks like we faced up with the same situation after upgrade from 2.89 to 2.107.3 (please, see here https://issues.jenkins-ci.org/browse/JENKINS-52150 ). So, I believe that the bug is still not fixed.
            Show
            capf Carsten Pfeiffer added a comment - Found the cause and a workaround, see https://issues.jenkins-ci.org/browse/JENKINS-52150?focusedCommentId=345858&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-345858

              People

              • Assignee:
                Unassigned
                Reporter:
                mcklaus Klaus Azesberger
              • Votes:
                13 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: