Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core
    • Environment:
    • Similar Issues:

      Description

      Jenkins starts using 100% CPU after a few days. Using jstack I see several threads trying to write compressed output, and apparently not changing over time:

      • java.util.zip.Deflater.deflate(byte[], int, int) @bci=55, line=322 (Compiled frame; information may be imprecise)
      • java.util.zip.DeflaterOutputStream.deflate() @bci=14, line=176 (Compiled frame)
      • java.util.zip.DeflaterOutputStream.write(byte[], int, int) @bci=108, line=135 (Compiled frame)
      • java.util.zip.GZIPOutputStream.write(byte[], int, int) @bci=4, line=89 (Compiled frame)
      • org.kohsuke.stapler.compression.FilterServletOutputStream.write(byte[], int, int) @bci=7, line=31 (Compiled frame)
      • sun.nio.cs.StreamEncoder.writeBytes() @bci=120, line=220 (Interpreted frame)
      • sun.nio.cs.StreamEncoder.implClose() @bci=84, line=315 (Interpreted frame)
      • sun.nio.cs.StreamEncoder.close() @bci=18, line=148 (Interpreted frame)
      • java.io.OutputStreamWriter.close() @bci=4, line=233 (Interpreted frame)
      • java.io.PrintWriter.close() @bci=21, line=312 (Interpreted frame)
      • org.kohsuke.stapler.compression.CompressionFilter.reportException(java.lang.Exception, javax.servlet.http.HttpServletResponse) @bci=112, line=77 (Interpreted frame)
      • org.kohsuke.stapler.compression.CompressionFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=57, line=53 (Compiled frame)
      • winstone.FilterConfiguration.execute(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=25, line=194 (Compiled frame)
      • winstone.RequestDispatcher.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=48, line=366 (Compiled frame)
      • hudson.util.CharacterEncodingFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=43, line=81 (Compiled frame)
      • winstone.FilterConfiguration.execute(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=25, line=194 (Compiled frame)
      • winstone.RequestDispatcher.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=48, line=366 (Compiled frame)
      • winstone.RequestDispatcher.forward(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=483, line=331 (Compiled frame)
      • winstone.RequestHandlerThread.processRequest(winstone.WebAppConfiguration, winstone.WinstoneRequest, winstone.WinstoneResponse, java.lang.String) @bci=38, line=215 (Compiled frame)
      • winstone.RequestHandlerThread.run() @bci=631, line=138 (Compiled frame)
      • java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=471 (Interpreted frame)
      • java.util.concurrent.FutureTask$Sync.innerRun() @bci=29, line=334 (Interpreted frame)
      • java.util.concurrent.FutureTask.run() @bci=4, line=166 (Interpreted frame)
      • winstone.BoundedExecutorService$1.run() @bci=4, line=77 (Compiled frame)
      • java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Compiled frame)
      • java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame)
      • java.lang.Thread.run() @bci=11, line=679 (Interpreted frame)

      I'm suspecting but not 100% sure that these threads are in an infinite loop (livelocked). I'm struggling to see what other threads might be doing this.

      This JVM was not started with debugging enabled to attach a debugger for analysis. I've enabled it now. Stack traces attached as files below.

        Attachments

        1. jenkins.stacktrace.1
          53 kB
        2. jenkins.stacktrace.2
          51 kB
        3. jenkins.stacktrace.3
          46 kB
        4. thread-dump-prod.txt
          134 kB
        5. thread-dump-reproduce-1.txt
          72 kB
        6. thread-dump-reproduce-2.txt
          72 kB
        7. thread-dump-reproduce-3.txt
          72 kB

          Issue Links

            Activity

            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            http://jenkins-ci.org/commit/maven-plugin/fba17cd8497d9ddac9c55fead9c9a8da61059b23
            Log:
            JENKINS-14362 Switch to jzlib for GZIP streaming.

            Originally-Committed-As: 3701a327a75d35e5d0e5795341cc55bb123a2007

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml http://jenkins-ci.org/commit/maven-plugin/fba17cd8497d9ddac9c55fead9c9a8da61059b23 Log: JENKINS-14362 Switch to jzlib for GZIP streaming. Originally-Committed-As: 3701a327a75d35e5d0e5795341cc55bb123a2007
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/main/java/hudson/maven/AbstractMavenProcessFactory.java
            src/main/java/hudson/maven/AggregatingClassLoader.java
            src/main/java/hudson/maven/Maven3Builder.java
            src/main/java/hudson/maven/Maven3ProcessFactory.java
            src/main/java/hudson/maven/MavenBuildInformation.java
            src/main/java/hudson/maven/MavenEmbedderRequest.java
            src/main/java/hudson/maven/MavenModuleSet.java
            src/main/java/hudson/maven/MavenModuleSetBuild.java
            src/main/java/hudson/maven/MavenProbeAction.java
            src/main/java/hudson/maven/MavenProcessFactory.java
            src/main/java/hudson/maven/MavenUtil.java
            src/main/java/hudson/maven/PlexusModuleContributor.java
            src/main/java/hudson/maven/reporters/MavenAbstractArtifactRecord.java
            src/main/java/hudson/maven/reporters/MavenFingerprinter.java
            src/main/java/hudson/maven/reporters/TestMojo.java
            src/main/resources/hudson/maven/MavenBuild/executedMojos_fr.properties
            src/main/resources/hudson/maven/MavenModuleSetBuild/main_pl.properties
            src/main/resources/hudson/maven/Messages.properties
            src/main/resources/hudson/maven/Messages_es.properties
            src/main/resources/hudson/maven/Messages_zh_TW.properties
            src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_fr.properties
            src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
            src/test/java/hudson/maven/reporters/TestMojoTest.java
            http://jenkins-ci.org/commit/maven-plugin/14863a0498e0e6c389f2779b9150e9280b363d3e
            Log:
            Merge branch 'master' into JENKINS-14362-jzlib

            Originally-Committed-As: dedb24ef034a04fea288eaa50089e8e774714e99

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/hudson/maven/AbstractMavenProcessFactory.java src/main/java/hudson/maven/AggregatingClassLoader.java src/main/java/hudson/maven/Maven3Builder.java src/main/java/hudson/maven/Maven3ProcessFactory.java src/main/java/hudson/maven/MavenBuildInformation.java src/main/java/hudson/maven/MavenEmbedderRequest.java src/main/java/hudson/maven/MavenModuleSet.java src/main/java/hudson/maven/MavenModuleSetBuild.java src/main/java/hudson/maven/MavenProbeAction.java src/main/java/hudson/maven/MavenProcessFactory.java src/main/java/hudson/maven/MavenUtil.java src/main/java/hudson/maven/PlexusModuleContributor.java src/main/java/hudson/maven/reporters/MavenAbstractArtifactRecord.java src/main/java/hudson/maven/reporters/MavenFingerprinter.java src/main/java/hudson/maven/reporters/TestMojo.java src/main/resources/hudson/maven/MavenBuild/executedMojos_fr.properties src/main/resources/hudson/maven/MavenModuleSetBuild/main_pl.properties src/main/resources/hudson/maven/Messages.properties src/main/resources/hudson/maven/Messages_es.properties src/main/resources/hudson/maven/Messages_zh_TW.properties src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_fr.properties src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java src/test/java/hudson/maven/reporters/TestMojoTest.java http://jenkins-ci.org/commit/maven-plugin/14863a0498e0e6c389f2779b9150e9280b363d3e Log: Merge branch 'master' into JENKINS-14362 -jzlib Originally-Committed-As: dedb24ef034a04fea288eaa50089e8e774714e99
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/resources/hudson/maven/Messages.properties
            src/main/resources/hudson/maven/Messages_de.properties
            src/main/resources/hudson/maven/RedeployPublisher/config_de.properties
            src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_de.properties
            http://jenkins-ci.org/commit/maven-plugin/d75a8c218b6ffd24fe2ddea8953ffaf22da63bc6
            Log:
            Merge branch 'master' into JENKINS-14362-jzlib

            Conflicts:
            core/pom.xml

            Originally-Committed-As: b7e2d6127f77711681113a020681508d57622ea5

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/resources/hudson/maven/Messages.properties src/main/resources/hudson/maven/Messages_de.properties src/main/resources/hudson/maven/RedeployPublisher/config_de.properties src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_de.properties http://jenkins-ci.org/commit/maven-plugin/d75a8c218b6ffd24fe2ddea8953ffaf22da63bc6 Log: Merge branch 'master' into JENKINS-14362 -jzlib Conflicts: core/pom.xml Originally-Committed-As: b7e2d6127f77711681113a020681508d57622ea5
            Hide
            bbonn bbonn added a comment -

            Hi There,

            Just verifying. I see this bug has a 1.509.4 fixed label, but on Jenkins-CI.org, it is not listed in the change log until 1.520. Can someone please validate where the fix is? Thanks a lot.

            Show
            bbonn bbonn added a comment - Hi There, Just verifying. I see this bug has a 1.509.4 fixed label, but on Jenkins-CI.org, it is not listed in the change log until 1.520. Can someone please validate where the fix is? Thanks a lot.
            Hide
            jglick Jesse Glick added a comment -

            It was first fixed in trunk in 1.520, then backported to 1.509.4. http://jenkins-ci.org/changelog-stable neglects to mention it, but that changelog is generally unreliable anyway.

            Show
            jglick Jesse Glick added a comment - It was first fixed in trunk in 1.520, then backported to 1.509.4. http://jenkins-ci.org/changelog-stable neglects to mention it, but that changelog is generally unreliable anyway.

              People

              • Assignee:
                lmcazra Audrey Azra
                Reporter:
                gcc Chris Wilson
              • Votes:
                44 Vote for this issue
                Watchers:
                62 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: