Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9349

Viewing large console logs with timestamper plugin cause Jenkins to crash

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: timestamper-plugin
    • Labels:
      None
    • Environment:
      Running on Solaris 10 (sparc). Tomcat 6.0.29. Jenkins 401 and 404. This was not an issue in Hudson 380. JVM is set to 1.5GB on one system and 2GB on another.
    • Similar Issues:

      Description

      Run a build that generates a large console output. Say around 10MB or more. View the console log for the build. It should work just fine. Add the timestamper and run the build again (version 0.6). View the console log. Select to view full log. This frequently crashes Jenkins/Tomcat. Sometimes I actually get the whole log. This has occurred after a restart as well as after it has been running for a bit. I don't see the issue when view logs without the timestamper enabled (at least not yet).

        Attachments

          Activity

          Hide
          kohsuke Kohsuke Kawaguchi added a comment -

          I have a hunch. timestamper uses an exntesion point called console annotation which stores a small compressed gzip stream of data into the log file. It uses a lot of those, because each timestamp it's adding, I believe, is a new annotation.

          If there's a race condition in the write operations, it could corrupt the gzip encoded stream, and I wonder if it can happen just in the right away that breaks the decompressor like this.

          If my hypothesis is correct, trying to look at the problematic build log will consistently reproduce the problem (as opposed to "sometimes I can see build #15 but some times it crashes when looking at the same build #15.)

          If this fits the issue you are seeing, please attach your console output (or send it personally to me if it's sensitive), and that'd greatly simplify our efforts to fix the issue.

          Show
          kohsuke Kohsuke Kawaguchi added a comment - I have a hunch. timestamper uses an exntesion point called console annotation which stores a small compressed gzip stream of data into the log file. It uses a lot of those, because each timestamp it's adding, I believe, is a new annotation. If there's a race condition in the write operations, it could corrupt the gzip encoded stream, and I wonder if it can happen just in the right away that breaks the decompressor like this. If my hypothesis is correct, trying to look at the problematic build log will consistently reproduce the problem (as opposed to "sometimes I can see build #15 but some times it crashes when looking at the same build #15.) If this fits the issue you are seeing, please attach your console output (or send it personally to me if it's sensitive), and that'd greatly simplify our efforts to fix the issue.
          Hide
          kohsuke Kohsuke Kawaguchi added a comment -

          I take it back. Neither 32756 nor 59208 is a big memory chunk.

          Show
          kohsuke Kohsuke Kawaguchi added a comment - I take it back. Neither 32756 nor 59208 is a big memory chunk.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          changelog.html
          core/src/main/java/hudson/console/AnnotatedLargeText.java
          core/src/main/java/hudson/console/ConsoleNote.java
          http://jenkins-ci.org/commit/jenkins/313fb5940f63ca7e13281f4bba1cdbcfcfd8f2c3
          Log:
          [FIXED JENKINS-9349] let GZipInputStream release its native memory
          eagerly.

          Compare: https://github.com/jenkinsci/jenkins/compare/e7cc141...313fb59

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/src/main/java/hudson/console/AnnotatedLargeText.java core/src/main/java/hudson/console/ConsoleNote.java http://jenkins-ci.org/commit/jenkins/313fb5940f63ca7e13281f4bba1cdbcfcfd8f2c3 Log: [FIXED JENKINS-9349] let GZipInputStream release its native memory eagerly. Compare: https://github.com/jenkinsci/jenkins/compare/e7cc141...313fb59
          Hide
          dogfood dogfood added a comment -
          Show
          dogfood dogfood added a comment - Integrated in jenkins_main_trunk #1267
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          changelog.html
          core/src/main/java/hudson/console/AnnotatedLargeText.java
          core/src/main/java/hudson/console/ConsoleNote.java
          http://jenkins-ci.org/commit/jenkins/7aab7ebf317d8ad1c81b72e813c339fbab05dfca
          Log:
          [FIXED JENKINS-9349] let GZipInputStream release its native memory
          eagerly.
          (cherry picked from commit 313fb5940f63ca7e13281f4bba1cdbcfcfd8f2c3)

          Conflicts:

          changelog.html

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/src/main/java/hudson/console/AnnotatedLargeText.java core/src/main/java/hudson/console/ConsoleNote.java http://jenkins-ci.org/commit/jenkins/7aab7ebf317d8ad1c81b72e813c339fbab05dfca Log: [FIXED JENKINS-9349] let GZipInputStream release its native memory eagerly. (cherry picked from commit 313fb5940f63ca7e13281f4bba1cdbcfcfd8f2c3) Conflicts: changelog.html

            People

            • Assignee:
              stevengbrown Steven G Brown
              Reporter:
              ford30066 Matthew Ford
            • Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: