Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-13109

Huge changelogs eat all the Java memory

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: perforce-plugin
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      With really huge changelists the p4 plugin can run out of java heap space. At least it looks like the reason for memory problems would be huge changelog.xml files. An example case:

      • a changelist with > 400000 files
      • results in a changelog.xml size > 110MB
      • Jenkins runs out of memory: java.lang.OutOfMemoryError: Java heap space

      Maybe there should be an option to limit the amount of files that p4 plugin reads to from changelists?

        Attachments

          Activity

          Hide
          rpetti Rob Petti added a comment -

          Limiting the number of files is probably the easiest option here. Though another option is to somehow stream the information from disk on-demand, rather than deserializing the entire thing into the heap.

          For now, what would a good limit be?

          Show
          rpetti Rob Petti added a comment - Limiting the number of files is probably the easiest option here. Though another option is to somehow stream the information from disk on-demand, rather than deserializing the entire thing into the heap. For now, what would a good limit be?
          Hide
          miktap Mikko Tapaninen added a comment -

          I was thinking of letting user to decide the limit. I have an implementation for that but I'll need to investigate further does it help and how the memory gets reserved.

          Show
          miktap Mikko Tapaninen added a comment - I was thinking of letting user to decide the limit. I have an implementation for that but I'll need to investigate further does it help and how the memory gets reserved.
          Hide
          lestin lestin added a comment -

          We seem to have the same problem as well. Our files are not that big and it start to halt the webpage around log part 1560.
          The biggest impact was, the webpage was not responsive and the only way we can do is kill the java process. — Is there a better way?

          Thanks.
          =========================
          Mar 23, 2012 2:33:15 PM hudson.plugins.logparser.LogParserReader readLogPart
          INFO: Done reading log part 1560
          Mar 23, 2012 2:33:15 PM hudson.plugins.logparser.LogParserThread getLineStatuses
          INFO: LogParserThread: Start parsing log part 1560
          Mar 23, 2012 2:33:15 PM hudson.triggers.SCMTrigger$Runner runPolling
          SEVERE: Failed to record SCM polling
          java.lang.OutOfMemoryError: GC overhead limit exceeded
          at java.io.BufferedReader.<init>(BufferedReader.java:80)
          at java.io.BufferedReader.<init>(BufferedReader.java:91)
          at hudson.plugins.perforce.HudsonP4DefaultExecutor.getReader(HudsonP4DefaultExecutor.java:123)
          at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:322)
          at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:292)
          at com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:61)
          at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:1325)
          at hudson.plugins.perforce.PerforceSCM.compareRemoteRevisionWith(PerforceSCM.java:1019)
          at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:356)
          at hudson.scm.SCM.poll(SCM.java:373)
          at hudson.model.AbstractProject.poll(AbstractProject.java:1340)
          at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:420)
          at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:449)
          at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:619)
          Mar 23, 2012 2:33:22 PM hudson.plugins.logparser.LogParserThread getLineStatuses
          INFO: LogParserThread: Done parsing log part 1553
          Mar 23, 2012 2:33:15 PM hudson.plugins.logparser.LogParserReader readLogPart
          INFO: Start reading log part 1561 in thread #104
          ====================================

          Show
          lestin lestin added a comment - We seem to have the same problem as well. Our files are not that big and it start to halt the webpage around log part 1560. The biggest impact was, the webpage was not responsive and the only way we can do is kill the java process. — Is there a better way? Thanks. ========================= Mar 23, 2012 2:33:15 PM hudson.plugins.logparser.LogParserReader readLogPart INFO: Done reading log part 1560 Mar 23, 2012 2:33:15 PM hudson.plugins.logparser.LogParserThread getLineStatuses INFO: LogParserThread: Start parsing log part 1560 Mar 23, 2012 2:33:15 PM hudson.triggers.SCMTrigger$Runner runPolling SEVERE: Failed to record SCM polling java.lang.OutOfMemoryError: GC overhead limit exceeded at java.io.BufferedReader.<init>(BufferedReader.java:80) at java.io.BufferedReader.<init>(BufferedReader.java:91) at hudson.plugins.perforce.HudsonP4DefaultExecutor.getReader(HudsonP4DefaultExecutor.java:123) at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:322) at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:292) at com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:61) at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:1325) at hudson.plugins.perforce.PerforceSCM.compareRemoteRevisionWith(PerforceSCM.java:1019) at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:356) at hudson.scm.SCM.poll(SCM.java:373) at hudson.model.AbstractProject.poll(AbstractProject.java:1340) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:420) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:449) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Mar 23, 2012 2:33:22 PM hudson.plugins.logparser.LogParserThread getLineStatuses INFO: LogParserThread: Done parsing log part 1553 Mar 23, 2012 2:33:15 PM hudson.plugins.logparser.LogParserReader readLogPart INFO: Start reading log part 1561 in thread #104 ====================================
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Rob Petti
          Path:
          src/main/java/com/tek42/perforce/parse/ChangelistBuilder.java
          src/main/java/com/tek42/perforce/parse/Changes.java
          src/main/java/hudson/plugins/perforce/PerforceSCM.java
          src/main/resources/hudson/plugins/perforce/PerforceSCM/config.jelly
          src/main/webapp/help/fileLimit.html
          src/test/java/hudson/plugins/perforce/PerforceSCMTest.java
          http://jenkins-ci.org/commit/perforce-plugin/a2523100d02ca360fd23b382af4e531c4dc6ac59
          Log:
          Merge pull request #18 from miktap/master

          JENKINS-13109 Huge changelogs eat all the Java memory

          Compare: https://github.com/jenkinsci/perforce-plugin/compare/11249ca...a252310

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Rob Petti Path: src/main/java/com/tek42/perforce/parse/ChangelistBuilder.java src/main/java/com/tek42/perforce/parse/Changes.java src/main/java/hudson/plugins/perforce/PerforceSCM.java src/main/resources/hudson/plugins/perforce/PerforceSCM/config.jelly src/main/webapp/help/fileLimit.html src/test/java/hudson/plugins/perforce/PerforceSCMTest.java http://jenkins-ci.org/commit/perforce-plugin/a2523100d02ca360fd23b382af4e531c4dc6ac59 Log: Merge pull request #18 from miktap/master JENKINS-13109 Huge changelogs eat all the Java memory Compare: https://github.com/jenkinsci/perforce-plugin/compare/11249ca...a252310
          Hide
          timdrury Tim Drury added a comment - - edited

          I noticed the same error today on 1.450 w/ perforce plugin 1.3.7. In my case, it isn't caused by huge changelogs - at least that I can tell. We don't have huge changelogs. We do have 2 slaves and all polling is done on the master. Here is the stack trace:

          Apr 1, 2012 7:48:48 PM hudson.triggers.SCMTrigger$Runner runPolling
          SEVERE: Failed to record SCM polling
          java.lang.OutOfMemoryError: Java heap space
          at hudson.remoting.FastPipedInputStream.<init>(FastPipedInputStream.java:78)
          at hudson.remoting.FastPipedInputStream.<init>(FastPipedInputStream.java:66)
          at hudson.plugins.perforce.HudsonP4RemoteExecutor.exec(HudsonP4RemoteExecutor.java:97)
          at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:321)
          at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:292)
          at com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:54)
          at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:1208)
          at hudson.plugins.perforce.PerforceSCM.compareRemoteRevisionWith(PerforceSCM.java:903)
          at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:356)
          at hudson.scm.SCM.poll(SCM.java:373)
          at hudson.model.AbstractProject.poll(AbstractProject.java:1323)
          at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:420)
          at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:449)
          at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:662)

          We do keep Jenkins running without periodic reboots so I'm thinking there may be a memory leak somewhere as well. If anyone can give me hints how to get more information, I'll try to provide it.

          -tim

          Show
          timdrury Tim Drury added a comment - - edited I noticed the same error today on 1.450 w/ perforce plugin 1.3.7. In my case, it isn't caused by huge changelogs - at least that I can tell. We don't have huge changelogs. We do have 2 slaves and all polling is done on the master. Here is the stack trace: Apr 1, 2012 7:48:48 PM hudson.triggers.SCMTrigger$Runner runPolling SEVERE: Failed to record SCM polling java.lang.OutOfMemoryError: Java heap space at hudson.remoting.FastPipedInputStream.<init>(FastPipedInputStream.java:78) at hudson.remoting.FastPipedInputStream.<init>(FastPipedInputStream.java:66) at hudson.plugins.perforce.HudsonP4RemoteExecutor.exec(HudsonP4RemoteExecutor.java:97) at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:321) at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:292) at com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:54) at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:1208) at hudson.plugins.perforce.PerforceSCM.compareRemoteRevisionWith(PerforceSCM.java:903) at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:356) at hudson.scm.SCM.poll(SCM.java:373) at hudson.model.AbstractProject.poll(AbstractProject.java:1323) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:420) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:449) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) We do keep Jenkins running without periodic reboots so I'm thinking there may be a memory leak somewhere as well. If anyone can give me hints how to get more information, I'll try to provide it. -tim
          Hide
          rpetti Rob Petti added a comment -

          Tim, the wiki has some information on how to debug memory leaks:
          https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+OutOfMemoryError

          I'll also note that you are getting a different error from the one filed in the ticket, so your cause might be different.

          Show
          rpetti Rob Petti added a comment - Tim, the wiki has some information on how to debug memory leaks: https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+OutOfMemoryError I'll also note that you are getting a different error from the one filed in the ticket, so your cause might be different.
          Hide
          timdrury Tim Drury added a comment - - edited

          Ok, I'll move my stack trace to a new issue and I'll update with OOM Error info when/if it occurs again. Thanks again, Rob.

          Created this issue: https://issues.jenkins-ci.org/browse/JENKINS-13332

          Show
          timdrury Tim Drury added a comment - - edited Ok, I'll move my stack trace to a new issue and I'll update with OOM Error info when/if it occurs again. Thanks again, Rob. Created this issue: https://issues.jenkins-ci.org/browse/JENKINS-13332

            People

            • Assignee:
              Unassigned
              Reporter:
              miktap Mikko Tapaninen
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: