Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42026

Memory leak while running concurrent pipelines

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Fix
    • Icon: Major Major
    • workflow-cps-plugin
    • None
    • openjdk version "1.8.0_45-internal"
      Ubuntu 15.04
      workflow-cps 2.27-SNAPSHOT (private-1155b889-jenkins)
      pipeline-model-definition 1.0.1
      core 2.32.2

      Running >100 Pipeline builds can cause OutOfMemoryErrors, leading to Jenkins eventually becoming unresponsive.

      Sample log snippet below. After I get this issue opened, I'll recreate to produce cleaner logs.

      WARNING: RunListener failed
      java.lang.OutOfMemoryError: GC overhead limit exceeded
      
      Feb 14, 2017 2:47:04 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem
      WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[PipelinePipelines/pipeline-testcases---parallel-abuse180/198:PipelinePipelines/pipeline-testcases---parallel-abuse180 #198]]
      java.lang.OutOfMemoryError: GC overhead limit exceeded
      
      Feb 14, 2017 2:47:03 PM hudson.model.AsyncPeriodicWork doRun
      INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
      Feb 14, 2017 2:47:10 PM hudson.model.AsyncPeriodicWork doRun
      INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
      Feb 14, 2017 2:47:20 PM hudson.model.AsyncPeriodicWork doRun
      INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
      Feb 14, 2017 2:47:27 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
      INFO: PipelinePipelines/pipeline-testcases---parallel-abuse24 #198 completed: FAILURE
      Feb 14, 2017 2:47:28 PM hudson.model.AsyncPeriodicWork doRun
      INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
      Feb 14, 2017 2:47:30 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem
      WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[PipelinePipelines/pipeline-testcases---parallel-abuse180/198:PipelinePipelines/pipeline-testcases---parallel-abuse180 #198]]
      java.lang.NullPointerException
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:41)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:165)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      Feb 14, 2017 2:47:35 PM hudson.triggers.SafeTimerTask run
      SEVERE: Timer task hudson.model.Queue$MaintainTask@34463f11 failed
      java.lang.OutOfMemoryError: GC overhead limit exceeded
      
      Feb 14, 2017 2:47:46 PM hudson.model.AsyncPeriodicWork doRun
      INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
      Feb 14, 2017 2:47:55 PM hudson.model.AsyncPeriodicWork doRun
      INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
      Feb 14, 2017 2:48:00 PM hudson.model.AsyncPeriodicWork doRun
      INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
      Feb 14, 2017 2:48:06 PM hudson.model.AsyncPeriodicWork doRun
      INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
      Feb 14, 2017 2:48:14 PM hudson.model.AsyncPeriodicWork doRun
      INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
      Feb 14, 2017 2:48:21 PM hudson.model.AsyncPeriodicWork doRun
      INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
      Feb 14, 2017 2:48:24 PM org.jenkinsci.plugins.workflow.flow.FlowExecutionList load
      WARNING: null
      java.io.IOException: Unable to read /var/lib/jenkins/org.jenkinsci.plugins.workflow.flow.FlowExecutionList.xml
      	at hudson.XmlFile.read(XmlFile.java:144)
      	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList.load(FlowExecutionList.java:99)
      	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList.register(FlowExecutionList.java:112)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:233)
      	at hudson.model.ResourceController.execute(ResourceController.java:98)
      	at hudson.model.Executor.run(Executor.java:404)
      Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
      
      

            kshultz Karl Shultz
            kshultz Karl Shultz
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: