Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56947

File handle leak in EnvActionImpl when accessing environment of completed builds

    Details

    • Similar Issues:

      Description

      We recently upgraded workflow-job-plugin from 2.24 => 2.32 and started seeing file handle leaks in our builds.

      We installed the file handle leak plugin and saw a lot of file handles being locked by other builds. This only seemed to happen when we were accessing the previous build's environment information.

      We narrowed it down to it using the EnvActionImpl to access environment variables. To workaround the issue, we called getEnvironment(TaskListener.NULL) to access the environment before accessing the variables.

      I'm wondering if this change is intentional. I believe the cause of this issue is that the EnvActionImpl getListener() uses the job's FlowExecution getListener(). This listener is set to always initialize whenever it is null. Whenever, a WorkFlowRun finished (calls finish()), the listener is set to null, so whenever something else accesses the build environment, the listener is re-initialized.

       

      Function causing file leak:

      @NonCPS
      private static def getFailureCause(build) {
          return build.previousBuild.getAction(org.jenkinsci.plugins.workflow.cps.EnvActionImpl)?.failureCause
        }
      

      File handle detector output:

      #99 /var/lib/jenkins/jobs/MikeTest/builds/6/log by thread:Running CpsFlowExecution[Owner[MikeTest/7:MikeTest #7]] on Fri Apr 05 17:29:43 PDT 2019
      	at java.io.FileOutputStream.<init>(FileOutputStream.java:214)
      	at org.jenkinsci.plugins.workflow.log.FileLogStorage.open(FileLogStorage.java:87)
      	at org.jenkinsci.plugins.workflow.log.FileLogStorage.access$000(FileLogStorage.java:61)
      	at org.jenkinsci.plugins.workflow.log.FileLogStorage$IndexOutputStream.<init>(FileLogStorage.java:146)
      	at org.jenkinsci.plugins.workflow.log.FileLogStorage.overallListener(FileLogStorage.java:115)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getListener(WorkflowRun.java:219)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.getListener(WorkflowRun.java:954)
      	at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getListener(EnvActionImpl.java:75)
      	at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getProperty(EnvActionImpl.java:99)
      	at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getProperty(EnvActionImpl.java:53)
      	at org.codehaus.groovy.runtime.InvokerHelper.getProperty(InvokerHelper.java:174)
      	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.getProperty(ScriptBytecodeAdapter.java:456)
      	at org.kohsuke.groovy.sandbox.impl.Checker$6.call(Checker.java:290)
      	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onGetProperty(GroovyInterceptor.java:68)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:279)
      	at org.kohsuke.groovy.sandbox.impl.Checker$6.call(Checker.java:288)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:292)
      	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.getProperty(SandboxInvoker.java:29)
      	at com.cloudbees.groovy.cps.impl.PropertyAccessBlock.rawGet(PropertyAccessBlock.java:20)
      	at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.get(PropertyishBlock.java:74)
      	at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
      	at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.fixName(PropertyishBlock.java:66)
      	at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

        Attachments

          Activity

          There are no comments yet on this issue.

            People

            • Assignee:
              Unassigned
              Reporter:
              michaelw Michael Wong
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: