Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42186

File leak in modern SVNRepositoryView, "Too many open files"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • subversion-plugin
    • None
    • Jenkins 2.47
      Suversion plugin 2.7.1
      Linux and windows

      We are running into a problem where we are running out of file descriptors on our Jenkins master. The problems has started since we started to migrate our jobs to Pipelines which relies heavily on a library loaded using Global Pipeline Library. During the weekend we got this exception:

      java.io.IOException: failed to make/load /var/lib/jenkins/caches/svn-0f165bb6-15b2-11e6-ac23-b9db5cb4ce28.db: java.io.IOError: java.io.FileNotFoundException: /var/lib/jenkins/caches/svn-0f165bb6-15b2-11e6-ac23-b9db5cb4ce28.db.p (Too many open files)
	at jenkins.scm.impl.subversion.SVNRepositoryView.<init>(SVNRepositoryView.java:118)
	at jenkins.scm.impl.subversion.SubversionSCMSource.openSession(SubversionSCMSource.java:328)
	at jenkins.scm.impl.subversion.SubversionSCMSource.retrieve(SubversionSCMSource.java:287)
	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:278)
	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:79)
	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:150)
	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:131)
	at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:99)
	at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:970)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:548)
	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:526)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:503)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:302)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:281)
	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:731)
	at groovy.lang.GroovyShell.parse(GroovyShell.java:743)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:67)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:429)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:392)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:221)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:410)
Caused by: java.io.IOError: java.io.FileNotFoundException: /var/lib/jenkins/caches/svn-0f165bb6-15b2-11e6-ac23-b9db5cb4ce28.db.p (Too many open files)
	at org.mapdb.Volume$FileChannelVol.<init>(Volume.java:672)
	at org.mapdb.Volume.volumeForFile(Volume.java:176)
	at org.mapdb.Volume$1.createPhysVolume(Volume.java:208)
	at org.mapdb.StoreDirect.<init>(StoreDirect.java:203)
	at org.mapdb.StoreWAL.<init>(StoreWAL.java:74)
	at org.mapdb.DBMaker.extendStoreWAL(DBMaker.java:928)
	at org.mapdb.DBMaker.makeEngine(DBMaker.java:722)
	at org.mapdb.DBMaker.make(DBMaker.java:665)
	at jenkins.scm.impl.subversion.SVNRepositoryView.<init>(SVNRepositoryView.java:113)
	... 21 more
Caused by: java.io.FileNotFoundException: /var/lib/jenkins/caches/svn-0f165bb6-15b2-11e6-ac23-b9db5cb4ce28.db.p (Too many open files)
	at java.io.RandomAccessFile.open(Native Method)
	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
	at org.mapdb.Volume$FileChannelVol.<init>(Volume.java:667)
	... 29 more
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: Loading libraries failed

      Using lsof i can see that we have loads 6000+ file descriptors pointing to:

      java    58260 jenkins 8181u      REG              254,6      33096    33119 /var/lib/jenkins/caches/svn-0f165bb6-15b2-11e6-ac23-b9db5cb4ce28.db (deleted)
java    58260 jenkins 8182u      REG              254,6       7728    33120 /var/lib/jenkins/caches/svn-0f165bb6-15b2-11e6-ac23-b9db5cb4ce28.db.p
java    58260 jenkins 8183u      REG              254,6         16    33121 /var/lib/jenkins/caches/svn-0f165bb6-15b2-11e6-ac23-b9db5cb4ce28.db.t

      This was on our Linux server (SMP Debian 3.2.81-2 x86_64 GNU/Linux). So, in order to isolate the issue I downloaded and setup the latest and shiniest version of Jenkins (2.47, default plugins) on my local Windows 7 machine. I was able to reproduce the problem here as well.

      To reproduce:

      • Add a dummy global pipeline library with a single function foo that uses modern svn.
      • Add a new pipeline job: 
        @Library("foo") _

node {
    foo();
}
build job:'foo', quietPeriod: 1, wait: false
      • And build it. As you will see this will produce builds until stopped. After a few 100 builds stop it
        Now, using a system tool like lsof or Process Explorer inspect file-handles used by Jenkins, you will have loads of file handles pointing to the caches folder in JENKINS_HOME.

      I'm setting severity as critical considering that it affects anyone using GPL and SVN. More people will probably run in to this issue considering that the Jenkins community is pushing hard for Pipelines and GPL. Also since it's a file leak issue, it only shows up in system which has been running for a while and not your sandbox where you play around!

            jamesdumay James Dumay
            jons Jon Sten
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: