Jenkins reports some executors to be "Dead!", and in the jenkins log there's multiple exceptions [1] where the root cause seems to be
Caused by: java.io.IOException: Too many open files
Attached file contains the output of
lsof |grep jenkins
There seems to be lot of lines beginning
/home/jenkins-slave/.jenkins/cache/jars/..
Disconnecting and connecting slave did not help. Clicking on "Dead" text
and then "Restart this thread" worked.
[1] Few Exceptions where too many open files seems to be the reason
SEVERE: Unexpected executor death
hudson.util.IOException2: Failed to create a temporary file in /var/lib/jenkins/jobs/RequesteX_Tests
at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:67)
at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:54)
at hudson.util.TextFile.write(TextFile.java:84)
at hudson.model.Job.saveNextBuildNumber(Job.java:265)
...
Caused by: java.io.IOException: Too many open files
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:1879)
at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:65)
... 15 more
hudson.util.IOException2: Failed to create a temporary file in /var/lib/jenkins/jobs/RequesteX_Tests/builds/2013-11-08_15-17-54
at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:67)
at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:54)
at hudson.XmlFile.write(XmlFile.java:180)
at hudson.model.Run.save(Run.java:1848)
...
Caused by: java.io.IOException: Too many open files
at java.io.UnixFileSystem.createFileExclusively(Native Method)
Nov 08, 2013 2:48:55 PM hudson.triggers.SCMTrigger$Runner runPolling
SEVERE: Failed to record SCM polling for hudson.maven.MavenModuleSet@4e49e681[RequesteX_Final]
java.io.FileNotFoundException: /var/lib/jenkins/jobs/RequesteX_Final/scm-polling.log (Too many open files)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:212)
at java.io.FileOutputStream.<init>(FileOutputStream.java:165)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:433)