Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18524

Too many open files caused Jenkins crashed

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Cannot Reproduce
    • Component/s: other
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      After Jenkins running for a long time, it suddenly stuck. We checked the log and found too many socket was opened and the number of opened files comes 65535. So it failed to accept new request. Could you please help to check why Jenkins opened so many socket and can't close it?
      We have tried to use gc to collect garbage memory, but it can't close these socket. So this should be a bug of Jenkins.
      Jun 13, 2013 10:21:12 AM hudson.model.Executor run
      SEVERE: Executor threw an exception
      hudson.util.IOException2: Failed to create a temporary file in /proj/RBS_IoV/.hudson/jobs/KillZombieProcess
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:67)
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:54)
      at hudson.util.TextFile.write(TextFile.java:78)
      at hudson.model.Job.saveNextBuildNumber(Job.java:234)
      at hudson.model.Job.assignBuildNumber(Job.java:291)
      at hudson.model.Run.<init>(Run.java:256)
      at hudson.model.AbstractBuild.<init>(AbstractBuild.java:158)
      at hudson.model.Build.<init>(Build.java:92)
      at hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:37)
      at sun.reflect.GeneratedConstructorAccessor100.newInstance(Unknown Source)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at hudson.model.AbstractProject.newBuild(AbstractProject.java:982)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1197)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:136)
      at hudson.model.Executor.run(Executor.java:211)
      Caused by: java.io.IOException: Too many open files
      at java.io.UnixFileSystem.createFileExclusively(Native Method)
      at java.io.File.checkAndCreate(File.java:1704)
      at java.io.File.createTempFile(File.java:1792)
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:65)
      ... 15 more
      Jun 13, 2013 10:21:12 AM hudson.model.Executor run
      SEVERE: Executor threw an exception
      hudson.util.IOException2: Failed to create a temporary file in /proj/RBS_IoV/.hudson/jobs/W12A_CLA_R6_CI_testExec_Trigger_Job
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:67)
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:54)
      at hudson.util.TextFile.write(TextFile.java:78)
      at hudson.model.Job.saveNextBuildNumber(Job.java:234)
      at hudson.model.Job.assignBuildNumber(Job.java:291)
      at hudson.model.Run.<init>(Run.java:256)
      at hudson.model.AbstractBuild.<init>(AbstractBuild.java:158)
      at hudson.model.Build.<init>(Build.java:92)
      at hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:37)
      at sun.reflect.GeneratedConstructorAccessor100.newInstance(Unknown Source)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at hudson.model.AbstractProject.newBuild(AbstractProject.java:982)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1197)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:136)
      at hudson.model.Executor.run(Executor.java:211)
      Caused by: java.io.IOException: Too many open files
      at java.io.UnixFileSystem.createFileExclusively(Native Method)
      at java.io.File.checkAndCreate(File.java:1704)
      at java.io.File.createTempFile(File.java:1792)
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:65)
      ... 15 more
      Jun 13, 2013 10:21:13 AM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
      SEVERE: Socket accept failed
      java.net.SocketException: Too many open files

        Attachments

        1. jstack.log
          0.7 kB
        2. lsof.log
          8.41 MB
        3. tomcat.log
          6 kB
        4. ulimit.log
          0.2 kB

          Issue Links

            Activity

            hannyjxb Youcai HAN created issue -
            oleg_nenashev Oleg Nenashev made changes -
            Field Original Value New Value
            Link This issue is related to JENKINS-20051 [ JENKINS-20051 ]
            danielbeck Daniel Beck made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Cannot Reproduce [ 5 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 149831 ] JNJira + In-Review [ 193312 ]

              People

              • Assignee:
                Unassigned
                Reporter:
                hannyjxb Youcai HAN
              • Votes:
                3 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: