Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18524

Too many open files caused Jenkins crashed

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Cannot Reproduce
    • Component/s: other
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      After Jenkins running for a long time, it suddenly stuck. We checked the log and found too many socket was opened and the number of opened files comes 65535. So it failed to accept new request. Could you please help to check why Jenkins opened so many socket and can't close it?
      We have tried to use gc to collect garbage memory, but it can't close these socket. So this should be a bug of Jenkins.
      Jun 13, 2013 10:21:12 AM hudson.model.Executor run
      SEVERE: Executor threw an exception
      hudson.util.IOException2: Failed to create a temporary file in /proj/RBS_IoV/.hudson/jobs/KillZombieProcess
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:67)
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:54)
      at hudson.util.TextFile.write(TextFile.java:78)
      at hudson.model.Job.saveNextBuildNumber(Job.java:234)
      at hudson.model.Job.assignBuildNumber(Job.java:291)
      at hudson.model.Run.<init>(Run.java:256)
      at hudson.model.AbstractBuild.<init>(AbstractBuild.java:158)
      at hudson.model.Build.<init>(Build.java:92)
      at hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:37)
      at sun.reflect.GeneratedConstructorAccessor100.newInstance(Unknown Source)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at hudson.model.AbstractProject.newBuild(AbstractProject.java:982)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1197)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:136)
      at hudson.model.Executor.run(Executor.java:211)
      Caused by: java.io.IOException: Too many open files
      at java.io.UnixFileSystem.createFileExclusively(Native Method)
      at java.io.File.checkAndCreate(File.java:1704)
      at java.io.File.createTempFile(File.java:1792)
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:65)
      ... 15 more
      Jun 13, 2013 10:21:12 AM hudson.model.Executor run
      SEVERE: Executor threw an exception
      hudson.util.IOException2: Failed to create a temporary file in /proj/RBS_IoV/.hudson/jobs/W12A_CLA_R6_CI_testExec_Trigger_Job
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:67)
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:54)
      at hudson.util.TextFile.write(TextFile.java:78)
      at hudson.model.Job.saveNextBuildNumber(Job.java:234)
      at hudson.model.Job.assignBuildNumber(Job.java:291)
      at hudson.model.Run.<init>(Run.java:256)
      at hudson.model.AbstractBuild.<init>(AbstractBuild.java:158)
      at hudson.model.Build.<init>(Build.java:92)
      at hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:37)
      at sun.reflect.GeneratedConstructorAccessor100.newInstance(Unknown Source)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at hudson.model.AbstractProject.newBuild(AbstractProject.java:982)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1197)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:136)
      at hudson.model.Executor.run(Executor.java:211)
      Caused by: java.io.IOException: Too many open files
      at java.io.UnixFileSystem.createFileExclusively(Native Method)
      at java.io.File.checkAndCreate(File.java:1704)
      at java.io.File.createTempFile(File.java:1792)
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:65)
      ... 15 more
      Jun 13, 2013 10:21:13 AM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
      SEVERE: Socket accept failed
      java.net.SocketException: Too many open files

        Attachments

        1. jstack.log
          0.7 kB
        2. lsof.log
          8.41 MB
        3. tomcat.log
          6 kB
        4. ulimit.log
          0.2 kB

          Issue Links

            Activity

            Hide
            martin_naughton martin naughton added a comment -

            did you investigate what the sockets might be doing?

            Show
            martin_naughton martin naughton added a comment - did you investigate what the sockets might be doing?
            Hide
            oleg_nenashev Oleg Nenashev added a comment - - edited

            Maybe, the issue duplicates https://issues.jenkins-ci.org/browse/JENKINS-20051
            In this case, LDAP sockets hang due to LDAP server overloading.

            Show
            oleg_nenashev Oleg Nenashev added a comment - - edited Maybe, the issue duplicates https://issues.jenkins-ci.org/browse/JENKINS-20051 In this case, LDAP sockets hang due to LDAP server overloading.
            Hide
            danielbeck Daniel Beck added a comment -

            Are you using the Jenkins CLI (e.g. JENKINS-23572)? Does this issue also occur on current Jenkins versions?

            Show
            danielbeck Daniel Beck added a comment - Are you using the Jenkins CLI (e.g. JENKINS-23572 )? Does this issue also occur on current Jenkins versions?
            Hide
            eyouhan Youcai Han added a comment -

            Yes, we used JCLI. But we haven't found this issue for about 1 year with the same Jenkins version 1.480.1.

            Show
            eyouhan Youcai Han added a comment - Yes, we used JCLI. But we haven't found this issue for about 1 year with the same Jenkins version 1.480.1.
            Hide
            danielbeck Daniel Beck added a comment -

            Youcai Han To clarify, you have not experienced the issue reported here for one year?

            If so, I suggest resolving as Cannot Reproduce. You can file a new issue anytime it occurs again with updated logs and config information...

            Show
            danielbeck Daniel Beck added a comment - Youcai Han To clarify, you have not experienced the issue reported here for one year? If so, I suggest resolving as Cannot Reproduce. You can file a new issue anytime it occurs again with updated logs and config information...
            Hide
            eyouhan Youcai Han added a comment -

            Yes, please close it.

            Show
            eyouhan Youcai Han added a comment - Yes, please close it.
            Hide
            danielbeck Daniel Beck added a comment -

            As discussed, resolving as Cannot Reproduce as the reporter did not experience the issue in a year.

            Show
            danielbeck Daniel Beck added a comment - As discussed, resolving as Cannot Reproduce as the reporter did not experience the issue in a year.

              People

              • Assignee:
                Unassigned
                Reporter:
                hannyjxb Youcai HAN
              • Votes:
                3 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: