Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20051

Winstone goes down on 1.509.4 (due to LDAP issues?)

    Details

    • Similar Issues:

      Description

      We have experienced a strange issue with Jenkins 1.509.4. Web interface has gone offline after several days.

      • "RMI TCP Connection" thread is not present in the thread dump
      • "RMI Scheduler" and "RMI accept" exist in the thread
      • There is no info in jenkins.log
      • There's no BLOCKED threads
      • Jenkins internal jobs work well, only the web interface is not available

      After analysis of logs, seems that issue has been caused by "Too many open files" error:

      Oct 14, 2013 5:27:48 PM winstone.Logger logInternal
      SEVERE: Error during HTTP listener init or shutdown
      java.net.SocketException: Too many open files
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
      at java.net.ServerSocket.implAccept(ServerSocket.java:530)
      at java.net.ServerSocket.accept(ServerSocket.java:498)
      at winstone.HttpListener.run(HttpListener.java:139)
      at java.lang.Thread.run(Thread.java:724)

      According to logs, there is a huge number of LDAP errors

      WARNING: Failed to search LDAP for username=USER_NAME
      org.acegisecurity.ldap.LdapDataAccessException: Failed to obtain InitialDirContext due to unexpected exception; nested exception is javax.naming.ServiceUnavailableException: ru20waddc.internal.synopsys.com:389; socket closed
      at org.acegisecurity.ldap.DefaultInitialDirContextFactory.connect(DefaultInitialDirContextFactory.java:193)
      at org.acegisecurity.ldap.DefaultInitialDirContextFactory.newInitialDirContext(DefaultInitialDirContextFactory.java:261)
      at org.acegisecurity.ldap.DefaultInitialDirContextFactory.newInitialDirContext(DefaultInitialDirContextFactory.java:241)
      at org.acegisecurity.ldap.LdapTemplate.execute(LdapTemplate.java:123)
      at org.acegisecurity.ldap.LdapTemplate.searchForSingleEntry(LdapTemplate.java:246)
      at org.acegisecurity.ldap.search.FilterBasedLdapUserSearch.searchForUser(FilterBasedLdapUserSearch.java:119)
      at hudson.security.LDAPSecurityRealm$LDAPUserDetailsService.loadUserByUsername(LDAPSecurityRealm.java:591)
      at hudson.security.LDAPSecurityRealm$LDAPUserDetailsService.loadUserByUsername(LDAPSecurityRealm.java:553)
      at org.acegisecurity.ui.rememberme.TokenBasedRememberMeServices.loadUserDetails(TokenBasedRememberMeServices.java:308)
      at org.acegisecurity.ui.rememberme.TokenBasedRememberMeServices.autoLogin(TokenBasedRememberMeServices.java:218)
      at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:104)
      ...
      at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
      at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227)
      at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:724)
      Caused by: javax.naming.ServiceUnavailableException: LDAP_ADDRESS; socket closed
      at com.sun.jndi.ldap.Connection.readReply(Connection.java:454)
      at com.sun.jndi.ldap.LdapClient.ldapBind(LdapClient.java:364)
      at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:213)
      at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2740)
      at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:316)
      at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:193)
      at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:211)
      at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:154)
      at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:84)
      at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
      at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
      at javax.naming.InitialContext.init(InitialContext.java:242)
      at javax.naming.InitialContext.<init>(InitialContext.java:216)
      at javax.naming.directory.InitialDirContext.<init>(InitialDirContext.java:101)
      at org.acegisecurity.ldap.DefaultInitialDirContextFactory.connect(DefaultInitialDirContextFactory.java:180)
      ... 39 more

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                oleg_nenashev Oleg Nenashev
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: