Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37737

Intermittent login failures with Active Directory / Matrix-based security

    Details

    • Similar Issues:

      Description

      Helllo! We are experiencing intermittent login issues since early August, 2016 for all users from any browser or workstation (location does not seem to be an issue). We have a cross domain - VPN tunnel, which has not experienced recent outages to cause failed logons or AD lookups. Other systems relying on the VPN tunnel are not experiencing authentication issues. Successful manual telnet tests between the Domain Controllers were successful during Jenkins failed logins. We are not ruling out a network issue but we can't see any problems. We have not recently upgraded Jenkins or the Active Directory Plugin.

      Looking forward to any help to resolve our issue.

      Output from log:

      Aug 27, 2016 7:11:51 AM hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider retrieveUser
      WARNING: Credential exception trying to authenticate against ####### domain
      org.acegisecurity.BadCredentialsException: Failed to retrieve user information for ##############; nested exception is javax.naming.PartialResultException Root exception is javax.naming.CommunicationException: DomainDnsZones.######## [Root exception is java.net.ConnectException: Connection timed out: connect]
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider$1.call(ActiveDirectoryUnixAuthenticationProvider.java:332)
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider$1.call(ActiveDirectoryUnixAuthenticationProvider.java:235)
      at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
      at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
      at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
      at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
      at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
      at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
      at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:235)
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:200)
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:142)
      at org.acegisecurity.providers.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:122)
      at org.acegisecurity.providers.ProviderManager.doAuthentication(ProviderManager.java:200)
      at org.acegisecurity.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:47)
      at org.acegisecurity.ui.webapp.AuthenticationProcessingFilter.attemptAuthentication(AuthenticationProcessingFilter.java:74)
      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:252)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      at org.eclipse.jetty.server.Server.handle(Server.java:370)
      at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
      at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
      at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
      at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
      at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
      at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
      at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
      at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
      at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
      at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)
      Caused by: javax.naming.PartialResultException Root exception is javax.naming.CommunicationException: DomainDnsZones.####### [Root exception is java.net.ConnectException: Connection timed out: connect]
      at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMoreImpl(Unknown Source)
      at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMore(Unknown Source)
      at hudson.plugins.active_directory.LDAPSearchBuilder.searchOne(LDAPSearchBuilder.java:86)
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider$1.call(ActiveDirectoryUnixAuthenticationProvider.java:280)
      ... 55 more

        Attachments

          Activity

          Hide
          jr Jens Runge added a comment -

          Hello!

          We have the same problems here.
          In our case i supect the reason in offline locations (DomainControllers listet in DomainDnsZones.<domain>).
          Until i convince our administrators to remove this offline locations from DNS, my workaround is to clear the name resolving cache on the jenkins server.

          IMO the LDAP-plugin ignores the configured LDAP-servers and resolve the Domain Controllers over DNS lookup DomainDnsZones.<domain> and runs into the problem with not available servers listet in DNS.

          Greetings
          JR

          Show
          jr Jens Runge added a comment - Hello! We have the same problems here. In our case i supect the reason in offline locations (DomainControllers listet in DomainDnsZones.<domain>). Until i convince our administrators to remove this offline locations from DNS, my workaround is to clear the name resolving cache on the jenkins server. IMO the LDAP-plugin ignores the configured LDAP-servers and resolve the Domain Controllers over DNS lookup DomainDnsZones.<domain> and runs into the problem with not available servers listet in DNS. Greetings JR
          Hide
          dsakauye Derek Sakauye added a comment -

          JR,

          Thanks for your reply. We have yet to resolve our issue. However, I wanted to note that we are using the AD Plugin (not LDAP-plugin). Does your workaround still apply? If so, how do we clear the name resolving cache on the jenkins server?

          Thanks!

          • Derek
          Show
          dsakauye Derek Sakauye added a comment - JR, Thanks for your reply. We have yet to resolve our issue. However, I wanted to note that we are using the AD Plugin (not LDAP-plugin). Does your workaround still apply? If so, how do we clear the name resolving cache on the jenkins server? Thanks! Derek
          Hide
          jr Jens Runge added a comment -

          Hi Derek,

          i meant the Active Directory plugin too. It was only a mistake in writing.

          Our Jenkins runs on a Windows server. On console a "ipconfig /flushdns" should help you.

          For Linux i found this page: http://www.cyberciti.biz/faq/rhel-debian-ubuntu-flush-clear-dns-cache/
          Maybe you have to search a other solution for your Linux-Distribution.

          Greetings
          JR

          Show
          jr Jens Runge added a comment - Hi Derek, i meant the Active Directory plugin too. It was only a mistake in writing. Our Jenkins runs on a Windows server. On console a "ipconfig /flushdns" should help you. For Linux i found this page: http://www.cyberciti.biz/faq/rhel-debian-ubuntu-flush-clear-dns-cache/ Maybe you have to search a other solution for your Linux-Distribution. Greetings JR
          Hide
          dsakauye Derek Sakauye added a comment -

          Hi JR,

          Is your workaround to prevent the issue from occurring? Or do you apply the ipconfig /flushdns only when you experience the logon authentication issue?

          Thanks for the help!

          Best regards,

          • Derek
          Show
          dsakauye Derek Sakauye added a comment - Hi JR, Is your workaround to prevent the issue from occurring? Or do you apply the ipconfig /flushdns only when you experience the logon authentication issue? Thanks for the help! Best regards, Derek
          Hide
          jr Jens Runge added a comment -

          Hi Derek,

          I admit its only a dirty workaround. Our Teams isn't that big, so I have done it every time somone can't login.
          Meanwhile our Administrators cleaned up the DNS, so that DomainDnsZones.<domain> only points to available systems.

          I assume, your Jenkins Service is running with a local server account.
          Then you can try to use a domain account as Jenkins service account on your Jenkins Server.
          Maybe then the windows native authentication for directory listing should not fail and would not fallback to non native authentication.

          Show
          jr Jens Runge added a comment - Hi Derek, I admit its only a dirty workaround. Our Teams isn't that big, so I have done it every time somone can't login. Meanwhile our Administrators cleaned up the DNS, so that DomainDnsZones.<domain> only points to available systems. I assume, your Jenkins Service is running with a local server account. Then you can try to use a domain account as Jenkins service account on your Jenkins Server. Maybe then the windows native authentication for directory listing should not fail and would not fallback to non native authentication.
          Hide
          dsakauye Derek Sakauye added a comment -

          Hi JR,

          Our team is pretty small too. So we're in a similar pickle. We actually are running Jenkins with a domain service account. I'll try your "flushdns" workaround next time we can't login. I'll check with Domain Admins about cleaning up the DNS too.

          Thanks,

          • Derek
          Show
          dsakauye Derek Sakauye added a comment - Hi JR, Our team is pretty small too. So we're in a similar pickle. We actually are running Jenkins with a domain service account. I'll try your "flushdns" workaround next time we can't login. I'll check with Domain Admins about cleaning up the DNS too. Thanks, Derek
          Hide
          vkadiri Vijaya Bhaskar Kadiri added a comment -

          Hello!

          We have the same problems with Jenkins ver. 2.7.1. Any ETA for the fix?

          Show
          vkadiri Vijaya Bhaskar Kadiri added a comment - Hello! We have the same problems with Jenkins ver. 2.7.1. Any ETA for the fix?
          Hide
          romgo Hugo added a comment -

          Hello,

          I have the same issue, Jenkins can only reach 2 controller i my domain, so I use the servers option to point to the server it can reach (don't want to use round robin from Active directory).
          But I can see request to other domain controller which are not configured in my list.
          This cause login failure and various timeout.

          Here is my configuration :

            <securityRealm class="hudson.plugins.active_directory.ActiveDirectorySecurityRealm" plugin="active-directory@2.0">
              <domains>
                <hudson.plugins.active__directory.ActiveDirectoryDomain>
                  <!-- <name>domain.local</name> -->
                  <servers>192.168.1.2:636,192.168.1.3:636</servers>
                </hudson.plugins.active__directory.ActiveDirectoryDomain>
              </domains>
              <bindName>CN=LDAP,OU=Users,DC=domain,DC=local</bindName>
              <bindPassword>DDHCNCNCCC/r9Rxf0HvCqt0QVuU=</bindPassword>
              <groupLookupStrategy>AUTO</groupLookupStrategy>
              <removeIrrelevantGroups>false</removeIrrelevantGroups>
            </securityRealm>
          

          plugin : 2.0
          jenkins : 2.34

          Show
          romgo Hugo added a comment - Hello, I have the same issue, Jenkins can only reach 2 controller i my domain, so I use the servers option to point to the server it can reach (don't want to use round robin from Active directory). But I can see request to other domain controller which are not configured in my list. This cause login failure and various timeout. Here is my configuration : <securityRealm class= "hudson.plugins.active_directory.ActiveDirectorySecurityRealm" plugin= "active-directory@2.0" > <domains> <hudson.plugins.active__directory.ActiveDirectoryDomain> <!-- <name>domain.local</name> --> <servers>192.168.1.2:636,192.168.1.3:636</servers> </hudson.plugins.active__directory.ActiveDirectoryDomain> </domains> <bindName>CN=LDAP,OU=Users,DC=domain,DC=local</bindName> <bindPassword>DDHCNCNCCC/r9Rxf0HvCqt0QVuU=</bindPassword> <groupLookupStrategy>AUTO</groupLookupStrategy> <removeIrrelevantGroups> false </removeIrrelevantGroups> </securityRealm> plugin : 2.0 jenkins : 2.34
          Hide
          alphamikevictor Agustin Munoz added a comment -

          Hello,

          I am on the same situation, I was able to perform a small workaround by setting the environment variable com.sun.jndi.ldap.connect.timeout to 100 on advanced properties, now instead of getting stuck in the login screen for several minutes it only takes a few seconds to authenticate the user.

          plugin: 2.4

          jenkins: 2.46.2

          Show
          alphamikevictor Agustin Munoz added a comment - Hello, I am on the same situation, I was able to perform a small workaround by setting the environment variable com.sun.jndi.ldap.connect.timeout to 100 on advanced properties, now instead of getting stuck in the login screen for several minutes it only takes a few seconds to authenticate the user. plugin: 2.4 jenkins: 2.46.2

            People

            • Assignee:
              fbelzunc FĂ©lix Belzunce Arcos
              Reporter:
              dsakauye Derek Sakauye
            • Votes:
              3 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated: