Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12585

SECURITY: LDAP authenticated users switch accounts randomly

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: security
    • Labels:
      None
    • Environment:

      Description

      Running Jenkins behind Apache: mod_proxy with HTTPS
      https://wiki.jenkins-ci.org/display/JENKINS/Running+Jenkins+behind+Apache
      So our setup is
      Open Directory group
      jenkins-admin - Jenkins Admins all
      dev-group-a - Developers can view kick off builds

      Project-based Matrix Authorization Strategy
      Admin all checked
      dev-group-a checked: Overall:Read Job:Read,Build Run:Update
      dev-group-b checked: Overall:Read Job:Read

      issue is I'm an admin and random developer will login and see that there user id is mine and can admin jenkins.

      there has been reported cases that developer A will login and actually be reported by jenkins as Developer B
      were they can no longer trigger CI builds

      My biggest concern is when users login and are reporting as admins and have full access to jenkins.

        Activity

        Hide
        docwhat Christian Höltje added a comment - - edited

        I have the same issue. One user can hit refresh repeatedly and get different users or even logged out and then back in!

        My setup is:
        Jenkins 1.456
        JRE 7u3
        Plugins of interest: Role-based Authorization Strategy
        Authentication: LDAP

        Show
        docwhat Christian Höltje added a comment - - edited I have the same issue. One user can hit refresh repeatedly and get different users or even logged out and then back in! My setup is: Jenkins 1.456 JRE 7u3 Plugins of interest: Role-based Authorization Strategy Authentication: LDAP
        Hide
        tanmay Tanmay Sinha added a comment -

        +1

        This is causing major issues as users are gaining unauthorized access to controls.

        Show
        tanmay Tanmay Sinha added a comment - +1 This is causing major issues as users are gaining unauthorized access to controls.
        Hide
        docwhat Christian Höltje added a comment -

        A better description of symptoms:

        • I, as a user with admin powers, log in. After a while, I'll notice I'm logged in as normal user "singer". I have to log out and log back in to regain my normal account and admin powers.
        • "bambi" and "singer" can become me by conjuring the new hover thingies and then going to a new page. This includes admin powers.
        • "sniper" can hit refresh repeatedly on a page and become "singer" or logged out. "sniper" is also an admin.

        Note: The usernames have been changed to protect the guilty.

        Show
        docwhat Christian Höltje added a comment - A better description of symptoms: I, as a user with admin powers, log in. After a while, I'll notice I'm logged in as normal user "singer". I have to log out and log back in to regain my normal account and admin powers. "bambi" and "singer" can become me by conjuring the new hover thingies and then going to a new page. This includes admin powers. "sniper" can hit refresh repeatedly on a page and become "singer" or logged out. "sniper" is also an admin. Note: The usernames have been changed to protect the guilty.
        Hide
        docwhat Christian Höltje added a comment -

        Related bug: JENKINS-13203

        This is the symptoms described with "bambi" and "singer" above (number 2).

        Show
        docwhat Christian Höltje added a comment - Related bug: JENKINS-13203 This is the symptoms described with "bambi" and "singer" above (number 2).
        Hide
        docwhat Christian Höltje added a comment -

        Also: I'd be happy to help troubleshoot/debug this. I can get some of the Java experts here to help me look at thread dumps or whatever else might be useful.

        Show
        docwhat Christian Höltje added a comment - Also: I'd be happy to help troubleshoot/debug this. I can get some of the Java experts here to help me look at thread dumps or whatever else might be useful.
        Hide
        kohsuke Kohsuke Kawaguchi added a comment - - edited

        Looking into this.

        For other people seeing this, please report

        • the way you run Jenkins (be it via "java -jar jenkins.war" or on some other servlet containers)
        • version of Jenkins
        • The user realm that you use (does anyone see this with something other than LDAP?)
        Show
        kohsuke Kohsuke Kawaguchi added a comment - - edited Looking into this. For other people seeing this, please report the way you run Jenkins (be it via "java -jar jenkins.war" or on some other servlet containers) version of Jenkins The user realm that you use (does anyone see this with something other than LDAP?)
        Hide
        rpetti Rob Petti added a comment -

        For those who are seeing this issue, are you all running behind a reverse-proxy?

        Show
        rpetti Rob Petti added a comment - For those who are seeing this issue, are you all running behind a reverse-proxy?
        Hide
        docwhat Christian Höltje added a comment -

        For our setup, we use Tomcat. We use LDAP exclusively, so we don't have another realm.

        We are behind an apache reverse proxy.

        I had "sniper" try using :8080 (the non-reverse-proxied url) and he isn't losing his login. This may be because a) he has never had a cookie/session for that URL or b) the proxy is messing things up (but how?).

        PS: I'm 'docwhat' on IRC.

        Show
        docwhat Christian Höltje added a comment - For our setup, we use Tomcat. We use LDAP exclusively, so we don't have another realm. We are behind an apache reverse proxy. I had "sniper" try using :8080 (the non-reverse-proxied url) and he isn't losing his login. This may be because a) he has never had a cookie/session for that URL or b) the proxy is messing things up (but how?). PS: I'm 'docwhat' on IRC.
        Hide
        docwhat Christian Höltje added a comment - - edited

        We had a long troubleshooting session on IRC today and here is our summary (please correct if I make a mistake):

        On my server, we use Tomcat.

        It is behind an apache, running a reverse proxy with mod_cache and mod_*_cache.

        Tomcat sends out JSESSIONID cookie when it thinks a session needs to be set and the previous JSESSIONID is expired or no JSESSIONID is set.

        Our thought is that some cacheable resources had also had the "Set-Cookie: JSESSIONID" header set. So when someone else loaded that cached resource, they had their JSESSIONID set to someone elses.

        This is born out by the fact that adding "CacheIgnoreHeaders Set-Cookie" to apache seems to have fixed the problem.

        Show
        docwhat Christian Höltje added a comment - - edited We had a long troubleshooting session on IRC today and here is our summary (please correct if I make a mistake): On my server, we use Tomcat. It is behind an apache, running a reverse proxy with mod_cache and mod_*_cache. Tomcat sends out JSESSIONID cookie when it thinks a session needs to be set and the previous JSESSIONID is expired or no JSESSIONID is set. Our thought is that some cacheable resources had also had the "Set-Cookie: JSESSIONID" header set. So when someone else loaded that cached resource, they had their JSESSIONID set to someone elses. This is born out by the fact that adding "CacheIgnoreHeaders Set-Cookie" to apache seems to have fixed the problem.
        Hide
        kohsuke Kohsuke Kawaguchi added a comment -

        Thanks to Christian for helping the trouble-shooting session.

        We couldn't exactly verify our hypothesis during the session. The hypothesis hinges upon having Jenkins send "Set-Cookie" header in a static resource that then gets cached by the reverse proxy, yet the code path that serves those files do not touch HTTP sessions before committing the output, and the Acegi's attempt to create a session afterward resulted in no-op, at least with Winstone.

        With that said, toward the end, our discussion was mainly around how to fix this, assuming the hypothesis is correct. Since Jenkins as a webapp doesn't exactly know when the 'Set-Cookie' header is set on the response by the container, the solution has to be done on every single response we produce.

        One thought was to add "Vary: Cookie", and the other was to add "Cache-Control: private". Relevant read here, which also says IE<9 fails to locally cache the resouce if the Vary header is set (and instead if requires conditional GET.)

        While driving after the trouble-shooting session, another reason "Vary: Cookie" header is wrong occured to me, which is that if a request does not contain a cookie header (say it expired) and if the response contains "Set-Cookie", "Vary: Cookie" will do no good as the said resource ends up getting cached by the intermediary.

        So I'm inclined to start serving everything with "Cache-Control: private" unless I hear otherwise.

        Incidentally, http://ci.jenkins-ci.org/ appears to have this header set on the response, although Jenkins doesn't do this. I suspect haproxy or Apache are doing it.

        Show
        kohsuke Kohsuke Kawaguchi added a comment - Thanks to Christian for helping the trouble-shooting session. We couldn't exactly verify our hypothesis during the session. The hypothesis hinges upon having Jenkins send "Set-Cookie" header in a static resource that then gets cached by the reverse proxy, yet the code path that serves those files do not touch HTTP sessions before committing the output, and the Acegi's attempt to create a session afterward resulted in no-op, at least with Winstone. With that said, toward the end, our discussion was mainly around how to fix this, assuming the hypothesis is correct. Since Jenkins as a webapp doesn't exactly know when the 'Set-Cookie' header is set on the response by the container, the solution has to be done on every single response we produce. One thought was to add "Vary: Cookie", and the other was to add "Cache-Control: private". Relevant read here , which also says IE<9 fails to locally cache the resouce if the Vary header is set (and instead if requires conditional GET.) While driving after the trouble-shooting session, another reason "Vary: Cookie" header is wrong occured to me, which is that if a request does not contain a cookie header (say it expired) and if the response contains "Set-Cookie", "Vary: Cookie" will do no good as the said resource ends up getting cached by the intermediary. So I'm inclined to start serving everything with "Cache-Control: private" unless I hear otherwise. Incidentally, http://ci.jenkins-ci.org/ appears to have this header set on the response, although Jenkins doesn't do this. I suspect haproxy or Apache are doing it.
        Hide
        docwhat Christian Höltje added a comment -

        Just a clarification. It doesn't have to be a static asset.

        Tomcat could have (according to the brief description on some stackoverflow article I can't find now) sent it on any page. It doesn't send JSESSIONID on every URL requested, but has some logic for when to do it.

        The only thing needed to poison apache's cache would be that:

        a) A page that is cacheable. This isn't always just pages with an "Expires" or "Last-Modified" header. The rules are... a little esoteric at times, and who knows how apache interprets them.
        b) A page that sent the Set-Cookie header.

        Also, as you mentioned in IRC, there is no point trying to "recover" a poisoned cache. Your only real choice is to turn off the proxy-cache, or reset it with either a fix (like CacheIgnoreHeader Set-Cache) or the new as-yet-unwritten Jenkins.

        Show
        docwhat Christian Höltje added a comment - Just a clarification. It doesn't have to be a static asset. Tomcat could have (according to the brief description on some stackoverflow article I can't find now) sent it on any page. It doesn't send JSESSIONID on every URL requested, but has some logic for when to do it. The only thing needed to poison apache's cache would be that: a) A page that is cacheable. This isn't always just pages with an "Expires" or "Last-Modified" header. The rules are... a little esoteric at times, and who knows how apache interprets them. b) A page that sent the Set-Cookie header. Also, as you mentioned in IRC, there is no point trying to "recover" a poisoned cache. Your only real choice is to turn off the proxy-cache, or reset it with either a fix (like CacheIgnoreHeader Set-Cache) or the new as-yet-unwritten Jenkins.
        Hide
        alexlehm Alex Lehmann added a comment -

        I forgot to mention yesterday, I compared one jenkins instance with logged-in access only to one that has anonymous, both set a cookie on the start page, the logged-in instance has 403 and no no-cache header, the anonymous instance has Cache-Control: no-cache,must-revalidate.

        Both looks correct and the static files do not send a set-cookie header.

        Show
        alexlehm Alex Lehmann added a comment - I forgot to mention yesterday, I compared one jenkins instance with logged-in access only to one that has anonymous, both set a cookie on the start page, the logged-in instance has 403 and no no-cache header, the anonymous instance has Cache-Control: no-cache,must-revalidate. Both looks correct and the static files do not send a set-cookie header.
        Hide
        scm_issue_link SCM/JIRA link daemon added a comment -

        Code changed in jenkins
        User: Kohsuke Kawaguchi
        Path:
        changelog.html
        core/src/main/java/hudson/security/AuthenticationProcessingFilter2.java
        core/src/main/resources/lib/layout/layout.jelly
        war/src/main/webapp/WEB-INF/security/SecurityFilters.groovy
        http://jenkins-ci.org/commit/jenkins/7a4858d65f2431396c2f4dadbc3d654712bc02a8
        Log:
        [FIXED JENKINS-12585] restrict where sessions are created.

        If a resource with 'Set-Cookie' header is cached (either by intermediary
        like HTTP proxy and reverse proxy, or by the browser), it'll cause
        identity swap / session mix-up as discussed in this ticket.

        I suspect this was caused by HttpSessionContextIntegrationFilter2, which
        is the only code path that attempts to create a session when a request
        to a static resource is made.

        So I'm disabling the creation of session in
        HttpSessionContextIntegrationFilter2. This in turn requires that we
        have sessions already created when the authentication was successful and
        people need to login (or else the login will have no effect.)

        We already do so in layout.jelly, so any request that renders a Jenkins
        page would have a session, but I've also added it in
        AuthenticationProcessingFilter2, which ensures that a successful login
        does have a session.

        Show
        scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/src/main/java/hudson/security/AuthenticationProcessingFilter2.java core/src/main/resources/lib/layout/layout.jelly war/src/main/webapp/WEB-INF/security/SecurityFilters.groovy http://jenkins-ci.org/commit/jenkins/7a4858d65f2431396c2f4dadbc3d654712bc02a8 Log: [FIXED JENKINS-12585] restrict where sessions are created. If a resource with 'Set-Cookie' header is cached (either by intermediary like HTTP proxy and reverse proxy, or by the browser), it'll cause identity swap / session mix-up as discussed in this ticket. I suspect this was caused by HttpSessionContextIntegrationFilter2, which is the only code path that attempts to create a session when a request to a static resource is made. So I'm disabling the creation of session in HttpSessionContextIntegrationFilter2. This in turn requires that we have sessions already created when the authentication was successful and people need to login (or else the login will have no effect.) We already do so in layout.jelly, so any request that renders a Jenkins page would have a session, but I've also added it in AuthenticationProcessingFilter2, which ensures that a successful login does have a session.
        Hide
        dogfood dogfood added a comment -

        Integrated in jenkins_main_trunk #1701
        [FIXED JENKINS-12585] restrict where sessions are created. (Revision 7a4858d65f2431396c2f4dadbc3d654712bc02a8)

        Result = UNSTABLE
        Kohsuke Kawaguchi : 7a4858d65f2431396c2f4dadbc3d654712bc02a8
        Files :

        • war/src/main/webapp/WEB-INF/security/SecurityFilters.groovy
        • core/src/main/resources/lib/layout/layout.jelly
        • changelog.html
        • core/src/main/java/hudson/security/AuthenticationProcessingFilter2.java
        Show
        dogfood dogfood added a comment - Integrated in jenkins_main_trunk #1701 [FIXED JENKINS-12585] restrict where sessions are created. (Revision 7a4858d65f2431396c2f4dadbc3d654712bc02a8) Result = UNSTABLE Kohsuke Kawaguchi : 7a4858d65f2431396c2f4dadbc3d654712bc02a8 Files : war/src/main/webapp/WEB-INF/security/SecurityFilters.groovy core/src/main/resources/lib/layout/layout.jelly changelog.html core/src/main/java/hudson/security/AuthenticationProcessingFilter2.java
        Hide
        dogfood dogfood added a comment -

        Integrated in jenkins_ui-changes_branch #30
        [FIXED JENKINS-12585] restrict where sessions are created. (Revision 7a4858d65f2431396c2f4dadbc3d654712bc02a8)

        Result = SUCCESS
        Kohsuke Kawaguchi : 7a4858d65f2431396c2f4dadbc3d654712bc02a8
        Files :

        • changelog.html
        • core/src/main/java/hudson/security/AuthenticationProcessingFilter2.java
        • core/src/main/resources/lib/layout/layout.jelly
        • war/src/main/webapp/WEB-INF/security/SecurityFilters.groovy
        Show
        dogfood dogfood added a comment - Integrated in jenkins_ui-changes_branch #30 [FIXED JENKINS-12585] restrict where sessions are created. (Revision 7a4858d65f2431396c2f4dadbc3d654712bc02a8) Result = SUCCESS Kohsuke Kawaguchi : 7a4858d65f2431396c2f4dadbc3d654712bc02a8 Files : changelog.html core/src/main/java/hudson/security/AuthenticationProcessingFilter2.java core/src/main/resources/lib/layout/layout.jelly war/src/main/webapp/WEB-INF/security/SecurityFilters.groovy

          People

          • Assignee:
            kohsuke Kohsuke Kawaguchi
            Reporter:
            geevez guillermo c
          • Votes:
            10 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: