Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61211

QueueItemAuthenticator causes Build Queue contention

    Details

    • Type: Improvement
    • Status: In Review (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: core
    • Labels:
    • Environment:
      core:2.222
    • Similar Issues:

      Description

      When using the Authorize Project plugin to configure authorization for build execution, this has severe consequences on the overall performances of a Jenkins master because it request information repeatedly "trying to get user details for impersonation" from the Security Realm. This can happen during in a thread that hold the Queue.Lock

      An instance will periodically become unavailable or take a very long time in builds especially when the build queue becomes large or when a lot of builds are happening all at the same time.

      If start a fresh instance with 2 master executors, configure a "Project default Build Authorization" under Manage Jenkins > configure Global Security, then create a single pipeline and execute it, you will see through FINE logging of hudson.model.User that the {{User#getUserDetailsForImpersonation} is requested 8 times:

      Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
      Impersonation of the user admin was a success
      Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
      Impersonation of the user admin was a success
      Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
      Impersonation of the user admin was a success
      Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
      Impersonation of the user admin was a success
      Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
      Impersonation of the user admin was a success
      Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
      Impersonation of the user admin was a success
      Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
      Impersonation of the user admin was a success
      Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
      Impersonation of the user admin was a success
      

      Workaround

      Caching can be configured at the Security Realm level depending on the solution (for example for LDAP / Active directory).

      Improvement Proposal

      I do however think that something could be improved and that since impersonation is happening so frequently, maybe the UserDetailsCache can be used to improve performance.

        Attachments

          Issue Links

            Activity

            allan_burdajewicz Allan BURDAJEWICZ created issue -
            allan_burdajewicz Allan BURDAJEWICZ made changes -
            Field Original Value New Value
            Link This issue relates to JENKINS-60111 [ JENKINS-60111 ]
            allan_burdajewicz Allan BURDAJEWICZ made changes -
            Assignee Allan BURDAJEWICZ [ allan_burdajewicz ]
            allan_burdajewicz Allan BURDAJEWICZ made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            allan_burdajewicz Allan BURDAJEWICZ made changes -
            Remote Link This issue links to "core #4523 (Web Link)" [ 24518 ]
            allan_burdajewicz Allan BURDAJEWICZ made changes -
            Description When using the Authorize Project plugin to configure authorization for build execution, this has severe consequences on the overall performances of a Jenkins master because it request information repeatedly "trying to get user details for impersonation" from the Security Realm. This can happen during in a thread that hold the {{Queue.Lock}}

            An instance will periodically become unavailable or take a very long time in builds especially when the build queue becomes large or when a lot of builds are happening all at the same time.

            If start a fresh instance with 2 master executors, configure a "Project default Build Authorization" under **Manage Jenkins > configure Global Security**, then create a single pipeline and execute it, you will see through FINE logging of {{hudson.model.User}} that the {{User#getUserDetailsForImpersonation} is requested 8 times:

            {code}
            eb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            {code}


            ### Workaround

            Caching can be configured at the Security Realm level depending on the solution (for example for LDAP / Active directory).

            ### Improvement Proposal

            I do however think that something could be improved and that since impersonation is happening so frequently, maybe the [UserDetailsCache|https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/jenkins/security/UserDetailsCache.java] can be used to improve performance.
            When using the Authorize Project plugin to configure authorization for build execution, this has severe consequences on the overall performances of a Jenkins master because it request information repeatedly "trying to get user details for impersonation" from the Security Realm. This can happen during in a thread that hold the {{Queue.Lock}}

            An instance will periodically become unavailable or take a very long time in builds especially when the build queue becomes large or when a lot of builds are happening all at the same time.

            If start a fresh instance with 2 master executors, configure a "Project default Build Authorization" under *Manage Jenkins > configure Global Security*, then create a single pipeline and execute it, you will see through FINE logging of {{hudson.model.User}} that the {{User#getUserDetailsForImpersonation} is requested 8 times:

            {code}
            eb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            {code}


            ### Workaround

            Caching can be configured at the Security Realm level depending on the solution (for example for LDAP / Active directory).

            ### Improvement Proposal

            I do however think that something could be improved and that since impersonation is happening so frequently, maybe the [UserDetailsCache|https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/jenkins/security/UserDetailsCache.java] can be used to improve performance.
            allan_burdajewicz Allan BURDAJEWICZ made changes -
            Description When using the Authorize Project plugin to configure authorization for build execution, this has severe consequences on the overall performances of a Jenkins master because it request information repeatedly "trying to get user details for impersonation" from the Security Realm. This can happen during in a thread that hold the {{Queue.Lock}}

            An instance will periodically become unavailable or take a very long time in builds especially when the build queue becomes large or when a lot of builds are happening all at the same time.

            If start a fresh instance with 2 master executors, configure a "Project default Build Authorization" under *Manage Jenkins > configure Global Security*, then create a single pipeline and execute it, you will see through FINE logging of {{hudson.model.User}} that the {{User#getUserDetailsForImpersonation} is requested 8 times:

            {code}
            eb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            {code}


            ### Workaround

            Caching can be configured at the Security Realm level depending on the solution (for example for LDAP / Active directory).

            ### Improvement Proposal

            I do however think that something could be improved and that since impersonation is happening so frequently, maybe the [UserDetailsCache|https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/jenkins/security/UserDetailsCache.java] can be used to improve performance.
            When using the Authorize Project plugin to configure authorization for build execution, this has severe consequences on the overall performances of a Jenkins master because it request information repeatedly "trying to get user details for impersonation" from the Security Realm. This can happen during in a thread that hold the {{Queue.Lock}}

            An instance will periodically become unavailable or take a very long time in builds especially when the build queue becomes large or when a lot of builds are happening all at the same time.

            If start a fresh instance with 2 master executors, configure a "Project default Build Authorization" under *Manage Jenkins > configure Global Security*, then create a single pipeline and execute it, you will see through FINE logging of {{hudson.model.User}} that the {{User#getUserDetailsForImpersonation} is requested 8 times:

            {code}
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            {code}


            ### Workaround

            Caching can be configured at the Security Realm level depending on the solution (for example for LDAP / Active directory).

            ### Improvement Proposal

            I do however think that something could be improved and that since impersonation is happening so frequently, maybe the [UserDetailsCache|https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/jenkins/security/UserDetailsCache.java] can be used to improve performance.
            allan_burdajewicz Allan BURDAJEWICZ made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            allan_burdajewicz Allan BURDAJEWICZ made changes -
            Description When using the Authorize Project plugin to configure authorization for build execution, this has severe consequences on the overall performances of a Jenkins master because it request information repeatedly "trying to get user details for impersonation" from the Security Realm. This can happen during in a thread that hold the {{Queue.Lock}}

            An instance will periodically become unavailable or take a very long time in builds especially when the build queue becomes large or when a lot of builds are happening all at the same time.

            If start a fresh instance with 2 master executors, configure a "Project default Build Authorization" under *Manage Jenkins > configure Global Security*, then create a single pipeline and execute it, you will see through FINE logging of {{hudson.model.User}} that the {{User#getUserDetailsForImpersonation} is requested 8 times:

            {code}
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            {code}


            ### Workaround

            Caching can be configured at the Security Realm level depending on the solution (for example for LDAP / Active directory).

            ### Improvement Proposal

            I do however think that something could be improved and that since impersonation is happening so frequently, maybe the [UserDetailsCache|https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/jenkins/security/UserDetailsCache.java] can be used to improve performance.
            When using the Authorize Project plugin to configure authorization for build execution, this has severe consequences on the overall performances of a Jenkins master because it request information repeatedly "trying to get user details for impersonation" from the Security Realm. This can happen during in a thread that hold the {{Queue.Lock}}

            An instance will periodically become unavailable or take a very long time in builds especially when the build queue becomes large or when a lot of builds are happening all at the same time.

            If start a fresh instance with 2 master executors, configure a "Project default Build Authorization" under *Manage Jenkins > configure Global Security*, then create a single pipeline and execute it, you will see through FINE logging of {{hudson.model.User}} that the {{User#getUserDetailsForImpersonation} is requested 8 times:

            {code}
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
            Impersonation of the user admin was a success
            {code}


            h3. Workaround

            Caching can be configured at the Security Realm level depending on the solution (for example for LDAP / Active directory).

            h3. Improvement Proposal

            I do however think that something could be improved and that since impersonation is happening so frequently, maybe the [UserDetailsCache|https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/jenkins/security/UserDetailsCache.java] can be used to improve performance.

              People

              • Assignee:
                allan_burdajewicz Allan BURDAJEWICZ
                Reporter:
                allan_burdajewicz Allan BURDAJEWICZ
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: