Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47429

Absolutely atrocious user lookup performance when many legacy users

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Environment:
      core 2.60.x, Git plugin, and user viewing changesets when a lot of users exist.
    • Epic Link:
    • Similar Issues:

      Description

      The performance of user lookup when many users with legacy config files exist has been identified as the cause of a master's CPU usage spiking to the point of unresponsiveness. 

      Stack trace:

      at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
      at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
      at java.io.File.isFile(File.java:882)
      at hudson.model.User$3.accept(User.java:697)
      at java.io.File.listFiles(File.java:1291)
      at hudson.model.User.getLegacyConfigFilesFor(User.java:694)
      at hudson.model.User.getOrCreate(User.java:437)
      at hudson.model.User.getById(User.java:535)
      at hudson.model.User$UserIDCanonicalIdResolver.resolveCanonicalId(User.java:1086)
      at hudson.model.User.get(User.java:405)
      at hudson.model.User.get(User.java:374)
      at hudson.plugins.git.GitChangeSet.findOrCreateUser(GitChangeSet.java:379)
      at hudson.plugins.git.GitChangeSet.getAuthor(GitChangeSet.java:448)

      The root cause of the problem is that for every user we're doing an O(n) scan through the directories in that folder looking for legacy user config files:

       private static final File[] getLegacyConfigFilesFor(final String id) {
              return getRootDir().listFiles(new FileFilter() {
                  @Override
                  public boolean accept(File pathname) {
                      return pathname.isDirectory() && new File(pathname, "config.xml").isFile() && idStrategy().equals(
                              pathname.getName(), id);
                  }
              });
          }
      
      

      Code here: https://github.com/jenkinsci/jenkins/blob/2d2101215dc39dfcb03279e3cb8898b8b9e5bc5f/core/src/main/java/hudson/model/User.java#L680

      The likely best fix is either to be able to check for a single file (if case sensitive mode is on) or to cache the list of entries in that folder so we don't need to do a listFiles and linear-time scan through all entries with a FileFilter.

        Attachments

          Issue Links

            Activity

            svanoort Sam Van Oort created issue -
            svanoort Sam Van Oort made changes -
            Field Original Value New Value
            Environment modern Jenkins core, Git plugin, and user viewing changesets when a lot of users exist. core 2.60.x, Git plugin, and user viewing changesets when a lot of users exist.
            svanoort Sam Van Oort made changes -
            Priority Critical [ 2 ] Major [ 3 ]
            jglick Jesse Glick made changes -
            Component/s blueocean-plugin [ 21481 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-43936 [ JENKINS-43936 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "core PR 2251 (Web Link)" [ 17978 ]
            jamesdumay James Dumay made changes -
            Epic Link JENKINS-37957 [ 174099 ]
            jamesdumay James Dumay made changes -
            Sprint Blue Ocean 1.4 - beta 2 [ 426 ]
            michaelneale Michael Neale made changes -
            Sprint Blue Ocean 1.4 - beta 2 [ 426 ]
            jglick Jesse Glick made changes -
            Component/s blueocean-plugin [ 21481 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-24317 [ JENKINS-24317 ]
            jglick Jesse Glick made changes -
            Assignee Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            jglick Jesse Glick made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal OSS-2527 (Web Link)" [ 18263 ]
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal OSS-2524 (Web Link)" [ 18265 ]
            oleg_nenashev Oleg Nenashev made changes -
            Labels performance lts-candidate performance
            olivergondza Oliver Gondža made changes -
            Description The performance of user lookup when many users with legacy config files exist has been identified as the cause of a master's CPU usage spiking to the point of unresponsiveness. 

            Stack trace:
            {quote}at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
             at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
             at java.io.File.isFile(File.java:882)
             at hudson.model.User$3.accept(User.java:697)
             at java.io.File.listFiles(File.java:1291)
             at hudson.model.User.getLegacyConfigFilesFor(User.java:694)
             at hudson.model.User.getOrCreate(User.java:437)
             at hudson.model.User.getById(User.java:535)
             at hudson.model.User$UserIDCanonicalIdResolver.resolveCanonicalId(User.java:1086)
             at hudson.model.User.get(User.java:405)
             at hudson.model.User.get(User.java:374)
             at hudson.plugins.git.GitChangeSet.findOrCreateUser(GitChangeSet.java:379)
             at hudson.plugins.git.GitChangeSet.getAuthor(GitChangeSet.java:448)
            {quote}
            The root cause of the problem is that for *every* user we're doing an O(n) scan through the directories in that folder looking for legacy user config files:
            {code:java}
             private static final File[] getLegacyConfigFilesFor(final String id) {
                    return getRootDir().listFiles(new FileFilter() {
                        @Override
                        public boolean accept(File pathname) {
                            return pathname.isDirectory() && new File(pathname, "config.xml").isFile() && idStrategy().equals(
                                    pathname.getName(), id);
                        }
                    });
                }

            {code}
            Code here: [https://github.com/jenkinsci/jenkins/blob/2d2101215dc39dfcb03279e3cb8898b8b9e5bc5f/core/src/main/java/hudson/model/User.java#L680]

            The likely best fix is either to be able to check for a *single* file (if case sensitive mode is on) *or* to cache the list of entries in that folder so we don't need to do a listFiles and linear-time scan through all entries with a FileFilter.
            The performance of user lookup when many users with legacy config files exist has been identified as the cause of a master's CPU usage spiking to the point of unresponsiveness. 

            Stack trace:
            {quote}at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
             at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
             at java.io.File.isFile(File.java:882)
             at hudson.model.User$3.accept(User.java:697)
             at java.io.File.listFiles(File.java:1291)
             at hudson.model.User.getLegacyConfigFilesFor(User.java:694)
             at hudson.model.User.getOrCreate(User.java:437)
             at hudson.model.User.getById(User.java:535)
             at hudson.model.User$UserIDCanonicalIdResolver.resolveCanonicalId(User.java:1086)
             at hudson.model.User.get(User.java:405)
             at hudson.model.User.get(User.java:374)
             at hudson.plugins.git.GitChangeSet.findOrCreateUser(GitChangeSet.java:379)
             at hudson.plugins.git.GitChangeSet.getAuthor(GitChangeSet.java:448)
            {quote}
            The root cause of the problem is that for *every* user we're doing an O\(n) scan through the directories in that folder looking for legacy user config files:
            {code:java}
             private static final File[] getLegacyConfigFilesFor(final String id) {
                    return getRootDir().listFiles(new FileFilter() {
                        @Override
                        public boolean accept(File pathname) {
                            return pathname.isDirectory() && new File(pathname, "config.xml").isFile() && idStrategy().equals(
                                    pathname.getName(), id);
                        }
                    });
                }

            {code}
            Code here: [https://github.com/jenkinsci/jenkins/blob/2d2101215dc39dfcb03279e3cb8898b8b9e5bc5f/core/src/main/java/hudson/model/User.java#L680]

            The likely best fix is either to be able to check for a *single* file (if case sensitive mode is on) *or* to cache the list of entries in that folder so we don't need to do a listFiles and linear-time scan through all entries with a FileFilter.
            olivergondza Oliver Gondža made changes -
            Labels lts-candidate performance 2.89.2-fixed performance
            danielbeck Daniel Beck made changes -
            Labels 2.89.2-fixed performance 2.89.3-fixed performance
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is duplicated by JENKINS-23281 [ JENKINS-23281 ]

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                svanoort Sam Van Oort
              • Votes:
                2 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: