Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9090

Too many directory scans in warnings, analysis-core plugins

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Labels:
      None
    • Environment:
      Linux, 16 core, 12GB RAM
      Hudson 1.395
    • Similar Issues:

      Description

      analysis-core 1.18
      warnings 3.15

      I have a fairly large C/C++ project, which in its current configuration, has about 200k files in the workspace. It takes about 20 minutes to build everything, but an hour and a half to scan the warnings (only ~1000 warnings, GCC4 parser).

      Looking at the threadDumps (one to follow), it's not the log parsing that's taking long, but it's spending enormous time in the DirectoryScanner, kicked off from the ModuleDetector class. An instance of analysis.util.ModuleDetector does two recursive scans: one to look for Ant build files, and one to look for Maven modules. I don't have either setup in a non-Java project and don't really care much about pretty printing modules anyway.

      To make matters worse, the WarningsPublisher class creates two instances of the ModuleDetector - once in WarningsPublisher::perform() and another indirectly through the use of an AnnotationsClassifier object. This results in 4x useless scans, and from what I've read, the Ant DirectoryScanner is slow enough already.

      I suspect there are other recursive scans going on too, as I have seen other stack traces trying to resolve relative paths, etc. These should also be considered for optimization.

      This poor of performance dwarfing my actual build times is making my continuous integration much less "continuous".
      Possible improvements:

      • Make Java module detection a configurable option of the plugin(s)!!!
      • Combine the scans of Ant and Maven files into a single recursive search
      • Cache the results in a ModuleDetector singleton.
      • Make the search path for resolving paths configurable (the warnings for the files I care about are only in a small section of the build tree)

      (Forgive me if I don't have a complete understanding of all the pieces involved... I was just browsing the GitHub repo).

      One snapshot of a stack trace

      "Executor #0 for master : executing DevBuild-CommSW_Jaguar_Dev #163" Id=63 Group=main RUNNABLE
      	at org.apache.tools.ant.util.VectorSet.doAdd(VectorSet.java:64)
      	-  locked org.apache.tools.ant.util.VectorSet@20ef8994
      	at org.apache.tools.ant.util.VectorSet.addElement(VectorSet.java:75)
      	-  locked org.apache.tools.ant.util.VectorSet@20ef8994
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1236)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1184)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1146)
      	at org.apache.tools.ant.DirectoryScanner.checkIncludePatterns(DirectoryScanner.java:928)
      	at org.apache.tools.ant.DirectoryScanner.scan(DirectoryScanner.java:882)
      	-  locked org.apache.tools.ant.DirectoryScanner@cb91d2b
      	at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:490)
      	at hudson.plugins.analysis.util.FileFinder.find(FileFinder.java:76)
      	at hudson.plugins.analysis.util.ModuleDetector.find(ModuleDetector.java:166)
      	at hudson.plugins.analysis.util.ModuleDetector.findAntProjects(ModuleDetector.java:153)
      	at hudson.plugins.analysis.util.ModuleDetector.createFilesToModuleMapping(ModuleDetector.java:103)
      	at hudson.plugins.analysis.util.ModuleDetector.(ModuleDetector.java:69)
      	at hudson.plugins.analysis.util.ModuleDetector.(ModuleDetector.java:56)
      	at hudson.plugins.warnings.WarningsPublisher.perform(WarningsPublisher.java:239)
      	at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:281)
      	at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:27)
      	at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:622)
      	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:601)
      	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:579)
      	at hudson.model.Build$RunnerImpl.post2(Build.java:156)
      	at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:548)
      	at hudson.model.Run.run(Run.java:1386)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:145)
      

        Attachments

          Issue Links

            Activity

            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Ulli Hafner
            Path:
            src/main/java/hudson/plugins/pmd/PmdPublisher.java
            http://jenkins-ci.org/commit/pmd-plugin/25c44c7247932df1f4dff4ebf8c1de839baf2593
            Log:
            JENKINS-9090 Added project configuration option to activate
            scanning of workspace for build.xml and pom.xml files to detect
            the module names of warnings.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Ulli Hafner Path: src/main/java/hudson/plugins/pmd/PmdPublisher.java http://jenkins-ci.org/commit/pmd-plugin/25c44c7247932df1f4dff4ebf8c1de839baf2593 Log: JENKINS-9090 Added project configuration option to activate scanning of workspace for build.xml and pom.xml files to detect the module names of warnings.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Ulli Hafner
            Path:
            src/main/java/hudson/plugins/tasks/TasksPublisher.java
            src/main/java/hudson/plugins/tasks/parser/WorkspaceScanner.java
            http://jenkins-ci.org/commit/tasks-plugin/ebfde8b48b4fb9e1dd550217350df7f522553b29
            Log:
            JENKINS-9090 Added project configuration option to activate
            scanning of workspace for build.xml and pom.xml files to detect
            the module names of warnings.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Ulli Hafner Path: src/main/java/hudson/plugins/tasks/TasksPublisher.java src/main/java/hudson/plugins/tasks/parser/WorkspaceScanner.java http://jenkins-ci.org/commit/tasks-plugin/ebfde8b48b4fb9e1dd550217350df7f522553b29 Log: JENKINS-9090 Added project configuration option to activate scanning of workspace for build.xml and pom.xml files to detect the module names of warnings.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Ulli Hafner
            Path:
            src/main/java/hudson/plugins/warnings/WarningsPublisher.java
            http://jenkins-ci.org/commit/warnings-plugin/5c6329d25f593f167e3d80de6fae7d25e92929d6
            Log:
            JENKINS-9090 Added project configuration option to activate
            scanning of workspace for build.xml and pom.xml files to detect
            the module names of warnings.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Ulli Hafner Path: src/main/java/hudson/plugins/warnings/WarningsPublisher.java http://jenkins-ci.org/commit/warnings-plugin/5c6329d25f593f167e3d80de6fae7d25e92929d6 Log: JENKINS-9090 Added project configuration option to activate scanning of workspace for build.xml and pom.xml files to detect the module names of warnings.
            Hide
            drulli Ulli Hafner added a comment -

            Added a configuration option to enable the scanning of ant/maven modules.

            Show
            drulli Ulli Hafner added a comment - Added a configuration option to enable the scanning of ant/maven modules.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Ulli Hafner
            Path:
            src/main/java/hudson/plugins/analysis/collector/AnalysisPublisher.java
            http://jenkins-ci.org/commit/analysis-collector-plugin/769d1376930c92456eea8c05fe99b83611140bc9
            Log:
            JENKINS-9090 Added project configuration option to activate scanning
            of workspace for build.xml and pom.xml files to detect the module
            names of warnings.)

            Compare: https://github.com/jenkinsci/analysis-collector-plugin/compare/f48b669...769d137

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Ulli Hafner Path: src/main/java/hudson/plugins/analysis/collector/AnalysisPublisher.java http://jenkins-ci.org/commit/analysis-collector-plugin/769d1376930c92456eea8c05fe99b83611140bc9 Log: JENKINS-9090 Added project configuration option to activate scanning of workspace for build.xml and pom.xml files to detect the module names of warnings.) Compare: https://github.com/jenkinsci/analysis-collector-plugin/compare/f48b669...769d137

              People

              • Assignee:
                drulli Ulli Hafner
                Reporter:
                mrmalek Mark Malek
              • Votes:
                3 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: