Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18715

scandir being triggered by FingerPrint.save

    Details

    • Similar Issues:

      Description

      We were experiencing extrem slow fingerprinting in post-build step. It turns out that the include pattern of scm sync configure plugin is triggered by FingerPrint.save() (details see discussion in mailing list[1]). Here is a thread dump:

      Executor #1 for comitdev16 : executing SD-2112 #3" Id=62 Group=main RUNNABLE
      	at java.io.UnixFileSystem.list(Native Method)
      	at java.io.File.list(File.java:973)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1257)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287)
      	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1211)
      	at org.apache.tools.ant.DirectoryScanner.checkIncludePatterns(DirectoryScanner.java:1030)
      	at org.apache.tools.ant.DirectoryScanner.scan(DirectoryScanner.java:909)
      	-  locked org.apache.tools.ant.DirectoryScanner@34cf6c49
      	at hudson.plugins.scm_sync_configuration.strategies.model.PatternsEntityMatcher.matchingFilesFrom(PatternsEntityMatcher.java:41)
      	at hudson.plugins.scm_sync_configuration.strategies.model.PatternsEntityMatcher.matches(PatternsEntityMatcher.java:25)
      	at hudson.plugins.scm_sync_configuration.strategies.AbstractScmSyncStrategy.isSaveableApplicable(AbstractScmSyncStrategy.java:53)
      	at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationPlugin.getStrategyForSaveable(ScmSyncConfigurationPlugin.java:277)
      	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationSaveableListener.onChange(ScmSyncConfigurationSaveableListener.java:22)
      	at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:78)
      	at hudson.model.Fingerprint.save(Fingerprint.java:862)
      	-  locked hudson.model.Fingerprint@281c2370
      	at hudson.model.Fingerprint.<init>(Fingerprint.java:597)
      	at hudson.model.FingerprintMap.create(FingerprintMap.java:90)
      	at hudson.model.FingerprintMap.create(FingerprintMap.java:45)
      	at hudson.util.KeyedDataStorage.get(KeyedDataStorage.java:156)
      	at hudson.model.FingerprintMap.get(FingerprintMap.java:79)
      	at hudson.model.FingerprintMap.get(FingerprintMap.java:45)
      	at hudson.util.KeyedDataStorage.getOrCreate(KeyedDataStorage.java:108)
      	at hudson.model.FingerprintMap.getOrCreate(FingerprintMap.java:65)
      	at hudson.tasks.Fingerprinter$1Record.addRecord(Fingerprinter.java:210)
      	at hudson.tasks.Fingerprinter.record(Fingerprinter.java:254)
      	at hudson.tasks.Fingerprinter.perform(Fingerprinter.java:133)
      	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:776)
      	at hudson.model.Build$BuildExecution.post2(Build.java:183)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:726)
      	at hudson.model.Run.execute(Run.java:1601)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:241)
      

      I think reacting to all SaveableListener.fireOnChange might be too aggressive, since in this paticular case, no configuration changes are involved.

      [1] https://groups.google.com/forum/#!topic/jenkinsci-users/AftANOOYgt8

        Attachments

          Issue Links

            Activity

            Hide
            zhh Hua Zhang added a comment -

            Careful use of the include pattern may avoid huge negative performance impact.

            Show
            zhh Hua Zhang added a comment - Careful use of the include pattern may avoid huge negative performance impact.
            Hide
            zhh Hua Zhang added a comment - - edited

            Well, this behavior is documented in the online help (I didnt read it carefully enough), now I am not sure whether this is a bug or not. But improvement on this spot would be very desirable.

            Show
            zhh Hua Zhang added a comment - - edited Well, this behavior is documented in the online help (I didnt read it carefully enough), now I am not sure whether this is a bug or not. But improvement on this spot would be very desirable.
            Hide
            fcamblor Frédéric Camblor added a comment -

            PR #15 improves the way files are included.

            It may improve the scanning time.

            Could you please have a look at attached hpi file on JENKINS-18526 and see if it solves your problem ?

            Show
            fcamblor Frédéric Camblor added a comment - PR #15 improves the way files are included. It may improve the scanning time. Could you please have a look at attached hpi file on JENKINS-18526 and see if it solves your problem ?
            Hide
            zhh Hua Zhang added a comment -

            The first test of this PR looks good. The fingerprinting was fast enough not giving me enough time to take a thread dump via GUI (next week I will be able to put this on field test).

            Show
            zhh Hua Zhang added a comment - The first test of this PR looks good. The fingerprinting was fast enough not giving me enough time to take a thread dump via GUI (next week I will be able to put this on field test).
            Hide
            fcamblor Frédéric Camblor added a comment -

            Seems good news

            Show
            fcamblor Frédéric Camblor added a comment - Seems good news
            Hide
            zhh Hua Zhang added a comment -

            Seems to me that the PR improves the performance a lot. At least for my situation, I cannot recognise the difference between with and without the include pattern. On the other hand, I also cannot confirm whether scandir still be triggered by the fingerprinting, or it is just fast enough.
            Anyway, the performance problem is gone. Thanks for the quick fix!

            Show
            zhh Hua Zhang added a comment - Seems to me that the PR improves the performance a lot. At least for my situation, I cannot recognise the difference between with and without the include pattern. On the other hand, I also cannot confirm whether scandir still be triggered by the fingerprinting, or it is just fast enough. Anyway, the performance problem is gone. Thanks for the quick fix!
            Hide
            zhh Hua Zhang added a comment -

            Please feel free to close the issue.

            Show
            zhh Hua Zhang added a comment - Please feel free to close the issue.
            Hide
            fcamblor Frédéric Camblor added a comment -

            ok great
            PR will be available in upcoming 0.0.7.2 I'll try to release today

            Show
            fcamblor Frédéric Camblor added a comment - ok great PR will be available in upcoming 0.0.7.2 I'll try to release today

              People

              • Assignee:
                fcamblor Frédéric Camblor
                Reporter:
                zhh Hua Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: