Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-32322

multi-branch-project-plugin 0.4.1 creates extreme startup delay

    Details

    • Similar Issues:

      Description

      I just upgraded the Jenkins multibranch-project plugin from 0.3 to 0.4.1. When Jenkins went to restart, it never came up, and didn't say why in the logs. The last line was a statement that plugins had loaded.

      I started a strace of the Java process, and it was scanning basically every single file in $JENKINS_HOME/jobs. In our environment, that directory had at least 1 million files, resulting in an extreme startup delay.

      Since I'm not actually using the plugin, I've disabled and then removed it. I'd imagine a downgrade would have worked, too.

      The most visibly relevant thread from jstack:

      "AbstractMultiBranchProject.migrate" daemon prio=10 tid=0x00007f2010165000 nid=0x7610 runnable [0x00007f20312a7000]
         java.lang.Thread.State: RUNNABLE
      	at java.io.UnixFileSystem.list(Native Method)
      	at java.io.File.list(File.java:1116)
      	at java.io.File.listFiles(File.java:1280)
      	at org.apache.commons.io.FileUtils.innerListFiles(FileUtils.java:478)
      	at org.apache.commons.io.FileUtils.innerListFiles(FileUtils.java:486)
      	at org.apache.commons.io.FileUtils.innerListFiles(FileUtils.java:486)
      	at org.apache.commons.io.FileUtils.innerListFiles(FileUtils.java:486)
      	at org.apache.commons.io.FileUtils.innerListFiles(FileUtils.java:486)
      	at org.apache.commons.io.FileUtils.innerListFiles(FileUtils.java:486)
      	at org.apache.commons.io.FileUtils.innerListFiles(FileUtils.java:486)
      	at org.apache.commons.io.FileUtils.innerListFiles(FileUtils.java:486)
      	at org.apache.commons.io.FileUtils.innerListFiles(FileUtils.java:486)
      	at org.apache.commons.io.FileUtils.innerListFiles(FileUtils.java:486)
      	at org.apache.commons.io.FileUtils.innerListFiles(FileUtils.java:486)
      	at org.apache.commons.io.FileUtils.innerListFiles(FileUtils.java:486)
      	at org.apache.commons.io.FileUtils.innerListFiles(FileUtils.java:486)
      	at org.apache.commons.io.FileUtils.innerListFiles(FileUtils.java:486)
      	at org.apache.commons.io.FileUtils.innerListFiles(FileUtils.java:486)
      	at org.apache.commons.io.FileUtils.innerListFiles(FileUtils.java:486)
      	at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:528)
      	at com.github.mjdetullio.jenkins.plugins.multibranch.AbstractMultiBranchProject.migrate(AbstractMultiBranchProject.java:1126)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:106)
      	at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:176)
      	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
      	at jenkins.model.Jenkins$8.runTask(Jenkins.java:913)
      	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
      	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:744)
      

        Attachments

          Activity

          Hide
          boerh1 Harry de Boer added a comment - - edited

          I found the same stacktrack while debugging slow startup times.
          Actually it starts just fine, only it takes 31 minutes to do so in my case.

          Jenkins version: 1.651.1 (LTS)
          Plugin version: 0.4.1

          Show
          boerh1 Harry de Boer added a comment - - edited I found the same stacktrack while debugging slow startup times. Actually it starts just fine, only it takes 31 minutes to do so in my case. Jenkins version: 1.651.1 (LTS) Plugin version: 0.4.1
          Hide
          evildeece Alastair D'Silva added a comment -

          Pull request #141 addresses this, please review.

          Show
          evildeece Alastair D'Silva added a comment - Pull request #141 addresses this, please review.
          Hide
          boerh1 Harry de Boer added a comment - - edited

          Maybe also exclude 'workspace' and 'workspace@2' etc.
          This is a typical space distribution for one of our branches (although builds is about 2x larger typically):

          669M builds
          8.0K config.xml
          4.0K disk-usage.xml
          0 lastStable
          0 lastSuccessful
          15M modules
          4.0K nextBuildNumber
          4.0K scm-polling.log
          2.4G workspace
          2.4G workspace@2
          5.5G total

          Show
          boerh1 Harry de Boer added a comment - - edited Maybe also exclude 'workspace' and 'workspace@2' etc. This is a typical space distribution for one of our branches (although builds is about 2x larger typically): 669M builds 8.0K config.xml 4.0K disk-usage.xml 0 lastStable 0 lastSuccessful 15M modules 4.0K nextBuildNumber 4.0K scm-polling.log 2.4G workspace 2.4G workspace@2 5.5G total
          Hide
          boerh1 Harry de Boer added a comment -

          You may want to skip lastStable and lastSuccesful too. Not sure how they work on windows, but on linux they are symbolic links into the builds directory.

          lrwxrwxrwx. 1 jenkins jenkins 22 Jun 16 13:27 lastStable -> builds/lastStableBuild
          lrwxrwxrwx. 1 jenkins jenkins 26 Jun 16 13:27 lastSuccessful -> builds/lastSuccessfulBuild

          Other than that, looks good

          Show
          boerh1 Harry de Boer added a comment - You may want to skip lastStable and lastSuccesful too. Not sure how they work on windows, but on linux they are symbolic links into the builds directory. lrwxrwxrwx. 1 jenkins jenkins 22 Jun 16 13:27 lastStable -> builds/lastStableBuild lrwxrwxrwx. 1 jenkins jenkins 26 Jun 16 13:27 lastSuccessful -> builds/lastSuccessfulBuild Other than that, looks good
          Hide
          mjdetullio Matthew DeTullio added a comment -

          Part of 0.5 release

          Show
          mjdetullio Matthew DeTullio added a comment - Part of 0.5 release

            People

            • Assignee:
              mjdetullio Matthew DeTullio
              Reporter:
              danagoyette Dana Goyette
            • Votes:
              4 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: