Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21003

ProjectDiskUsageAction.getBuildsDiskUsageAllSubItems breaks lazy-loading

    Details

    • Similar Issues:

      Description

      A mostly unresponsive 1.509.4 instance shows in its thread dump:

      "Handling GET /plugin/disk-usage/ : http-8088-1" Id=26 Group=main RUNNABLE
      	at java.lang.String.intern(Native Method)
      	at org.xmlpull.mxp1.MXParserCachingStrings.newStringIntern(MXParserCachingStrings.java:317)
      	at org.xmlpull.mxp1.MXParserCachingStrings.newString(MXParserCachingStrings.java:287)
      	at org.xmlpull.mxp1.MXParser.parseStartTag(MXParser.java:1779)
      	at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1127)
      	at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093)
      	at com.thoughtworks.xstream.io.xml.XppReader.pullNextEvent(XppReader.java:109)
      	at com.thoughtworks.xstream.io.xml.AbstractPullReader.readRealEvent(AbstractPullReader.java:148)
      	at com.thoughtworks.xstream.io.xml.AbstractPullReader.readEvent(AbstractPullReader.java:135)
      	at com.thoughtworks.xstream.io.xml.AbstractPullReader.hasMoreChildren(AbstractPullReader.java:87)
      	at com.thoughtworks.xstream.io.ReaderWrapper.hasMoreChildren(ReaderWrapper.java:32)
      	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:261)
      	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:222)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
      	at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85)
      	at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:61)
      	at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      	at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:333)
      	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:275)
      	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:222)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
      	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
      	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1061)
      	at hudson.util.XStream2.unmarshal(XStream2.java:109)
      	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1045)
      	at hudson.XmlFile.unmarshal(XmlFile.java:166)
      	at hudson.model.Run.reload(Run.java:310)
      	at hudson.model.Run.<init>(Run.java:299)
      	at hudson.model.AbstractBuild.<init>(AbstractBuild.java:182)
      	at hudson.maven.AbstractMavenBuild.<init>(AbstractMavenBuild.java:54)
      	at hudson.maven.MavenBuild.<init>(MavenBuild.java:113)
      	at sun.reflect.GeneratedConstructorAccessor2641.newInstance(Unknown Source)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
      	at java.lang.reflect.Constructor.newInstance(Unknown Source)
      	at hudson.model.AbstractProject.loadBuild(AbstractProject.java:1129)
      	at hudson.model.AbstractProject$1.create(AbstractProject.java:325)
      	at hudson.model.AbstractProject$1.create(AbstractProject.java:323)
      	at hudson.model.RunMap.retrieve(RunMap.java:225)
      	at hudson.model.RunMap.retrieve(RunMap.java:59)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674)
      	-  locked hudson.model.RunMap@4e1b74e0
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:636)
      	-  locked hudson.model.RunMap@4e1b74e0
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:368)
      	at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:220)
      	at hudson.tasks.Fingerprinter$FingerprintAction.compact(Fingerprinter.java:356)
      	at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:345)
      	at hudson.model.Run.onLoad(Run.java:323)
      	at hudson.model.RunMap.retrieve(RunMap.java:226)
      	at hudson.model.RunMap.retrieve(RunMap.java:59)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674)
      	-  locked hudson.model.RunMap@4e1b74e0
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:636)
      	-  locked hudson.model.RunMap@4e1b74e0
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:368)
      	at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:220)
      	at hudson.tasks.Fingerprinter$FingerprintAction.compact(Fingerprinter.java:356)
      	at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:345)
      	at hudson.model.Run.onLoad(Run.java:323)
      	at hudson.model.RunMap.retrieve(RunMap.java:226)
      	at hudson.model.RunMap.retrieve(RunMap.java:59)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674)
      	-  locked hudson.model.RunMap@4e1b74e0
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:636)
      	-  locked hudson.model.RunMap@4e1b74e0
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:368)
      	at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:220)
      	at hudson.tasks.Fingerprinter$FingerprintAction.compact(Fingerprinter.java:356)
      	at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:345)
      	at hudson.model.Run.onLoad(Run.java:323)
      	at hudson.model.RunMap.retrieve(RunMap.java:226)
      	at hudson.model.RunMap.retrieve(RunMap.java:59)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674)
      	-  locked hudson.model.RunMap@4e1b74e0
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:636)
      	-  locked hudson.model.RunMap@4e1b74e0
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:368)
      	at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:220)
      	at hudson.tasks.Fingerprinter$FingerprintAction.compact(Fingerprinter.java:356)
      	at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:345)
      	at hudson.model.Run.onLoad(Run.java:323)
      	at hudson.model.RunMap.retrieve(RunMap.java:226)
      	at hudson.model.RunMap.retrieve(RunMap.java:59)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674)
      	-  locked hudson.model.RunMap@4e1b74e0
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.getById(AbstractLazyLoadRunMap.java:529)
      	at jenkins.model.lazy.BuildReferenceMapAdapter.unwrap(BuildReferenceMapAdapter.java:41)
      	at jenkins.model.lazy.BuildReferenceMapAdapter.access$200(BuildReferenceMapAdapter.java:26)
      	at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter._unwrap(BuildReferenceMapAdapter.java:351)
      	at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter.access$400(BuildReferenceMapAdapter.java:243)
      	at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter$1.adapt(BuildReferenceMapAdapter.java:266)
      	at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter$1.adapt(BuildReferenceMapAdapter.java:264)
      	at hudson.util.AdaptedIterator.next(AdaptedIterator.java:54)
      	at com.google.common.collect.Iterators$7.computeNext(Iterators.java:648)
      	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
      	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
      	at java.util.Collections$UnmodifiableCollection$1.hasNext(Unknown Source)
      	at java.util.AbstractMap$2$1.hasNext(Unknown Source)
      	at hudson.plugins.disk_usage.ProjectDiskUsageAction.getBuildsDiskUsageAllSubItems(ProjectDiskUsageAction.java:115)
      	at hudson.plugins.disk_usage.ProjectDiskUsageAction.getBuildsDiskUsage(ProjectDiskUsageAction.java:165)
      
      "Handling GET /plugin/disk-usage/ : http-8088-14" Id=20039 Group=main BLOCKED on hudson.model.RunMap@4e1b74e0 owned by "Handling GET /plugin/disk-usage/ : http-8088-1" Id=26
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674)
      	-  blocked on hudson.model.RunMap@4e1b74e0
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.getById(AbstractLazyLoadRunMap.java:529)
      	at jenkins.model.lazy.BuildReferenceMapAdapter.unwrap(BuildReferenceMapAdapter.java:41)
      	at jenkins.model.lazy.BuildReferenceMapAdapter.access$200(BuildReferenceMapAdapter.java:26)
      	at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter._unwrap(BuildReferenceMapAdapter.java:351)
      	at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter.access$400(BuildReferenceMapAdapter.java:243)
      	at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter$1.adapt(BuildReferenceMapAdapter.java:266)
      	at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter$1.adapt(BuildReferenceMapAdapter.java:264)
      	at hudson.util.AdaptedIterator.next(AdaptedIterator.java:54)
      	at com.google.common.collect.Iterators$7.computeNext(Iterators.java:648)
      	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
      	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
      	at java.util.Collections$UnmodifiableCollection$1.hasNext(Unknown Source)
      	at java.util.AbstractMap$2$1.hasNext(Unknown Source)
      	at hudson.plugins.disk_usage.ProjectDiskUsageAction.getBuildsDiskUsageAllSubItems(ProjectDiskUsageAction.java:115)
      	at hudson.plugins.disk_usage.ProjectDiskUsageAction.getBuildsDiskUsage(ProjectDiskUsageAction.java:165)
      

      (and some more threads waiting for the same lock).

      Iterating all builds of a project, especially from an HTTP handling thread, is unacceptable. If this information can be reasonably cached, then great; or ProgressiveRendering could be used to display disk usage for a particular project when the user is really interested (with the calculation being aborted if and when the user navigates away); otherwise the functionality must be removed entirely.

        Attachments

          Activity

          Hide
          lvotypkova Lucie Votypkova added a comment -

          Moving information into separate xml file diskUsage.xml from build.xml should solve problems with lazy loading. During starting all information will be cashed from diskUsage.xml files of non loaded builds. Recalculation thread will update only diskUsage.xml job and update cashed information from nonloaded builds. I hope tested implementation will be soon.

          Show
          lvotypkova Lucie Votypkova added a comment - Moving information into separate xml file diskUsage.xml from build.xml should solve problems with lazy loading. During starting all information will be cashed from diskUsage.xml files of non loaded builds. Recalculation thread will update only diskUsage.xml job and update cashed information from nonloaded builds. I hope tested implementation will be soon.
          Hide
          jglick Jesse Glick added a comment -

          Why not keep a single file per job listing disk usage for every build? (Possibly in properties format, if the content is sufficiently simple.) This would be a lot faster to load than hundreds of little XML files. Of course you would need to update it when a new build is created; unnecessary to do so when an old build is deleted, since you can just ignore entries for builds whose directories (Run.id) do not still exist.

          Show
          jglick Jesse Glick added a comment - Why not keep a single file per job listing disk usage for every build? (Possibly in properties format, if the content is sufficiently simple.) This would be a lot faster to load than hundreds of little XML files. Of course you would need to update it when a new build is created; unnecessary to do so when an old build is deleted, since you can just ignore entries for builds whose directories ( Run.id ) do not still exist.
          Hide
          lvotypkova Lucie Votypkova added a comment -

          It is easiest solution for backward compatibility (it does not need a lot of changes so there is not a big chance for mistake). It is true that your suggestion is faster for loading, I will check the code how complicated it will be. Thanks!

          Show
          lvotypkova Lucie Votypkova added a comment - It is easiest solution for backward compatibility (it does not need a lot of changes so there is not a big chance for mistake). It is true that your suggestion is faster for loading, I will check the code how complicated it will be. Thanks!
          Hide
          schristou Steven Christou added a comment - - edited

          I created PR 28 which prevents duplicate calls to ProjectDiskUsageAction.getJobRootDirDiskUsage which is an expensive operation. This will not prevent lazy loading operation, however the jobs root disk usage will not be calculated for each build.

          Show
          schristou Steven Christou added a comment - - edited I created PR 28 which prevents duplicate calls to ProjectDiskUsageAction.getJobRootDirDiskUsage which is an expensive operation. This will not prevent lazy loading operation, however the jobs root disk usage will not be calculated for each build.
          Hide
          lvotypkova Lucie Votypkova added a comment -

          Now Disk usage should be able to work without loading builds all the time in the new version 0.26 (will be released this month). Only one exception is the first recalcualtion (which is done by recalculation threads). Once it is done disk usage should not need loading build for its work. Please share if this problem disappears. Thanks

          Show
          lvotypkova Lucie Votypkova added a comment - Now Disk usage should be able to work without loading builds all the time in the new version 0.26 (will be released this month). Only one exception is the first recalcualtion (which is done by recalculation threads). Once it is done disk usage should not need loading build for its work. Please share if this problem disappears. Thanks

            People

            • Assignee:
              lvotypkova Lucie Votypkova
              Reporter:
              jglick Jesse Glick
            • Votes:
              6 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated: