Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56863

Cloud stats persistence file grows continually

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • cloud-stats-plugin
    • None
    • Operating System: RHEL 7.4 Linux 64 bit, Kernel 4.4.114-94.11-default
      JDK: OpenJDK 1.8.0_161-b14
      Jenkins version: 2.150.2
      cloud-stats-plugin version: 0.22
    • 2.23

      Hey,

      the cloud-stats-plugin keeps track of every provisioning activity in order to create a statistic that is accessible via <Jenkins-Base-URL>/cloud-stats/. The underlying data is persistet in <Jenkins-Home>/org.jenkinsci.plugins.cloudstats.CloudStatistics.xml. However, there seems to be no rotation mechanism, causing that file to grow continually.

      In our case that file reached 150 MB which caused various issues:

      1. Provisioning time took up to 15 minutes, depending on the number of currently requested agents to provision. There are about 50 short-lived (per build) agents connected at any time.
      2. CPU usage was going through the roof. Jenkins (the master only) is running on a 24 core machine with 128 Gig of RAM and used about 800%-2000% of all CPUs.

      I assume that most of that CPU-time is spent on XML serialization as well as sychronization operations.

      Workaround: Manually delete the persistence file and restart Jenkins. After pruning that persistence file, our CPU-Usage went down to 20%-200%. Provisioning times went down to about 10 seconds, regardless of load.

            olivergondza Oliver Gondža
            seaspu Dennis Keitzel
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: