Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22822

Queue.maintain does disk I/O via PeepholePermalink.resolve

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      https://gist.githubusercontent.com/jglick/de0cf09312d8fce00ab1/raw/gistfile1.txt

      Every time Queue.maintain is called, which is often, for every buildable job in the queue, which could be a lot, Job.getEstimatedDurationCandidates is called. During startup this could do some work loading the last few builds, but we expect those to be held in memory thereafter. However simply finding the last successful/any build involves PeepholePermalink.resolve, which hits the disk every time, with no caching. (It claims to "cache" but this is caching via symlink, not in memory.)

      While we are waiting for the disk I/O operation to complete, the Queue is locked, and lots of other threads will typically be waiting for it.

        Attachments

          Activity

          jglick Jesse Glick created issue -
          Hide
          abayer Andrew Bayer added a comment -

          This is fairly nasty - it caused a lot of UI loading problems on builds.apache.org, and high CPU usage on the master with 1.554.1. I'm now running 1.554.1 with a tiny patch to make Job.getEstimatedDurationCandidates always return an empty list, and woof, the performance improvement was dramatic, to say the least. I think this is particularly painful with MavenModuleSet projects, to no real surprise.

          Show
          abayer Andrew Bayer added a comment - This is fairly nasty - it caused a lot of UI loading problems on builds.apache.org, and high CPU usage on the master with 1.554.1. I'm now running 1.554.1 with a tiny patch to make Job.getEstimatedDurationCandidates always return an empty list, and woof, the performance improvement was dramatic, to say the least. I think this is particularly painful with MavenModuleSet projects, to no real surprise.
          abayer Andrew Bayer made changes -
          Field Original Value New Value
          Priority Major [ 3 ] Critical [ 2 ]
          jglick Jesse Glick made changes -
          Assignee Jesse Glick [ jglick ]
          jglick Jesse Glick made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          jglick Jesse Glick made changes -
          Labels lazy-loading performance permalink queue symlink lazy-loading lts-candidate performance permalink queue symlink
          Hide
          jglick Jesse Glick added a comment -

          Mocking out getEstimatedDurationCandidates does not prove that this is the issue—you need to show that the problem is actually symlink resolution—but based on thread dumps I have seen, I am optimistic that avoiding the symlink calls should help.

          @stephenconnolly and I found this issue while reviewing preliminary results of scalability testing on a test Jenkins instance in EC2 with some thousands of jobs constantly running builds.

          Show
          jglick Jesse Glick added a comment - Mocking out getEstimatedDurationCandidates does not prove that this is the issue—you need to show that the problem is actually symlink resolution—but based on thread dumps I have seen, I am optimistic that avoiding the symlink calls should help. @stephenconnolly and I found this issue while reviewing preliminary results of scalability testing on a test Jenkins instance in EC2 with some thousands of jobs constantly running builds.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          core/src/main/java/jenkins/model/PeepholePermalink.java
          core/src/test/java/jenkins/model/PeepholePermalinkTest.java
          http://jenkins-ci.org/commit/jenkins/2ff7a905ac866aac5aea7aae0cb97ee8906b5829
          Log:
          [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/java/jenkins/model/PeepholePermalink.java core/src/test/java/jenkins/model/PeepholePermalinkTest.java http://jenkins-ci.org/commit/jenkins/2ff7a905ac866aac5aea7aae0cb97ee8906b5829 Log: [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O.
          scm_issue_link SCM/JIRA link daemon made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          dogfood dogfood added a comment -

          Integrated in jenkins_main_trunk #3412
          [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O. (Revision 2ff7a905ac866aac5aea7aae0cb97ee8906b5829)

          Result = SUCCESS
          Jesse Glick : 2ff7a905ac866aac5aea7aae0cb97ee8906b5829
          Files :

          • core/src/test/java/jenkins/model/PeepholePermalinkTest.java
          • core/src/main/java/jenkins/model/PeepholePermalink.java
          • changelog.html
          Show
          dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3412 [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O. (Revision 2ff7a905ac866aac5aea7aae0cb97ee8906b5829) Result = SUCCESS Jesse Glick : 2ff7a905ac866aac5aea7aae0cb97ee8906b5829 Files : core/src/test/java/jenkins/model/PeepholePermalinkTest.java core/src/main/java/jenkins/model/PeepholePermalink.java changelog.html
          Hide
          abayer Andrew Bayer added a comment -

          I'll try cherrypicking this onto a build of 1.554.1 and seeing how the results do. Thanks!

          Show
          abayer Andrew Bayer added a comment - I'll try cherrypicking this onto a build of 1.554.1 and seeing how the results do. Thanks!
          Hide
          jglick Jesse Glick added a comment -

          Yes, please do.

          Show
          jglick Jesse Glick added a comment - Yes, please do.
          Hide
          abayer Andrew Bayer added a comment -

          Looking very good so far - queue is only up to 16, so it could get worse later, but no page load slowdown at all currently.

          Show
          abayer Andrew Bayer added a comment - Looking very good so far - queue is only up to 16, so it could get worse later, but no page load slowdown at all currently.
          Hide
          abayer Andrew Bayer added a comment -

          So yeah, big performance improvement on the UI - great fix, jglick! Thanks!

          Show
          abayer Andrew Bayer added a comment - So yeah, big performance improvement on the UI - great fix, jglick! Thanks!
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jesse Glick
          Path:
          core/src/main/java/jenkins/model/PeepholePermalink.java
          core/src/test/java/jenkins/model/PeepholePermalinkTest.java
          http://jenkins-ci.org/commit/jenkins/c74dc66f4e9d954b43b64ff5be83b74ebb0f3a54
          Log:
          [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O.

          (cherry picked from commit 2ff7a905ac866aac5aea7aae0cb97ee8906b5829)

          Conflicts:
          changelog.html

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/jenkins/model/PeepholePermalink.java core/src/test/java/jenkins/model/PeepholePermalinkTest.java http://jenkins-ci.org/commit/jenkins/c74dc66f4e9d954b43b64ff5be83b74ebb0f3a54 Log: [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O. (cherry picked from commit 2ff7a905ac866aac5aea7aae0cb97ee8906b5829) Conflicts: changelog.html
          olivergondza Oliver Gondža made changes -
          Labels lazy-loading lts-candidate performance permalink queue symlink 1.554.3-fixed lazy-loading performance permalink queue symlink
          jglick Jesse Glick made changes -
          Labels 1.554.3-fixed lazy-loading performance permalink queue symlink 1.554.3-fixed lazy-loading lts-candidate performance permalink queue symlink
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jesse Glick
          Path:
          core/src/main/java/jenkins/model/PeepholePermalink.java
          core/src/test/java/jenkins/model/PeepholePermalinkTest.java
          http://jenkins-ci.org/commit/jenkins/2853564b2387a19893975ed6f042c3c25bf5e062
          Log:
          [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O.

          (cherry picked from commit 2ff7a905ac866aac5aea7aae0cb97ee8906b5829)

          Conflicts:
          changelog.html

          Compare: https://github.com/jenkinsci/jenkins/compare/f3848d2d4865...2853564b2387

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/jenkins/model/PeepholePermalink.java core/src/test/java/jenkins/model/PeepholePermalinkTest.java http://jenkins-ci.org/commit/jenkins/2853564b2387a19893975ed6f042c3c25bf5e062 Log: [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O. (cherry picked from commit 2ff7a905ac866aac5aea7aae0cb97ee8906b5829) Conflicts: changelog.html Compare: https://github.com/jenkinsci/jenkins/compare/f3848d2d4865...2853564b2387
          olivergondza Oliver Gondža made changes -
          Labels 1.554.3-fixed lazy-loading lts-candidate performance permalink queue symlink 1.554.3-fixed 1.565.1-fixed lazy-loading performance permalink queue symlink
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 154942 ] JNJira + In-Review [ 195091 ]

            People

            • Assignee:
              jglick Jesse Glick
              Reporter:
              jglick Jesse Glick
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: