Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44784

Branch indexing blocked by workspace cleanup hung on bad connection

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Branch indexing was observed to hang in several threads:

      "Executor #-1 for master : executing BranchIndexing[...] / waiting for ... state=TIMED_WAITING ...
          - waiting on <...> (a hudson.remoting.UserRequest)
          - locked <...> (a hudson.remoting.UserRequest)
          at java.lang.Object.wait(Native Method)
          at hudson.remoting.Request.call(Request.java:147)
          at hudson.remoting.Channel.call(Channel.java:829)
          at hudson.FilePath.act(FilePath.java:985)
          at hudson.FilePath.act(FilePath.java:974)
          at hudson.FilePath.isDirectory(FilePath.java:1507)
          at jenkins.branch.WorkspaceLocatorImpl$Deleter.cleanUp(WorkspaceLocatorImpl.java:133)
          at jenkins.branch.WorkspaceLocatorImpl$Deleter.onDeleted(WorkspaceLocatorImpl.java:125)
          at hudson.model.listeners.ItemListener$4.apply(ItemListener.java:205)
          at hudson.model.listeners.ItemListener$4.apply(ItemListener.java:203)
          at hudson.model.listeners.ItemListener.forAll(ItemListener.java:167)
          at hudson.model.listeners.ItemListener.fireOnDeleted(ItemListener.java:203)
          at com.cloudbees.hudson.plugins.folder.AbstractFolder.onDeleted(AbstractFolder.java:1195)
          at com.cloudbees.hudson.plugins.folder.AbstractFolder.onDeleted(AbstractFolder.java:149)
          at hudson.model.AbstractItem.delete(AbstractItem.java:575)
          at hudson.model.Job.delete(Job.java:683)
          at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
          at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:162)
          at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:967)
          at hudson.model.ResourceController.execute(ResourceController.java:98)
          at hudson.model.Executor.run(Executor.java:405)
      

      Apparently the agent connection had gone sour, which may indicate a bug in Remoting. Regardless, I have observed cases where, say, FilePath.isDirectory just hangs indefinitely for no reason I have been able to diagnose or reproduce. The Timeout utility in Pipeline tries to work around this for specific cases, introduced for JENKINS-32986. The fix of JENKINS-34564 should be amended to perform workspace cleanup asynchronously; there is no reason to block the thread calling Job.delete (usually branch indexing; for dead branches, could potentially be an HTTP request).

        Attachments

          Issue Links

            Activity

            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/jenkins/branch/Timeout.java
            src/main/java/jenkins/branch/WorkspaceLocatorImpl.java
            src/test/java/jenkins/branch/WorkspaceLocatorImplTest.java
            http://jenkins-ci.org/commit/branch-api-plugin/bc1fe57c5b955e6aa3b6f9047b43c44d2007165d
            Log:
            [FIXED JENKINS-44784] Ensure workspace cleanup for deleted branch projects proceeds asynchronously, and will time out if the agent is stuck.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/jenkins/branch/Timeout.java src/main/java/jenkins/branch/WorkspaceLocatorImpl.java src/test/java/jenkins/branch/WorkspaceLocatorImplTest.java http://jenkins-ci.org/commit/branch-api-plugin/bc1fe57c5b955e6aa3b6f9047b43c44d2007165d Log: [FIXED JENKINS-44784] Ensure workspace cleanup for deleted branch projects proceeds asynchronously, and will time out if the agent is stuck.
            Hide
            jglick Jesse Glick added a comment -

            Released as 2.0.10. Due to a wiki outage I am unable to update release notes (CC Stephen Connolly).

            Show
            jglick Jesse Glick added a comment - Released as 2.0.10. Due to a wiki outage I am unable to update release notes (CC Stephen Connolly ).

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                jglick Jesse Glick
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: