Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44751

Multi-Branch Pipeline Not Pruning Branches Deleted from Remote

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I have a multi-branch pipeline job setup in Jenkins set to automatically build open branches. However when I close a branch and delete it from the remote it does not seem to remove the project. I checked the discarded items settings to ensure it was unset, but it seems that Jenkins is picking up the branch even though it was deleted from the remote.

      I configured the orphaned item strategy, no change. I tried to scan the pipeline manually, but the branch was still detected. I added steps to git clean and git prune origin, but nothing is removed. I am currently on the latest version of the plugin: 2.15.

      For example below, the branch feature/INT-542-Sample was deleted from the remote, but it still is being detected as a branch in Jenkins even after pruning. 

      Started by user Dean Pearce
      [Wed Jun 07 13:51:52 EDT 2017] Starting branch indexing...
      > git rev-parse --is-inside-work-tree # timeout=10
      Setting origin to git@bitbucket.org: company-ondemand/project-sync.git
      > git config remote.origin.url git@bitbucket.org: company-ondemand/sf-st-sync.git # timeout=10
      Fetching origin...
      Fetching upstream changes from origin
      > git --version # timeout=10
      using GIT_SSH to set credentials Master SSH Key for Git Creation
      > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/*
      using GIT_SSH to set credentials Master SSH Key for Git Creation
      > git ls-remote git@bitbucket.org: company-ondemand/project-sync.git # timeout=10
      > git rev-parse --is-inside-work-tree # timeout=10
      Setting origin to git@bitbucket.org: company-ondemand/project-sync.git
      > git config remote.origin.url git@bitbucket.org:company-ondemand/project-sync.git # timeout=10
      Fetching & pruning origin...
      Fetching upstream changes from origin
      > git --version # timeout=10
      using GIT_SSH to set credentials Master SSH Key for Git Creation
      > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune
      Getting remote branches...
      Seen branch in repository origin/develop
      Seen branch in repository origin/feature/INT-542
      Seen branch in repository origin/feature/INT-542-Sample
      Seen branch in repository origin/master
      Seen 4 remote branches
      Checking branch feature/INT-542
            ‘Jenkinsfile’ found
      Met criteria
      No changes detected: feature/INT-542 (still at a88426b7ac1dad632b5df7b4e165aafa12bddce7)
      Checking branch develop
            ‘Jenkinsfile’ found
      Met criteria
      No changes detected: develop (still at ec776ac8e6c639ea2b395f92e8aad572945ae270)
      Checking branch master
            ‘Jenkinsfile’ not found
      Does not meet criteria
      Checking branch feature/INT-542-Sample
            ‘Jenkinsfile’ found
      Met criteria
      No changes detected: feature/INT-542-Sample (still at ec776ac8e6c639ea2b395f92e8aad572945ae270)
      Done.
      [Wed Jun 07 13:51:54 EDT 2017] Finished branch indexing. Indexing took 2.2 sec
      Finished: SUCCESS

        Attachments

          Issue Links

            Activity

            deanpearce Dean Pearce created issue -
            deanpearce Dean Pearce made changes -
            Field Original Value New Value
            Description I have a multi-branch pipeline job setup in Jenkins set to automatically build open branches. However when I close a branch and delete it from the remote it does not seem to remove the project. I checked the discarded items settings to ensure it was unset, but it seems that Jenkins is picking up the branch even though it was deleted from the remote.

            I configured the orphaned item strategy, no change. I tried to scan the pipeline manually, but the branch was still detected. I added steps to git clean and git prune origin, but nothing is removed.

            For example below, the branch feature/INT-542-Sample was deleted from the remote, but it still is being detected as a branch in Jenkins even after pruning. 
            {noformat}
            Started by user Dean Pearce
            [Wed Jun 07 13:51:52 EDT 2017] Starting branch indexing...
            > git rev-parse --is-inside-work-tree # timeout=10
            Setting origin to git@bitbucket.org: company-ondemand/project-sync.git
            > git config remote.origin.url git@bitbucket.org: company-ondemand/sf-st-sync.git # timeout=10
            Fetching origin...
            Fetching upstream changes from origin
            > git --version # timeout=10
            using GIT_SSH to set credentials Master SSH Key for Git Creation
            > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/*
            using GIT_SSH to set credentials Master SSH Key for Git Creation
            > git ls-remote git@bitbucket.org: company-ondemand/project-sync.git # timeout=10
            > git rev-parse --is-inside-work-tree # timeout=10
            Setting origin to git@bitbucket.org: company-ondemand/project-sync.git
            > git config remote.origin.url git@bitbucket.org:company-ondemand/project-sync.git # timeout=10
            Fetching & pruning origin...
            Fetching upstream changes from origin
            > git --version # timeout=10
            using GIT_SSH to set credentials Master SSH Key for Git Creation
            > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune
            Getting remote branches...
            Seen branch in repository origin/develop
            Seen branch in repository origin/feature/INT-542
            Seen branch in repository origin/feature/INT-542-Sample
            Seen branch in repository origin/master
            Seen 4 remote branches
            Checking branch feature/INT-542
                  ‘Jenkinsfile’ found
            Met criteria
            No changes detected: feature/INT-542 (still at a88426b7ac1dad632b5df7b4e165aafa12bddce7)
            Checking branch develop
                  ‘Jenkinsfile’ found
            Met criteria
            No changes detected: develop (still at ec776ac8e6c639ea2b395f92e8aad572945ae270)
            Checking branch master
                  ‘Jenkinsfile’ not found
            Does not meet criteria
            Checking branch feature/INT-542-Sample
                  ‘Jenkinsfile’ found
            Met criteria
            No changes detected: feature/INT-542-Sample (still at ec776ac8e6c639ea2b395f92e8aad572945ae270)
            Done.
            [Wed Jun 07 13:51:54 EDT 2017] Finished branch indexing. Indexing took 2.2 sec
            Finished: SUCCESS{noformat}
            I have a multi-branch pipeline job setup in Jenkins set to automatically build open branches. However when I close a branch and delete it from the remote it does not seem to remove the project. I checked the discarded items settings to ensure it was unset, but it seems that Jenkins is picking up the branch even though it was deleted from the remote.

            I configured the orphaned item strategy, no change. I tried to scan the pipeline manually, but the branch was still detected. I added steps to git clean and git prune origin, but nothing is removed. I am currently on the latest version of the plugin: 2.15.

            For example below, the branch feature/INT-542-Sample was deleted from the remote, but it still is being detected as a branch in Jenkins even after pruning. 
            {noformat}
            Started by user Dean Pearce
            [Wed Jun 07 13:51:52 EDT 2017] Starting branch indexing...
            > git rev-parse --is-inside-work-tree # timeout=10
            Setting origin to git@bitbucket.org: company-ondemand/project-sync.git
            > git config remote.origin.url git@bitbucket.org: company-ondemand/sf-st-sync.git # timeout=10
            Fetching origin...
            Fetching upstream changes from origin
            > git --version # timeout=10
            using GIT_SSH to set credentials Master SSH Key for Git Creation
            > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/*
            using GIT_SSH to set credentials Master SSH Key for Git Creation
            > git ls-remote git@bitbucket.org: company-ondemand/project-sync.git # timeout=10
            > git rev-parse --is-inside-work-tree # timeout=10
            Setting origin to git@bitbucket.org: company-ondemand/project-sync.git
            > git config remote.origin.url git@bitbucket.org:company-ondemand/project-sync.git # timeout=10
            Fetching & pruning origin...
            Fetching upstream changes from origin
            > git --version # timeout=10
            using GIT_SSH to set credentials Master SSH Key for Git Creation
            > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune
            Getting remote branches...
            Seen branch in repository origin/develop
            Seen branch in repository origin/feature/INT-542
            Seen branch in repository origin/feature/INT-542-Sample
            Seen branch in repository origin/master
            Seen 4 remote branches
            Checking branch feature/INT-542
                  ‘Jenkinsfile’ found
            Met criteria
            No changes detected: feature/INT-542 (still at a88426b7ac1dad632b5df7b4e165aafa12bddce7)
            Checking branch develop
                  ‘Jenkinsfile’ found
            Met criteria
            No changes detected: develop (still at ec776ac8e6c639ea2b395f92e8aad572945ae270)
            Checking branch master
                  ‘Jenkinsfile’ not found
            Does not meet criteria
            Checking branch feature/INT-542-Sample
                  ‘Jenkinsfile’ found
            Met criteria
            No changes detected: feature/INT-542-Sample (still at ec776ac8e6c639ea2b395f92e8aad572945ae270)
            Done.
            [Wed Jun 07 13:51:54 EDT 2017] Finished branch indexing. Indexing took 2.2 sec
            Finished: SUCCESS{noformat}
            Hide
            gbishop Greg Bishop added a comment -

            I am having this same problem. We recently started using the multibranch pipeline and in the process decided to clean up our repo and deleted 1600+ branches. We're still seeing them all listed whenever the Multibranch Pipeline Scan is run.

            Show
            gbishop Greg Bishop added a comment - I am having this same problem. We recently started using the multibranch pipeline and in the process decided to clean up our repo and deleted 1600+ branches. We're still seeing them all listed whenever the Multibranch Pipeline Scan is run.
            Hide
            jglick Jesse Glick added a comment -

            The bug is presumably in the branch source plugin, so only relevant which branch source you are using (plain Git?) and in which version.

            Show
            jglick Jesse Glick added a comment - The bug is presumably in the branch source plugin, so only relevant which branch source you are using (plain Git?) and in which version.
            jglick Jesse Glick made changes -
            Component/s git-plugin [ 15543 ]
            Component/s workflow-multibranch-plugin [ 21465 ]
            Labels multibranch
            markewaite Mark Waite made changes -
            Component/s bitbucket-branch-source-plugin [ 21428 ]
            Component/s git-plugin [ 15543 ]
            Hide
            gbishop Greg Bishop added a comment -

            I am using plain git, not bitbucket. I can get you the exact version numbers tomorrow, but I know everything was up to date as of Friday. 

            Show
            gbishop Greg Bishop added a comment - I am using plain git, not bitbucket. I can get you the exact version numbers tomorrow, but I know everything was up to date as of Friday. 
            jglick Jesse Glick made changes -
            Component/s git-plugin [ 15543 ]
            Component/s bitbucket-branch-source-plugin [ 21428 ]
            stephenconnolly Stephen Connolly made changes -
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Do the branches go away if you add the Prune Stale Remote Tracking Branches behaviour?

            Show
            stephenconnolly Stephen Connolly added a comment - Do the branches go away if you add the Prune Stale Remote Tracking Branches behaviour?
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Greg Bishop just to clarify, I expect your answer to be "Yes, when I add that behaviour the branches go away"

            In which case the actual bug here is that https://github.com/jenkinsci/git-plugin/blob/ce572609afc6738472b4dba3769e8b950204c2fd/src/main/java/jenkins/plugins/git/AbstractGitSCMSource.java#L411 is looking at the local cache of remote refs and not the result from git ls-remote

            If your answer is "No" then we have more digging to do to identify the root cause.

            Hopefully your answer is "Yes" in which case you would have a workaround to use in the interim

            Show
            stephenconnolly Stephen Connolly added a comment - Greg Bishop just to clarify, I expect your answer to be "Yes, when I add that behaviour the branches go away" In which case the actual bug here is that https://github.com/jenkinsci/git-plugin/blob/ce572609afc6738472b4dba3769e8b950204c2fd/src/main/java/jenkins/plugins/git/AbstractGitSCMSource.java#L411 is looking at the local cache of remote refs and not the result from git ls-remote If your answer is "No" then we have more digging to do to identify the root cause. Hopefully your answer is "Yes" in which case you would have a workaround to use in the interim
            stephenconnolly Stephen Connolly made changes -
            Remote Link This issue links to "PR#514 (Web Link)" [ 17309 ]
            stephenconnolly Stephen Connolly made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            stephenconnolly Stephen Connolly made changes -
            Assignee Stephen Connolly [ stephenconnolly ]
            stephenconnolly Stephen Connolly made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            gbishop Greg Bishop added a comment - - edited

            Unfortunately the answer is no. I do have a workaround though. I created a job that clears the cache every night which forces a fresh clone.

            stage('Remove Jenkins Caches') {
                 dir ("/var/lib/jenkins/caches") {
                        sh 'rm -rf *'
                 }
            }
            

            I will attempt to verify later that I didn't set that setting afterwards. We aren't at the point where we are regularly deleting branches yet. However, I expect a number of branches to be deleted today.

            Show
            gbishop Greg Bishop added a comment - - edited Unfortunately the answer is no. I do have a workaround though. I created a job that clears the cache every night which forces a fresh clone. stage( 'Remove Jenkins Caches' ) { dir ( "/ var /lib/jenkins/caches" ) {             sh 'rm -rf *'      } } I will attempt to verify later that I didn't set that setting afterwards. We aren't at the point where we are regularly deleting branches yet. However, I expect a number of branches to be deleted today.
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Greg Bishop if you are the adventurous type you could build a -SNAPSHOT from the linked PR as that should in theory fix the issue... the tests pass but I haven't even done any manual testing and there has not been code review yet

            Show
            stephenconnolly Stephen Connolly added a comment - Greg Bishop if you are the adventurous type you could build a -SNAPSHOT from the linked PR as that should in theory fix the issue... the tests pass but I haven't even done any manual testing and there has not been code review yet
            Hide
            gbishop Greg Bishop added a comment -

            I'll try to find a few moments today to give it a try Stephen Connolly.

            Show
            gbishop Greg Bishop added a comment - I'll try to find a few moments today to give it a try Stephen Connolly .
            stephenconnolly Stephen Connolly made changes -
            Remote Link This issue links to "Page (Jenkins Wiki)" [ 17357 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Stephen Connolly
            Path:
            src/main/java/org/jenkinsci/plugin/gitea/MergeWithGitSCMExtension.java
            http://jenkins-ci.org/commit/gitea-plugin/527d1b853bd8083983ddbbe1dce07a200a5e936e
            Log:
            JENKINS-44751 Shut-up Findbugs. Oh and remove unused imports.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/org/jenkinsci/plugin/gitea/MergeWithGitSCMExtension.java http://jenkins-ci.org/commit/gitea-plugin/527d1b853bd8083983ddbbe1dce07a200a5e936e Log: JENKINS-44751 Shut-up Findbugs. Oh and remove unused imports.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Stephen Connolly
            Path:
            pom.xml
            http://jenkins-ci.org/commit/gitea-plugin/c7e7e4009ae68ae49c7806c4e231eb509fd10068
            Log:
            [FIXED JENKINS-44751] Pick up release of git plugin

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: pom.xml http://jenkins-ci.org/commit/gitea-plugin/c7e7e4009ae68ae49c7806c4e231eb509fd10068 Log: [FIXED JENKINS-44751] Pick up release of git plugin
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Stephen Connolly
            Path:
            src/main/java/org/jenkinsci/plugins/github_branch_source/MergeWithGitSCMExtension.java
            http://jenkins-ci.org/commit/github-branch-source-plugin/2b0e123f9bdfb4e48e71a830819ee24b090ff937
            Log:
            JENKINS-44751 Shut-up Findbugs. Oh and remove unused imports.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/org/jenkinsci/plugins/github_branch_source/MergeWithGitSCMExtension.java http://jenkins-ci.org/commit/github-branch-source-plugin/2b0e123f9bdfb4e48e71a830819ee24b090ff937 Log: JENKINS-44751 Shut-up Findbugs. Oh and remove unused imports.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Stephen Connolly
            Path:
            pom.xml
            http://jenkins-ci.org/commit/github-branch-source-plugin/95301a4fbb8ed0855f1a72a9b4cdd9713f03c7f8
            Log:
            [FIXED JENKINS-44751] Pick up release of git plugin

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: pom.xml http://jenkins-ci.org/commit/github-branch-source-plugin/95301a4fbb8ed0855f1a72a9b4cdd9713f03c7f8 Log: [FIXED JENKINS-44751] Pick up release of git plugin
            Hide
            markewaite Mark Waite added a comment -

            This change is included in git plugin 3.5.0 and git plugin 3.5.1

            Show
            markewaite Mark Waite added a comment - This change is included in git plugin 3.5.0 and git plugin 3.5.1
            stephenconnolly Stephen Connolly made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]

              People

              • Assignee:
                stephenconnolly Stephen Connolly
                Reporter:
                deanpearce Dean Pearce
              • Votes:
                2 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: