Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37727

Multibranch pipeline project jobs are not deleted when branches are deleted

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: git-plugin
    • Labels:
      None
    • Environment:
      git-plugin compiled from master.
      This bug does not exist in git-plugin 2.5.3
    • Similar Issues:

      Description

      Problem

      If you compile git-plugin from the source, and set 'Discard old items' in the multibranch pipeline, deleted branches on the SCM are not deleted from the job list. If you install the 2.5.3 version of this plugin and restart Jenkins, these branches are deleted.

      How to reproduce it

      Configure a Jenkins installation

      1. Download a fresh Jenkins (2.7.2), rename your current home
      2. Start it: java -jar jenkins.war
      3. Choose the default packages option during the first boot
      4. Clone git-plugin repository, compile it (mvn.cmd '-Dmaven.test.skip=true' clean install hpi:hpi)
      5. Import your git-plugin (the one you compiled) in Jenkins (advanced tab)
      6. Restart Jenkins

      See the error

      1. Create a test git repository with a Jenkinsfile on many branches
      2. Add a new multibranch pipeline, select "Discard old items"
      3. Save, check that your branches appear in the list
      4. Delete a branch from your repository. git push origin :features/a-random-branch
      5. Run a branch indexing
      6. See that your branch was not deleted

      Check that 2.5.3 is working

      1. Restore the git-plugin in its original version, 2.5.3
      2. Restart Jenkins
      3. Rerun branch indexing, see that your branch has disappeared !

      Investigations

      I didn't have the time to run a Jenkins instance in debug, or to read the commits yet. So I've no idea why this bug happens. I'll update this section later if I've time.

        Attachments

          Activity

          superboum Quentin Dufour created issue -
          superboum Quentin Dufour made changes -
          Field Original Value New Value
          Summary Branches of multibranch pipelines are not anymore deleted in upstream Jobs of a multibranch pipeline project are not anymore deleted in upstream
          superboum Quentin Dufour made changes -
          Description h2. Problem

          If you compile git-plugin from the source, and set 'Discard old items' in the multibranch pipeline, deleted branches on the SCM are not deleted from the job list. If you install the 2.5.3 version of this plugin and restart Jenkins, these branches are deleted.

          h2. How to reproduce it

          # Download a fresh Jenkins (2.7.2), rename your current home
          # Start it: java -jar jenkins.war
          # Choose the default packages for the first boot
          # Clone git-plugin repository, compile it (mvn.cmd '-Dmaven.test.skip=true' clean install hpi:hpi)
          # Import your git-plugin (the one you compiled) in Jenkins (advanced tab)
          # Restart Jenkins
          # Create a test git repository with a Jenkinsfile on many branches
          # Add a new multibranch pipeline, select "Discard old items"
          # Save, check that your branches appear in the list
          # Delete a branch from your repository. git push origin :features/a-random-branch
          # Run a branch indexing
          # See that your branch was not deleted
          # Restore the git-plugin in its original version, 2.5.3
          # Restart Jenkins
          # Rerun branch indexing, see that your branch has disappeared !
          h2. Problem

          If you compile git-plugin from the source, and set 'Discard old items' in the multibranch pipeline, deleted branches on the SCM are not deleted from the job list. If you install the 2.5.3 version of this plugin and restart Jenkins, these branches are deleted.

          h2. How to reproduce it

          *Configure a Jenkins installation*

          # Download a fresh Jenkins (2.7.2), rename your current home
          # Start it: java -jar jenkins.war
          # Choose the default packages option during the first boot
          # Clone git-plugin repository, compile it (mvn.cmd '-Dmaven.test.skip=true' clean install hpi:hpi)
          # Import your git-plugin (the one you compiled) in Jenkins (advanced tab)
          # Restart Jenkins

          *See the error*

          # Create a test git repository with a Jenkinsfile on many branches
          # Add a new multibranch pipeline, select "Discard old items"
          # Save, check that your branches appear in the list
          # Delete a branch from your repository. git push origin :features/a-random-branch
          # Run a branch indexing
          # See that your branch was not deleted

          *Check that 2.5.3 is working*

          # Restore the git-plugin in its original version, 2.5.3
          # Restart Jenkins
          # Rerun branch indexing, see that your branch has disappeared !
          superboum Quentin Dufour made changes -
          Description h2. Problem

          If you compile git-plugin from the source, and set 'Discard old items' in the multibranch pipeline, deleted branches on the SCM are not deleted from the job list. If you install the 2.5.3 version of this plugin and restart Jenkins, these branches are deleted.

          h2. How to reproduce it

          *Configure a Jenkins installation*

          # Download a fresh Jenkins (2.7.2), rename your current home
          # Start it: java -jar jenkins.war
          # Choose the default packages option during the first boot
          # Clone git-plugin repository, compile it (mvn.cmd '-Dmaven.test.skip=true' clean install hpi:hpi)
          # Import your git-plugin (the one you compiled) in Jenkins (advanced tab)
          # Restart Jenkins

          *See the error*

          # Create a test git repository with a Jenkinsfile on many branches
          # Add a new multibranch pipeline, select "Discard old items"
          # Save, check that your branches appear in the list
          # Delete a branch from your repository. git push origin :features/a-random-branch
          # Run a branch indexing
          # See that your branch was not deleted

          *Check that 2.5.3 is working*

          # Restore the git-plugin in its original version, 2.5.3
          # Restart Jenkins
          # Rerun branch indexing, see that your branch has disappeared !
          h2. Problem

          If you compile git-plugin from the source, and set 'Discard old items' in the multibranch pipeline, deleted branches on the SCM are not deleted from the job list. If you install the 2.5.3 version of this plugin and restart Jenkins, these branches are deleted.

          h2. How to reproduce it

          *Configure a Jenkins installation*

          # Download a fresh Jenkins (2.7.2), rename your current home
          # Start it: java -jar jenkins.war
          # Choose the default packages option during the first boot
          # Clone git-plugin repository, compile it (mvn.cmd '-Dmaven.test.skip=true' clean install hpi:hpi)
          # Import your git-plugin (the one you compiled) in Jenkins (advanced tab)
          # Restart Jenkins

          *See the error*

          # Create a test git repository with a Jenkinsfile on many branches
          # Add a new multibranch pipeline, select "Discard old items"
          # Save, check that your branches appear in the list
          # Delete a branch from your repository. git push origin :features/a-random-branch
          # Run a branch indexing
          # See that your branch was not deleted

          *Check that 2.5.3 is working*

          # Restore the git-plugin in its original version, 2.5.3
          # Restart Jenkins
          # Rerun branch indexing, see that your branch has disappeared !

          h2. Investigations

          I didn't have the time to run a Jenkins instance in debug, or to read the commits yet. So I've no idea why this bug happens. I'll update this section later if I've time.
          Hide
          markewaite Mark Waite added a comment - - edited

          Thanks for catching that bug before release!

          The master branch adds command line git support for multi-branch pipelines, and makes command line git the default (just as command line git is the default in other cases). Unfortunately, the calling sequence which worked for JGit (fetch() followed by prune()) behaves better in JGit than CliGit. Changing the calling sequence to fetch_().prune().execute() fixes the issue for both JGit and CliGit).

          I was able to duplicate the problem as reported. It is included in my bug verification docker instance and references the JENKINS-37727 branch of my jenkins-bugs repository.

          If I change the job definition from using the command line git implementation to use the jgit implementation, the jobs which correspond to deleted branches are removed, so long as the local branch extension is not enabled (create a branch for each build). If the local branch extension is enabled, or if I switch from jgit to the command line implementation, the jobs which correspond to deleted branches are not removed.

          I suspect there may be two problems here.

          1. If the local branch extension is enabled, I suspect that the cache repository (used to extract the Jenkinsfile) performs a checkout to a named branch matching the target branch. That checkout of a named branch in the cache repository is necessary (to extract the Jenkinsfile), but it does not need to checkout as a named branch. It can use a detached head checkout.
          2. Command line git branch pruning doesn't seem to be working in the cache

          I marked this as critical, since users will not want to modify their multi-branch job definitions to use JGit, especially since there are authentication cases which command git can handle that JGit cannot.

          I've found what appears to be a clean fix for the problem. Unit test and integration test are both available to confirm the problem is fixed. The unit test could use some further work to assure it tests both the CliGit and JGit implementations, but that will need to wait for another time.

          Show
          markewaite Mark Waite added a comment - - edited Thanks for catching that bug before release! The master branch adds command line git support for multi-branch pipelines, and makes command line git the default (just as command line git is the default in other cases). Unfortunately, the calling sequence which worked for JGit (fetch() followed by prune()) behaves better in JGit than CliGit. Changing the calling sequence to fetch_().prune().execute() fixes the issue for both JGit and CliGit). I was able to duplicate the problem as reported. It is included in my bug verification docker instance and references the JENKINS-37727 branch of my jenkins-bugs repository . If I change the job definition from using the command line git implementation to use the jgit implementation, the jobs which correspond to deleted branches are removed, so long as the local branch extension is not enabled (create a branch for each build). If the local branch extension is enabled, or if I switch from jgit to the command line implementation, the jobs which correspond to deleted branches are not removed. I suspect there may be two problems here. If the local branch extension is enabled, I suspect that the cache repository (used to extract the Jenkinsfile) performs a checkout to a named branch matching the target branch. That checkout of a named branch in the cache repository is necessary (to extract the Jenkinsfile), but it does not need to checkout as a named branch. It can use a detached head checkout. Command line git branch pruning doesn't seem to be working in the cache I marked this as critical, since users will not want to modify their multi-branch job definitions to use JGit, especially since there are authentication cases which command git can handle that JGit cannot. I've found what appears to be a clean fix for the problem . Unit test and integration test are both available to confirm the problem is fixed. The unit test could use some further work to assure it tests both the CliGit and JGit implementations, but that will need to wait for another time.
          markewaite Mark Waite made changes -
          Summary Jobs of a multibranch pipeline project are not anymore deleted in upstream Jobs of a multibranch pipeline project are not anymore deleted in upstream if "/" in branch name
          markewaite Mark Waite made changes -
          Summary Jobs of a multibranch pipeline project are not anymore deleted in upstream if "/" in branch name Jobs of a multibranch pipeline project are not deleted when branch is deleted
          markewaite Mark Waite made changes -
          Priority Minor [ 4 ] Critical [ 2 ]
          markewaite Mark Waite made changes -
          Summary Jobs of a multibranch pipeline project are not deleted when branch is deleted Jobs of a multibranch pipeline project are not deleted when branches are deleted
          markewaite Mark Waite made changes -
          Summary Jobs of a multibranch pipeline project are not deleted when branches are deleted Multibranch pipeline project jobs are not deleted when branches are deleted
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Mark Waite
          Path:
          src/main/java/jenkins/plugins/git/AbstractGitSCMSource.java
          src/test/java/jenkins/plugins/git/AbstractGitSCMSourceTest.java
          http://jenkins-ci.org/commit/git-plugin/0197c85c64807fa4f6acdab5c04c02bb36e33c42
          Log:
          Prune during AbstractGitSCMSource fetch

          Need for separate prune is removed by switching from deprecated fetch()
          method to fetch_() method.

          [Fix JENKINS-37727] CliGit leaves deleted branches in multi-branch jobs

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/jenkins/plugins/git/AbstractGitSCMSource.java src/test/java/jenkins/plugins/git/AbstractGitSCMSourceTest.java http://jenkins-ci.org/commit/git-plugin/0197c85c64807fa4f6acdab5c04c02bb36e33c42 Log: Prune during AbstractGitSCMSource fetch Need for separate prune is removed by switching from deprecated fetch() method to fetch_() method. [Fix JENKINS-37727] CliGit leaves deleted branches in multi-branch jobs
          markewaite Mark Waite made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Hide
          markewaite Mark Waite added a comment -

          Included in git plugin 2.6.0, released 2 Sep 2016. Was never visible in any released version of the plugin.

          Show
          markewaite Mark Waite added a comment - Included in git plugin 2.6.0 , released 2 Sep 2016. Was never visible in any released version of the plugin.
          markewaite Mark Waite made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          assafl Leibo added a comment - - edited

          Hi, I just installed 2.7.3 , using Git plugin 2.6.0, configured a multi-branch job, included feature* branches where I have 3 of them.
          3 jobs were created as expected.

          I then deleted one branch but I still see it in the indexing log...
          Seen branch in repository origin/feature/XAVI_MULTI_Demo1
          Seen branch in repository origin/feature/XAVI_MULTI_Demo2
          Seen branch in repository origin/feature/XAVI_MULTI_Demo3

          While on git there's no such branch

          $ git branch -a|grep -i demo
            feature/XAVI_MULTI_Demo1
            feature/XAVI_MULTI_Demo2
            remotes/origin/feature/XAVI_MULTI_Demo1
            remotes/origin/feature/XAVI_MULTI_Demo2
          

          On the master I see the workspace folders

          feature%2FXAVI_MULTI_Demo1@script
          feature%2FXAVI_MULTI_Demo2@script
          feature%2FXAVI_MULTI_Demo3@script
          

          and the deleted branch folder is empty...

          After I removed the git-xxxxx folder from Jenkins caches folder - reindexing was ok.
          Is it related to this issue as well?

          Show
          assafl Leibo added a comment - - edited Hi, I just installed 2.7.3 , using Git plugin 2.6.0, configured a multi-branch job, included feature* branches where I have 3 of them. 3 jobs were created as expected. I then deleted one branch but I still see it in the indexing log... Seen branch in repository origin/feature/XAVI_MULTI_Demo1 Seen branch in repository origin/feature/XAVI_MULTI_Demo2 Seen branch in repository origin/feature/XAVI_MULTI_Demo3 While on git there's no such branch $ git branch -a|grep -i demo feature/XAVI_MULTI_Demo1 feature/XAVI_MULTI_Demo2 remotes/origin/feature/XAVI_MULTI_Demo1 remotes/origin/feature/XAVI_MULTI_Demo2 On the master I see the workspace folders feature%2FXAVI_MULTI_Demo1@script feature%2FXAVI_MULTI_Demo2@script feature%2FXAVI_MULTI_Demo3@script and the deleted branch folder is empty... After I removed the git-xxxxx folder from Jenkins caches folder - reindexing was ok. Is it related to this issue as well?
          Hide
          markewaite Mark Waite added a comment - - edited

          Leibo, that sounds like the same condition which shows the problem (sometimes) in my Docker test instance.

          Could you try switching that specific pipeline job definition from using the "Default" git implementation to using "jgit" as the implementation? Prior to git plugin 2.6.0, the only implementation available to the pipeline branch indexing was the "jgit" implementation and it seemed well behaved unless "checkout to local branch" was specified. The switch from "Default" to "jgit" should be available in the Jenkins job definition with the label "Git executable". Choices in that pick list should be "Default" or "jgit".

          Could you also try without the "Checkout to local branch" extension enabled?

          Removing the git-xxxx folder from the Jenkins cache directory will (at least temporarily) fix the issue, since that will remove the deleted branch from the cache repository. However, it seems likely that the problem will reappear the next time a branch is deleted, since the prune() call in the cached repository seems to not be removing that deleted branch.

          Show
          markewaite Mark Waite added a comment - - edited Leibo , that sounds like the same condition which shows the problem (sometimes) in my Docker test instance. Could you try switching that specific pipeline job definition from using the "Default" git implementation to using "jgit" as the implementation? Prior to git plugin 2.6.0, the only implementation available to the pipeline branch indexing was the "jgit" implementation and it seemed well behaved unless "checkout to local branch" was specified. The switch from "Default" to "jgit" should be available in the Jenkins job definition with the label "Git executable". Choices in that pick list should be "Default" or "jgit". Could you also try without the "Checkout to local branch" extension enabled? Removing the git-xxxx folder from the Jenkins cache directory will (at least temporarily) fix the issue, since that will remove the deleted branch from the cache repository. However, it seems likely that the problem will reappear the next time a branch is deleted, since the prune() call in the cached repository seems to not be removing that deleted branch.
          Hide
          assafl Leibo added a comment -

          I don't see any options to change the definition... for the 'Add source' config item I can only pick from Git, Github, Single repo or subversion

          Show
          assafl Leibo added a comment - I don't see any options to change the definition... for the 'Add source' config item I can only pick from Git, Github, Single repo or subversion
          Hide
          markewaite Mark Waite added a comment - - edited

          If you don't see the phrase "Git executable" in your job definition page (it is inside the Git configuration section of the job definition page), then that means you haven't yet enabled "jgit" as a globally available git implementation in your Jenkins server.

          Add jgit as an available global git implementation in your Jenkins server by clicking "Manage Jenkins", then "Global Tool Configuration". In the "Git" section of that page, click the "Add" button and select "jgit" from the dropdown list.

          Show
          markewaite Mark Waite added a comment - - edited If you don't see the phrase "Git executable" in your job definition page (it is inside the Git configuration section of the job definition page), then that means you haven't yet enabled "jgit" as a globally available git implementation in your Jenkins server. Add jgit as an available global git implementation in your Jenkins server by clicking "Manage Jenkins", then "Global Tool Configuration". In the "Git" section of that page, click the "Add" button and select "jgit" from the dropdown list.
          Hide
          assafl Leibo added a comment - - edited

          Hi,

          Added the JGit as explained and got authentication error... after some issues searching I created a new user based on our SSH private key authentication and it seems working!!!

          So we're good I hope.

          Thanks a lot for the fast response

          Show
          assafl Leibo added a comment - - edited Hi, Added the JGit as explained and got authentication error... after some issues searching I created a new user based on our SSH private key authentication and it seems working!!! So we're good I hope. Thanks a lot for the fast response
          Hide
          markewaite Mark Waite added a comment -

          I'm glad it is working for you, though I'm skeptical that the issue is fully resolved. I don't yet understand the conditions which caused you (and me) to see the issue, while Quentin Dufour and I to not see the issue. I realize it is strange that I've both seen the problem, and not seen the problem, but that just means I don't really understand the circumstances which show the bug.

          Show
          markewaite Mark Waite added a comment - I'm glad it is working for you, though I'm skeptical that the issue is fully resolved. I don't yet understand the conditions which caused you (and me) to see the issue, while Quentin Dufour and I to not see the issue. I realize it is strange that I've both seen the problem, and not seen the problem, but that just means I don't really understand the circumstances which show the bug.
          Hide
          gelefisk Øystein Kjærnet added a comment - - edited

          I experience exactly the same as Leibo - old jobs from deleted branches not being deleted, even though I've selected "Discard old items" in the build config - choosing JGit as suggested by Mark fixed this. Jenkins version 2.20, Git plugin 3.0.0, Git client plugin 2.0.0, Build Pipeline 1.5.4, Pipeline: Multibranch 2.8, git version on system: 1.7.1. I'd be glad to help debugging this in any way...

          Show
          gelefisk Øystein Kjærnet added a comment - - edited I experience exactly the same as Leibo - old jobs from deleted branches not being deleted, even though I've selected "Discard old items" in the build config - choosing JGit as suggested by Mark fixed this. Jenkins version 2.20, Git plugin 3.0.0, Git client plugin 2.0.0, Build Pipeline 1.5.4, Pipeline: Multibranch 2.8, git version on system: 1.7.1. I'd be glad to help debugging this in any way...
          Hide
          markewaite Mark Waite added a comment - - edited

          Øystein Kjærnet can you confirm that in your environment git plugin 2.5.3 does not show the problem, while git plugin 2.6.0 and git plugin 3.0.0 show the problem?

          Can you also explore further to see if you can detect the set of steps which would show the problem on a fresh installation?

          If that works, then could you explore placing that fresh set of steps into a job on a docker instance (preferably as a pull request to the bugs folder of my lts-with-plugins branch)?

          Show
          markewaite Mark Waite added a comment - - edited Øystein Kjærnet can you confirm that in your environment git plugin 2.5.3 does not show the problem, while git plugin 2.6.0 and git plugin 3.0.0 show the problem? Can you also explore further to see if you can detect the set of steps which would show the problem on a fresh installation? If that works, then could you explore placing that fresh set of steps into a job on a docker instance (preferably as a pull request to the bugs folder of my lts-with-plugins branch )?
          Hide
          marius_balteanu Marius Balteanu added a comment - - edited

          I've the same issue and downgrading the Git plugin to 2.5.3 didn't solve the issue. I'll try the jgit workaround.

          • git client plugin 2.1.0
          • git plugin 2.5.3 / 3.0.0
          • pipeline multibranch 2.9.2
          • git version on the server 2.9.2

          Please ignore my above comment because my problem was from the job configuration.

          Show
          marius_balteanu Marius Balteanu added a comment - - edited I've the same issue and downgrading the Git plugin to 2.5.3 didn't solve the issue. I'll try the jgit workaround. git client plugin 2.1.0 git plugin 2.5.3 / 3.0.0 pipeline multibranch 2.9.2 git version on the server 2.9.2 Please ignore my above comment because my problem was from the job configuration.
          markewaite Mark Waite made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              markewaite Mark Waite
              Reporter:
              superboum Quentin Dufour
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: