Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47874

Support multiple repositories in multi-branch pipeline

    Details

    • Similar Issues:
    • Sprint:
      Blue Ocean 1.5 - beta 3

      Description

      Within a single job, we need to source branches from multiple repositories that we do not have control over. Currently, if the same branch name occurs in different repositories, only the first is built and subsequent identical branches are dropped.

       

      I suggest that the remote name (if configured) be used to disambiguate the branches, so that they can all be built.

      eg.
      Remote name: linux
      Branch: master

      Remote Name: linux-next
      Branch: master11

      These would be resolved as linux/master and linux-next/master.

       

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment -

          I suspect that changes will be needed in multiple plugins to support your use case.

          I don't think changes are needed in the git plugin, but I would expect changes would be needed in the branch api, and possibly in the branch source plugins and the organization folder plugin.

          Show
          markewaite Mark Waite added a comment - I suspect that changes will be needed in multiple plugins to support your use case. I don't think changes are needed in the git plugin, but I would expect changes would be needed in the branch api, and possibly in the branch source plugins and the organization folder plugin.
          Hide
          povserok Rok Povse added a comment -

          We would really need this issue to be fixed. Could you please provide us with a solution?

          Show
          povserok Rok Povse added a comment - We would really need this issue to be fixed. Could you please provide us with a solution?
          Hide
          jonkins Jon Kelley added a comment -

          Mark Waite

          Here is a use-case for you. See the multi-lined comment areas below. Is there any ideas to achieve something similar??

          We need to run an external pipeline script which lives in a scripts repository, because we don't want to put our shared pipeline scripts in 900+ chef module repositories or use submodules.

          We've hit a brick wall for our use case since updating Job DSL 1.70 to 1.74 since pipelineJob is dead for supporting SCM.

          multibranchPipelineJob is lacking multi git functionality.

          // Generate PR push pipeline jobs for all of the repos passed in via REPO_LISTdef repoList = "${REPO_LIST}".trim().replaceAll('"', '').split(",")
          
          println "Generating PR push jobs for the following repos:: $repoList"
          repoList.each {
              def cookbook = it
              multibranchPipelineJob("chef-${cookbook}-PR-push") {
                  description("Chef push pipeline job")
                  branchSources {
                      github {
                          buildOriginPRHead() // Build fork PRs (unmerged head).
                          // buildOriginPRMerge() // Build fork PRs (merged with base branch).
                          buildOriginBranchWithPR() // Build origin branches also filed as PRs.
                          checkoutCredentialsId('fffff-fffff-fffff-fffff')
                          // scanCredentialsId('github-ci')
                          repoOwner('myOwner')
                          repository("${cookbook}")
                      }
                  }
                  configure {
                      // my Jenkinsfile doesn't exist in the chef coookbooks for 900 repositories...
                      // how can we set a seperate git repo for shared scripts?
                      // https://stackoverflow.com/questions/48284589/jenkins-jobdsl-multibranchpipelinejob-change-script-path
                      it / factory(class: 'org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory') {
                          owner(class: 'org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject', reference: '../..')
                          scriptPath("jenkins/[where ever you want]/Jenkinsfile")
                      }
                  }
              }
           }

           

          pipelineJob worked excellent for our needs, the old broken syntax is below:

          repoList.each {
            def cookbook = it
            pipelineJob("chef-${cookbook}-PR-push") {
              definition {
                cpsScm {
                  lightweight(true)
                  scriptPath('jenkins/[where ever you want]/Jenkinsfile')
                  scm {
                    git {
                      branch('master')
                      remote {
                        github("organization/scripts", 'ssh')
                        credentials('ffff-fff-fff-ffffff)
                      }
                    }
                  }
                }
              }
            }
          }
           
          Show
          jonkins Jon Kelley added a comment - Mark Waite Here is a use-case for you. See the multi-lined comment areas below. Is there any ideas to achieve something similar?? We need to run an external pipeline script which lives in a scripts repository, because we don't want to put our shared pipeline scripts in 900+ chef module repositories or use submodules. We've hit a brick wall for our use case since updating Job DSL 1.70 to 1.74 since  pipelineJob  is dead for supporting SCM. multibranchPipelineJob is lacking multi git functionality. // Generate PR push pipeline jobs for all of the repos passed in via REPO_LISTdef repoList = "${REPO_LIST}" .trim().replaceAll( ' "' , '').split(" ,") println "Generating PR push jobs for the following repos:: $repoList" repoList.each { def cookbook = it multibranchPipelineJob( "chef-${cookbook}-PR-push" ) { description( "Chef push pipeline job" ) branchSources { github { buildOriginPRHead() // Build fork PRs (unmerged head). // buildOriginPRMerge() // Build fork PRs (merged with base branch). buildOriginBranchWithPR() // Build origin branches also filed as PRs. checkoutCredentialsId( 'fffff-fffff-fffff-fffff' ) // scanCredentialsId( 'github-ci' ) repoOwner( 'myOwner' ) repository( "${cookbook}" ) } } configure { // my Jenkinsfile doesn't exist in the chef coookbooks for 900 repositories... // how can we set a seperate git repo for shared scripts? // https://stackoverflow.com/questions/48284589/jenkins-jobdsl-multibranchpipelinejob-change-script-path it / factory(class: 'org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory' ) { owner(class: 'org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject' , reference: '../..' ) scriptPath( "jenkins/[where ever you want]/Jenkinsfile" ) } } } }   pipelineJob worked excellent for our needs, the old broken syntax is below: repoList.each { def cookbook = it pipelineJob( "chef-${cookbook}-PR-push" ) { definition { cpsScm { lightweight( true ) scriptPath( 'jenkins/[where ever you want]/Jenkinsfile' ) scm { git { branch( 'master' ) remote { github( "organization/scripts" , 'ssh' ) credentials('ffff-fff-fff-ffffff) } } } } } } }
          Hide
          ilhicas André Ilhicas Santos added a comment - - edited

          There is actually a TODO for exactly the same use case described in this issue in the responsible getName function for branch name in Branch class at branch-api-plugin

          https://github.com/jenkinsci/branch-api-plugin/blob/1039350970768b27c88bebb0dad6ff9c96e50e0e/src/main/java/jenkins/branch/Branch.java#L143

          What would be the impacts of adding another constructor to allow the remote name to be passed as a prefix for instance?

          Show
          ilhicas André Ilhicas Santos added a comment - - edited There is actually a TODO for exactly the same use case described in this issue in the responsible getName function for branch name in Branch class at branch-api-plugin https://github.com/jenkinsci/branch-api-plugin/blob/1039350970768b27c88bebb0dad6ff9c96e50e0e/src/main/java/jenkins/branch/Branch.java#L143 What would be the impacts of adding another constructor to allow the remote name to be passed as a prefix for instance?
          Hide
          thokari Thomas Hirsch added a comment - - edited

          I am using a multibranch pipeline on several repositories for some time now, and generally it works.
          There is one issue that I have which is that the sidebar-links to the various (GitHub) repositories that are involved are only showing the text "GitHub", and not the specific repository name. This is on the overview page, as well as the "Pull-Requests" tab of the multibranch job.

          Another similar bug occurs on any specific Pull-Requests. The link that is attacked to the specific PR (in this case there is a little "branching" icon, instead of the GitHub icon in the sidebar) is always pointing to the first repository/branch source that is listed in the job configuration, instead of the one that actually triggered the multibranch job.
          This leads to broken links, with the first part of the URL, the repository, getting put together with the path part i.e. PR number from a different repository...

          This is easy to reproduce:
          1. Create a Multibranch PR job with two repositories (default settings should work, as long as PRs are discovered).
          2. Create a PR on the second repository in the list.
          3. Watch the pipeline trigger, and the GitHub link on the job that will be generated for the build will contain the base URL of the first repository, and the PR number of the second repository

          Show
          thokari Thomas Hirsch added a comment - - edited I am using a multibranch pipeline on several repositories for some time now, and generally it works. There is one issue that I have which is that the sidebar-links to the various (GitHub) repositories that are involved are only showing the text "GitHub", and not the specific repository name. This is on the overview page, as well as the "Pull-Requests" tab of the multibranch job. Another similar bug occurs on any specific Pull-Requests. The link that is attacked to the specific PR (in this case there is a little "branching" icon, instead of the GitHub icon in the sidebar) is always pointing to the first repository/branch source that is listed in the job configuration, instead of the one that actually triggered the multibranch job. This leads to broken links, with the first part of the URL, the repository, getting put together with the path part i.e. PR number from a different repository... This is easy to reproduce: 1. Create a Multibranch PR job with two repositories (default settings should work, as long as PRs are discovered). 2. Create a PR on the second repository in the list. 3. Watch the pipeline trigger, and the GitHub link on the job that will be generated for the build will contain the base URL of the first repository, and the PR number of the second repository

            People

            • Assignee:
              Unassigned
              Reporter:
              evildeece Alastair D'Silva
            • Votes:
              17 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated: