Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43749

Support multiple Jenkinsfiles from the same repository

    Details

    • Similar Issues:

      Description

      This would support scenarios where different "configurations" of a pipeline cannot share the same Jenkinsfile.

      If I had multiple Jenkinsfiles... repository github.com/apple/swift

      /Package.jenkinsfile 
      /Incremental.jenkinsfile
      /Incremental-RA.jenkinsfile
      /Assert.jenkinsfile
      /src/…
      

      I would like to create multibranch Pipelines for each so I have the resulting structure:

      /Apple
      /Apple/Swift - Package
      /Apple/Swift - Incremental
      /Apple/Swift - Incremental-RA
      /Apple/Swfit - Assert
      

      Note that in this example I have an organization folder for github.com/apple and it is creating multiple multibranch pipelines for each Jenkinsfile discovered in each repository.

      I have written up examples and use cases in this doc

        Attachments

          Issue Links

            Activity

            Hide
            jbennett20912 Jeffrey Bennett added a comment - - edited

            I cannot speak to all the various use-cases that people have brought up, but it seems like a fair-number of people looking for this would be satisfied if multi-branch pipelines just supported an 'exclude regions' flag.  You get that behavior with regular-pipeline, but not multi-branch pipeline.

            In my situation (and seems like others have this as well), I have directoryA/Jenkinsfile and directoryB/Jenkinsfile, and I wish to trigger pipelineA when commits are made to DirectoryA, and pipelineB for DirectoryB.  There are [a few solutions floating around|https://stackoverflow.com/questions/49448029/multiple-jenkinsfile-in-one-repository/60316968] that effectively have both pipelineA and pipelineB starting, doing a 'validate' step, and then pipeline logic to short-circuit, causing completing when the commit is on the other side.  But I'd really prefer that the build NEVER start in the first place for the wrong pipeline.  Commit to DirectoryA -> only PipelineA is triggered.   That amounts to PipelineA configuring "DirectoryB" as an excludes region (and vice versa).

            That said, there are a bunch of use-cases all entwined herein, and this is only one of them.  But... it seems like a fairly well-defined one that might mitigate a lot of people's issues.

             

            Show
            jbennett20912 Jeffrey Bennett added a comment - - edited I cannot speak to all the various use-cases that people have brought up, but it seems like a fair-number of people looking for this would be satisfied if multi-branch pipelines just supported an 'exclude regions' flag.  You get that behavior with regular-pipeline, but not multi-branch pipeline. In my situation (and seems like others have this as well), I have directoryA/Jenkinsfile and directoryB/Jenkinsfile, and I wish to trigger pipelineA when commits are made to DirectoryA, and pipelineB for DirectoryB.  There are [a few solutions floating around| https://stackoverflow.com/questions/49448029/multiple-jenkinsfile-in-one-repository/60316968 ] that effectively have both pipelineA and pipelineB starting, doing a 'validate' step, and then pipeline logic to short-circuit, causing completing when the commit is on the other side.  But I'd really prefer that the build NEVER start in the first place for the wrong pipeline.  Commit to DirectoryA -> only PipelineA is triggered.   That amounts to PipelineA configuring "DirectoryB" as an excludes region (and vice versa). That said, there are a bunch of use-cases all entwined herein, and this is only one of them.  But... it seems like a fairly well-defined one that might mitigate a lot of people's issues.  
            Hide
            ceztko Francesco Pretto added a comment -

            I don't know if this precisely answer the original issue here but I achieved to have multiple Jenkinsfile in the same repository/branch with the current feature set, I explained the procedure in this StackOverflow answer. The main idea is having separate Build and CSM projects configured in Jenkins, with the former that can be a simple Freestyle project with a polling on a repository and the latter a pipeline with a trigger on the CSM project. The trickier part was to add to add the skipDefaultCheckout directive in the Build pipeline, which is strange since I expected "Lightweight checkout" flag in the pipeline project to actually disable the checkout itself. Is this intended?

            Show
            ceztko Francesco Pretto added a comment - I don't know if this precisely answer the original issue here but I achieved to have multiple Jenkinsfile in the same repository/branch with the current feature set, I explained the procedure in this StackOverflow answer . The main idea is having separate Build and CSM projects configured in Jenkins, with the former that can be a simple Freestyle project with a polling on a repository and the latter a pipeline with a trigger on the CSM project. The trickier part was to add to add the skipDefaultCheckout directive in the Build pipeline, which is strange since I expected "Lightweight checkout" flag in the pipeline project to actually disable the checkout itself. Is this intended?
            Hide
            victorpaul victor paul added a comment - - edited

            Fajran Rusadi Is it possible to share the code removing your firms references? Seems this has been the issue with Jenkins since years and there is not a stable solution.

            Show
            victorpaul victor paul added a comment - - edited Fajran Rusadi Is it possible to share the code removing your firms references? Seems this has been the issue with Jenkins since years and there is not a stable solution.
            Hide
            ceztko Francesco Pretto added a comment - - edited

            I'm relatively new to Jenkins but for me the reason this feature can't just be straightforwardly implemented in Jenkins is that it lacks a concept of a first-citizen SCM resource, meaning that SCM repositories can be configured in jobs but they are not treated as shareable resources themselves. If repositories could be configured as shareable resources (with no build steps configurable in them) and job could be configured to source on these resources (instead of configuring a separate git/svn connection for each job) it would be easier to implement a single queue of jobs to trigger for SCM changes on a single repository and it would be easier to filter which pipeline to trigger in case Job1 is configured with Jenkinsfile1 and Job2 is configured with JenkinsFile2. The single queue and the filter can probably be implemented with scripting but to have UI support and more declarative approach in the upstream pipelines (as opposed to imperatively select which job to trigger in the downstream SCM projects) something like what I am suggesting could be needed.

            Show
            ceztko Francesco Pretto added a comment - - edited I'm relatively new to Jenkins but for me the reason this feature can't just be straightforwardly implemented in Jenkins is that it lacks a concept of a first-citizen SCM resource, meaning that SCM repositories can be configured in jobs but they are not treated as shareable resources themselves. If repositories could be configured as shareable resources (with no build steps configurable in them) and job could be configured to source on these resources (instead of configuring a separate git/svn connection for each job) it would be easier to implement a single queue of jobs to trigger for SCM changes on a single repository and it would be easier to filter which pipeline to trigger in case Job1 is configured with Jenkinsfile1 and Job2 is configured with JenkinsFile2. The single queue and the filter can probably be implemented with  scripting  but to have UI support and more declarative approach in the upstream pipelines (as opposed to imperatively select which job to trigger in the downstream SCM projects) something like what I am suggesting could be needed.
            Hide
            fajran Fajran Rusadi added a comment -

            Tzach Yarimi to clarify, we had different multibranch pipeline job for different component. So what service does is create a new multibranch pipeline job when it detects a new component added to the repository. The new job will be configured to use the Jenkinsfile specific of that new component. 

            If the job already exists, it will check if the branch is already registered otherwise it will update the job config to add a new branch and start the scanning process to make the branch job available. Once the branch job is available, the service will trigger the build of that.

            Unfortunately I could not share the code.

            Show
            fajran Fajran Rusadi added a comment - Tzach Yarimi to clarify, we had different multibranch pipeline job for different component. So what service does is create a new multibranch pipeline job when it detects a new component added to the repository. The new job will be configured to use the Jenkinsfile specific of that new component.  If the job already exists, it will check if the branch is already registered otherwise it will update the job config to add a new branch and start the scanning process to make the branch job available. Once the branch job is available, the service will trigger the build of that. Unfortunately I could not share the code.

              People

              • Assignee:
                Unassigned
                Reporter:
                jamesdumay James Dumay
              • Votes:
                92 Vote for this issue
                Watchers:
                113 Start watching this issue

                Dates

                • Created:
                  Updated: