Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43749

Support multiple Jenkinsfiles from the same repository

    Details

    • Similar Issues:

      Description

      This would support scenarios where different "configurations" of a pipeline cannot share the same Jenkinsfile.

      If I had multiple Jenkinsfiles... repository github.com/apple/swift

      /Package.jenkinsfile 
      /Incremental.jenkinsfile
      /Incremental-RA.jenkinsfile
      /Assert.jenkinsfile
      /src/…
      

      I would like to create multibranch Pipelines for each so I have the resulting structure:

      /Apple
      /Apple/Swift - Package
      /Apple/Swift - Incremental
      /Apple/Swift - Incremental-RA
      /Apple/Swfit - Assert
      

      Note that in this example I have an organization folder for github.com/apple and it is creating multiple multibranch pipelines for each Jenkinsfile discovered in each repository.

      I have written up examples and use cases in this doc

        Attachments

          Issue Links

            Activity

            Hide
            marcelocarlos Marcelo Carlos added a comment - - edited

            Are there any plans to add/support this feature?

            Our scenario is fairly similar to the others. We have a monorepo where we store different projects. To better illustrate our scenario and why this is important to us, here is one simplified use-case:

            We have our repo (let's call it "monorepo"). In this repo we have several folders, such as:

            • toolOne (written in golang)
            • docker-images
              • base
              • imageTwo

            Ideally, we'd have three pipelines in the example above, one for `toolOne`, one for `docker-images/base` and one for `docker-images/imageTwo`.

            On top of that, an interesting additional use case is when we want to build `imageTwo`. That project contains a multi-stage Dockerfile which compiles part of the source code of `toolOne` and adds the compiled binary into the `imageTwo` image (that's one of the main reasons we choose the monorepo approach, this way we can easily and quickly handle cross-project/tools dependencies).

            At the moment, we've been setting all of that up using a single Jenkinsfile at the root of the repo and handling builds with lots of build parameters and conditionals, so we can run/skip stages according to the build we want to execute.

            We looked into the "Job DSL Plugin", but we'd rather avoid since we've been successfully using Jenkinsfile in several of other projects and it is great to the same format for all projects and repos.

            However, the scenario above is gradually becoming a blocker as the pipeline is growing too big due to all the conditionals and stages we had to define to work around the existing limitation of 1 Jenkinsfile per repo.

            So, back to the original question, are there any plans you could share about adding/supporting this feature?

            Show
            marcelocarlos Marcelo Carlos added a comment - - edited Are there any plans to add/support this feature? Our scenario is fairly similar to the others. We have a monorepo where we store different projects. To better illustrate our scenario and why this is important to us, here is one simplified use-case: We have our repo (let's call it "monorepo"). In this repo we have several folders, such as: toolOne (written in golang) docker-images base imageTwo Ideally, we'd have three pipelines in the example above, one for `toolOne`, one for `docker-images/base` and one for `docker-images/imageTwo`. On top of that, an interesting additional use case is when we want to build `imageTwo`. That project contains a multi-stage Dockerfile which compiles part of the source code of `toolOne` and adds the compiled binary into the `imageTwo` image (that's one of the main reasons we choose the monorepo approach, this way we can easily and quickly handle cross-project/tools dependencies). At the moment, we've been setting all of that up using a single Jenkinsfile at the root of the repo and handling builds with lots of build parameters and conditionals, so we can run/skip stages according to the build we want to execute. We looked into the "Job DSL Plugin", but we'd rather avoid since we've been successfully using Jenkinsfile in several of other projects and it is great to the same format for all projects and repos. However, the scenario above is gradually becoming a blocker as the pipeline is growing too big due to all the conditionals and stages we had to define to work around the existing limitation of 1 Jenkinsfile per repo. So, back to the original question, are there any plans you could share about adding/supporting this feature?
            Hide
            fajran Fajran Rusadi added a comment -

            What I ended up doing to support our monorepo approach was the following

            • Create separate (multibranch pipeline) Jenkins job for every individual components in the monorepo
            • All Jenkins jobs will use the same repository URL but each has its own path to Jenkinsfile. This way each component can have its own distinct build pipeline
            • I disabled the build triggering function in Jenkins and instead create an additional service to handle build trigger
            • The build trigger service also manages the Jenkins jobs. Whenever it detects a new component, it will create the corresponding Jenkins job automatically, with the correct configuration
            Show
            fajran Fajran Rusadi added a comment - What I ended up doing to support our monorepo approach was the following Create separate (multibranch pipeline) Jenkins job for every individual components in the monorepo All Jenkins jobs will use the same repository URL but each has its own path to Jenkinsfile. This way each component can have its own distinct build pipeline I disabled the build triggering function in Jenkins and instead create an additional service to handle build trigger The build trigger service also manages the Jenkins jobs. Whenever it detects a new component, it will create the corresponding Jenkins job automatically, with the correct configuration
            Hide
            j3p0uk Jon-Paul Sullivan added a comment -

            When creating multiple jobs off the same GitHub repository, I found we required this plugin to get sane PR checking: https://plugins.jenkins.io/github-scm-trait-notification-context

             

            Default behaviour can play havoc with CI pipelines.

             

            Show
            j3p0uk Jon-Paul Sullivan added a comment - When creating multiple jobs off the same GitHub repository, I found we required this plugin to get sane PR checking: https://plugins.jenkins.io/github-scm-trait-notification-context   Default behaviour can play havoc with CI pipelines.  
            Hide
            tzach Tzach Yarimi added a comment -

            Fajran Rusadi is your service creating jobs inside the multibranch folders? Can you please share some code for doing that?

            Show
            tzach Tzach Yarimi added a comment - Fajran Rusadi is your service creating jobs inside the multibranch folders? Can you please share some code for doing that?
            Hide
            fajran Fajran Rusadi added a comment -

            Tzach Yarimi to clarify, we had different multibranch pipeline job for different component. So what service does is create a new multibranch pipeline job when it detects a new component added to the repository. The new job will be configured to use the Jenkinsfile specific of that new component. 

            If the job already exists, it will check if the branch is already registered otherwise it will update the job config to add a new branch and start the scanning process to make the branch job available. Once the branch job is available, the service will trigger the build of that.

            Unfortunately I could not share the code.

            Show
            fajran Fajran Rusadi added a comment - Tzach Yarimi to clarify, we had different multibranch pipeline job for different component. So what service does is create a new multibranch pipeline job when it detects a new component added to the repository. The new job will be configured to use the Jenkinsfile specific of that new component.  If the job already exists, it will check if the branch is already registered otherwise it will update the job config to add a new branch and start the scanning process to make the branch job available. Once the branch job is available, the service will trigger the build of that. Unfortunately I could not share the code.

              People

              • Assignee:
                Unassigned
                Reporter:
                jamesdumay James Dumay
              • Votes:
                85 Vote for this issue
                Watchers:
                106 Start watching this issue

                Dates

                • Created:
                  Updated: