Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37220

Job DSL analogue of multibranch projects & organization folders

    Details

    • Similar Issues:

      Description

      To provide a more scalable alternative to the endless requested branch-api customizations such as JENKINS-32396, or the tricks based on JENKINS-30519, it would be useful to have a simple way of calling into scm-api implementations from a Job DSL folder (JENKINS-33275). So you could (for example) easily construct the equivalent of a GitHub organization folder, but from a script loop that could do arbitrary customizations to each, such as setting job properties.

      Need to also implement SCMSourceOwner, HeadByItem, etc. so that the branch source knows that a given generated project is associated with a given branch, which is needed for various purposes.

      TBD how webhooks should be integrated. The script needs to somehow indicate that changes to a given repository or organization should trigger a regeneration of the folder. May need to create an alternative to SCMTrigger that is keyed off the same criteria that branch-api branch indexing uses, though there are also use cases for traditional SCM polling—it is a little less efficient, but has more configuration options.

      For Pipeline projects you could use Jenkinsfile, if SCMBinder and SCMVar were generalized a bit, or something else.

      Needs a lot of design work; at this point this is more of a placeholder issue to collect ideas.

        Attachments

          Issue Links

            Activity

            Hide
            docwhat Christian Höltje added a comment -

            Use case:

            Joe creates a GitHub Org Folder.  His repos foo and bar are scanned and picked up because they have a Jenkinsfile in each.

            foo is a huge repository (2gb) and he only wants the last 10 commits when building (a shallow clone with depth 10) and without any tags.

            bar uses lots of submodules, but isn't big.  He wants the submodule extensions turned on and to use the GitHub Org credentials.

            This could be done by allow adding extensions at the GitHub Org Folder level, but that would mean forcing all builds to have a depth of 10, no tags, and submodules turned on.  This could cause people to split the GitHub Orgs up just to accommodate Jenkins (which may not be possible, if they are paying for the Orgs, etc.)

            An idea is we could change the scm from an Object to the list form (I don't know what it is called) that looks like [$class:GitSCM, extensions: ... ]

            I have some code I run in a trusted pipeline via the src/ directory that almost does this...

             class ScmEditablizer {
                 static def makeScmEditable(scm, env) {
                    Integer i
            
                    def branches = []
                    for (i = 0; i < scm.branches.size(); i++) {
                        def it = scm.branches.get(i)
                        branches.add([name: it.getName()])
                    }
            
                    def userRemoteConfigs = []
                    for (i = 0; i < scm.userRemoteConfigs.size(); i++) {
                        def it = scm.userRemoteConfigs.get(i)
                        userRemoteConfigs.add([
                        name:          it.getName(),
                        url:           it.getUrl(),
                        credentialsId: it.getCredentialsId(),
                        refspec:       it.getRefspec()
                        ])
                    }
            
                    return [ $class: 'GitSCM', extensions: [], branches: branches, userRemoteConfigs: userRemoteConfigs ]
                }
            }
            

            But this doesn't work because the extensions are missing and it doesn't understand Pull Requests (I don't know why exactly).

            This allows getting an "editable" scm object that can have extensions, etc. added to it.

             

            Show
            docwhat Christian Höltje added a comment - Use case: Joe creates a GitHub Org Folder.  His repos foo and bar are scanned and picked up because they have a Jenkinsfile in each. foo is a huge repository (2gb) and he only wants the last 10 commits when building (a shallow clone with depth 10) and without any tags. bar  uses lots of submodules, but isn't big.  He wants the submodule extensions turned on and to use the GitHub Org credentials. This could be done by allow adding extensions at the GitHub Org Folder level, but that would mean forcing all builds to have a depth of 10, no tags, and submodules turned on.  This could cause people to split the GitHub Orgs up just to accommodate Jenkins (which may not be possible, if they are paying for the Orgs, etc.) An idea is we could change the scm from an Object to the list form (I don't know what it is called) that looks like [$class:GitSCM, extensions: ... ] I have some code I run in a trusted pipeline via the src/ directory that almost  does this...   class ScmEditablizer { static def makeScmEditable(scm, env) { Integer i def branches = [] for (i = 0; i < scm.branches.size(); i++) { def it = scm.branches.get(i) branches.add([name: it.getName()]) } def userRemoteConfigs = [] for (i = 0; i < scm.userRemoteConfigs.size(); i++) { def it = scm.userRemoteConfigs.get(i) userRemoteConfigs.add([ name: it.getName(), url: it.getUrl(), credentialsId: it.getCredentialsId(), refspec: it.getRefspec() ]) } return [ $class: 'GitSCM' , extensions: [], branches: branches, userRemoteConfigs: userRemoteConfigs ] } } But this doesn't work because the extensions are missing and it doesn't understand Pull Requests (I don't know why exactly). This allows getting an "editable" scm object that can have extensions, etc. added to it.  
            Hide
            michaelneale Michael Neale added a comment -

            Is the aim here to "kill job DSL"? 

             

            "have a simple way of calling into scm-api implementations from a Job DSL folder" - interesting use of "simple". I would say "for very advanced users". 

            Show
            michaelneale Michael Neale added a comment - Is the aim here to "kill job DSL"?    "have a simple way of calling into scm-api implementations from a Job DSL folder" - interesting use of "simple". I would say "for very advanced users". 
            Hide
            jglick Jesse Glick added a comment -

            Christian Höltje your use case is solvable in various ways in organization folders as they stand, since you “only” need to adjust the behavior of the build. Now it happens that currently scm is not “editable” but that could be addressed I think. Really you can checkout any GitSCM you like, you just lose the feature of picking up the exact commit (including a PR merge with base branch) that the Jenkinsfile comes from.

            This issue was more about allowing users to programmatically customize things which are inherently aspects of the job, not any individual build. Or even decide when to create a job for a given repo/branch, or create multiple such jobs, etc.

            Michael Neale for years now, plenty of Job DSL users already iterate over the GitHub API (via github-api plugin, or roll-your-own), or something along those lines, so as to get a DIY version of multibranch. This proposal would (if it is feasible) bring some of the polish of branch-api and some of the features of branch source plugins to this usage style, while retaining its flexibility.

            Show
            jglick Jesse Glick added a comment - Christian Höltje your use case is solvable in various ways in organization folders as they stand, since you “only” need to adjust the behavior of the build . Now it happens that currently scm is not “editable” but that could be addressed I think. Really you can checkout any GitSCM you like, you just lose the feature of picking up the exact commit (including a PR merge with base branch) that the Jenkinsfile comes from. This issue was more about allowing users to programmatically customize things which are inherently aspects of the job , not any individual build. Or even decide when to create a job for a given repo/branch, or create multiple such jobs, etc. Michael Neale  for years now, plenty of Job DSL users already iterate over the GitHub API (via github-api plugin, or roll-your-own), or something along those lines, so as to get a DIY version of multibranch. This proposal would (if it is feasible) bring some of the polish of branch-api  and some of the features of branch source plugins to this usage style, while retaining its flexibility.
            Hide
            ccaraivan Costin Caraivan added a comment - - edited

            I don't know if the aim here is to "kill job DSL", but it should be

             

            There's a brand new code-oriented approach to Jenkins, the Jenkins Pipeline. There's a brand spanking new UI for it, Blue Ocean. But we can't really use it. Why? Because some things are not quite pipelines, they're more like Jenkins being used as a fancy cron. We would like to store all those in version control. We would rather not make a new repo for each Jenkinsfile. We would rather not make a new branch for each Jenkinsfile.

            We would prefer to plonk the Jenkinsfiles somewhere in git and have new pipelines be generated for them.

             
            Ideally a team wanting to script Jenkins would use only 1.5 Groovy DSLs instead of the 3 Groovy DSLs (! ) we have to use right now: declarative (+scripting only when needed) vs declarative + scripting + Job DSL...

             

            As a simple solution, maybe it's possible to make the multibranch pipeline accept multiple Jenkinsfiles. Ideally the inclusion would be done with wildcards. That way every job configured just goes *.jenkinsfile and everything automagically works from end-to-end. I have no idea about the full UX impact, though.

            Show
            ccaraivan Costin Caraivan added a comment - - edited I don't know if the aim here is to "kill job DSL", but it should be   There's a brand new code-oriented approach to Jenkins, the Jenkins Pipeline. There's a brand spanking new UI for it, Blue Ocean. But we can't really use it. Why? Because some things are not quite pipelines, they're more like Jenkins being used as a fancy cron. We would like to store all those in version control. We would rather not make a new repo for each Jenkinsfile. We would rather not make a new branch for each Jenkinsfile. We would prefer to plonk the Jenkinsfiles somewhere in git and have new pipelines be generated for them.   Ideally a team wanting to script Jenkins would use only 1.5 Groovy DSLs instead of the 3 Groovy DSLs (! ) we have to use right now: declarative (+scripting only when needed) vs declarative + scripting + Job DSL...   As a simple solution, maybe it's possible to make the multibranch pipeline accept multiple Jenkinsfiles. Ideally the inclusion would be done with wildcards. That way every job configured just goes *.jenkinsfile and everything automagically works from end-to-end. I have no idea about the full UX impact, though.

              People

              • Assignee:
                daspilker Daniel Spilker
                Reporter:
                jglick Jesse Glick
              • Votes:
                9 Vote for this issue
                Watchers:
                24 Start watching this issue

                Dates

                • Created:
                  Updated: