Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46202

Cannot discover all GitHub repos and their branches with Job DSL anymore

    Details

    • Similar Issues:

      Description

      Using the job DSL plugin v. 1.64, I can't automatically discover all my repositories and all it's branches within my GitHub organization folder.

      I used the job DSL plugin v. 1.61 before.

       

      organizationFolder('myOrg') {
        triggers {
          periodic(10)
        }
        organizations {
          github {
            apiUri("")
            repoOwner("myOrg")
            scanCredentialsId("github")
            checkoutCredentialsId("github")
          }
        }
      }

       

      This is the configuration view after executing the above script


      scanCredentialsId("") and checkoutCredentialsId("") are now deprecated and replaced by credentialsId("github"). Now, the credentials work. But discovering all repositories and their branches doesn't seem to be possible anymore because the "includes", "excludes" methods also have been deprecated.
      Now, with the following, fixed script, I don't get all the branches anymore:

       

      organizationFolder('myOrg') {
        triggers {
          periodic(10)
        }
        organizations {
          github {
            repoOwner("myOrg")
            credentialsId("github")
          }
        }
      }

       

      This is the configuration page after executing the new script:

      I now manually need to add the checkout behaviour "Discover branches -> All branches"

      How can I automatically discover all repos and their branches using the DSL script?

      I also tried the "fromSource" syntax as this one seems to have more options.

       

      organizationFolder('myOrg') {
        triggers {
          periodic(10)
        }
        organizations {
          fromSource {
            name("myOrg")
            sources {
              github {
                credentialsId("github")
                repoOwner("myOrg")
                repository("")
              }
            }
          }
        }
      }
      

       

      As you can see, I have tried to leave the string in the repository("") method empty, hoping that would cause it to discover all repos within my organization. I also tried asterisk: repository("*") which also led to an error.

        Attachments

          Issue Links

            Activity

            Hide
            stephenking Steffen Gebert added a comment -

            Workaround:

            configure {
              def traits = it / sources / data / 'jenkins.branch.BranchSource' / source / traits
              traits << 'org.jenkinsci.plugins.github__branch__source.BranchDiscoveryTrait' {
                strategyId(1)
              }
              traits << 'org.jenkinsci.plugins.github__branch__source.OriginPullRequestDiscoveryTrait' {
               strategyId(1)
             }
            }
            Show
            stephenking Steffen Gebert added a comment - Workaround: configure { def traits = it / sources / data / 'jenkins.branch.BranchSource' / source / traits traits << 'org.jenkinsci.plugins.github__branch__source.BranchDiscoveryTrait' { strategyId(1) } traits << 'org.jenkinsci.plugins.github__branch__source.OriginPullRequestDiscoveryTrait' { strategyId(1) } }
            Hide
            stephenking Steffen Gebert added a comment - - edited

            Auto-generated DSL only shows multibranchPipelineJob -> branchSources -> branchSource -> source -> github -> traits -> originPullRequestDiscoveryTrait, not the other traits.

            Show
            stephenking Steffen Gebert added a comment - - edited Auto-generated DSL only shows multibranchPipelineJob -> branchSources -> branchSource -> source -> github -> traits -> originPullRequestDiscoveryTrait , not the other traits.
            Hide
            daspilker Daniel Spilker added a comment -

            The Automatically Generated DSL does not show traits because the structs plugin does not handle wildcards well in generic type declarations. See JENKINS-26535.

            To reproduce this, run the following script in Script Console:

            import org.jenkinsci.plugins.structs.describable.DescribableModel
            import org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator
            
            new DescribableModel(GitHubSCMNavigator)
            

            The output is:

            Result: GitHubSCMNavigator(repoOwner: String, apiUri?: String, buildForkPRHead?(deprecated): boolean, buildForkPRMerge?(deprecated): boolean, buildOriginBranch?(deprecated): boolean, buildOriginBranchWithPR?(deprecated): boolean, buildOriginPRHead?(deprecated): boolean, buildOriginPRMerge?(deprecated): boolean, credentialsId?: String, excludes?(deprecated): String, includes?(deprecated): String, pattern?(deprecated): String, scanCredentialsId?(deprecated): String, traits?: java.lang.UnsupportedOperationException: do not know how to categorize attributes of type jenkins.scm.api.trait.SCMTrait<? extends jenkins.scm.api.trait.SCMTrait<?>>[])
            
            Show
            daspilker Daniel Spilker added a comment - The Automatically Generated DSL does not show traits because the structs plugin does not handle wildcards well in generic type declarations. See JENKINS-26535 . To reproduce this, run the following script in Script Console: import org.jenkinsci.plugins.structs.describable.DescribableModel import org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator new DescribableModel(GitHubSCMNavigator) The output is: Result: GitHubSCMNavigator(repoOwner: String , apiUri?: String , buildForkPRHead?(deprecated): boolean , buildForkPRMerge?(deprecated): boolean , buildOriginBranch?(deprecated): boolean , buildOriginBranchWithPR?(deprecated): boolean , buildOriginPRHead?(deprecated): boolean , buildOriginPRMerge?(deprecated): boolean , credentialsId?: String , excludes?(deprecated): String , includes?(deprecated): String , pattern?(deprecated): String , scanCredentialsId?(deprecated): String , traits?: java.lang.UnsupportedOperationException: do not know how to categorize attributes of type jenkins.scm.api.trait.SCMTrait<? extends jenkins.scm.api.trait.SCMTrait<?>>[])
            Hide
            stephenking Steffen Gebert added a comment - - edited

            Thanks Daniel for your quick feedback.

            Actually, it is showing the traits section to me, but lacking all but the originPullRequestDiscoveryTrait.

            Maybe it is working and the issue is only my naive try to specify it:

                        source {
                            github {
                                id('my-repo')
                                repoOwner('my-org')
                                repository('my-repo')
                                credentialsId('github-token')
                                apiUri('https://api.github.com')
                            }
                            traits {
                                originPullRequestDiscoveryTrait {
                                    strategyId(1)
                                }
                            }
                        }
            

            This results in

            ERROR: (script, line 24) No signature of method: javaposse.jobdsl.plugin.structs.DescribableListContext.traits() is applicable for argument types: (script$_run_closure1$_closure2$_closure6$_closure7$_closure10) values: [script$_run_closure1$_closure2$_closure6$_closure7$_closure10@d804df7]
            Possible solutions: wait(), wait(long), wait(long, int), split(groovy.lang.Closure), withTraits([Ljava.lang.Class;), is(java.lang.Object)
            

            It's listing a withTraits() method. Can we call this somehow?

            I've attached a config.xml of a manually created multi-branch job (including the <traits> section). multibranch-config-manual.xml

            Show
            stephenking Steffen Gebert added a comment - - edited Thanks Daniel for your quick feedback. Actually, it is showing the traits section to me, but lacking all but the originPullRequestDiscoveryTrait . Maybe it is working and the issue is only my naive try to specify it: source { github { id( 'my-repo' ) repoOwner( 'my-org' ) repository( 'my-repo' ) credentialsId( 'github-token' ) apiUri( 'https: //api.github.com' ) } traits { originPullRequestDiscoveryTrait { strategyId(1) } } } This results in ERROR: (script, line 24) No signature of method: javaposse.jobdsl.plugin.structs.DescribableListContext.traits() is applicable for argument types: (script$_run_closure1$_closure2$_closure6$_closure7$_closure10) values: [script$_run_closure1$_closure2$_closure6$_closure7$_closure10@d804df7] Possible solutions: wait(), wait(long), wait(long, int), split(groovy.lang.Closure), withTraits([Ljava.lang.Class;), is(java.lang.Object) It's listing a withTraits() method. Can we call this somehow? I've attached a config.xml of a manually created multi-branch job (including the <traits>  section).  multibranch-config-manual.xml
            Hide
            daspilker Daniel Spilker added a comment -

            This bug report was originally about the github-organization-folder-plugin but is actually a problem with the branch sources (see below), so it applies to workflow-multibranch as well.

            Steffen Gebert I see a lot of traits in my embedded API viewer (Jenkins 2.19.1, latest version of all plugins), but not the org.jenkinsci.plugins.github_branchsource.BranchDiscoveryTrait and org.jenkinsci.plugins.githubbranch_source.ForkPullRequestDiscoveryTrait traits from your config XML.

            To debug the problem, I ran the following script in Script Console:

            import org.jenkinsci.plugins.structs.describable.DescribableModel
            import org.jenkinsci.plugins.github_branch_source.GitHubSCMSource
            
            new DescribableModel(GitHubSCMSource)
            

            The output is (formatted for readability):

            GitHubSCMSource(
              repoOwner: String, 
              repository: String, 
              apiUri?: String, 
              buildForkPRHead?(deprecated): boolean, 
              buildForkPRMerge?(deprecated): boolean, 
              buildOriginBranch?(deprecated): boolean, 
              buildOriginBranchWithPR?(deprecated): boolean, 
              buildOriginPRHead?(deprecated): boolean, 
              buildOriginPRMerge?(deprecated): boolean, 
              credentialsId?: String, 
              excludes?(deprecated): String, 
              id?: String, 
              includes?(deprecated): String, 
              traits?: SCMSourceTrait{AuthorInChangelogTrait() | 
                       CheckoutOptionTrait(extension: CheckoutOption(timeout: int)) | 
            		   CleanAfterCheckoutTrait() | 
            		   CleanBeforeCheckoutTrait() | 
            		   CloneOptionTrait(extension: CloneOption(shallow: boolean, noTags: boolean, reference: String, timeout: int, depth?: int, honorRefspec?: boolean)) |
            		   ForkPullRequestDiscoveryTrait(strategyId: int, trust: java.lang.UnsupportedOperationException: do not know how to categorize attributes of type jenkins.scm.api.trait.SCMHeadAuthority<? super org.jenkinsci.plugins.github_branch_source.GitHubSCMSourceRequest, ? extends jenkins.scm.api.mixin.ChangeRequestSCMHead2, ? extends jenkins.scm.api.SCMRevision>) | 
            		   GitBrowserSCMSourceTrait(browser: GitRepositoryBrowser{AssemblaWeb(repoUrl: String) | BitbucketWeb(repoUrl: String) | CGit(repoUrl: String) | FisheyeGitRepositoryBrowser(repoUrl: String) | GitBlitRepositoryBrowser(repoUrl: String, projectName: String) | GitLab(repoUrl: String, version: String) | GitList(repoUrl: String) | GitWeb(repoUrl: String) | GithubWeb(repoUrl: String) | Gitiles(repoUrl: String) | GitoriousWeb(repoUrl: String) | GogsGit(repoUrl: String) | KilnGit(repoUrl: String) | Phabricator(repoUrl: String, repo: String) | RedmineWeb(repoUrl: String) | RhodeCode(repoUrl: String) | Stash(repoUrl: String) | TFS2013GitRepositoryBrowser(repoUrl: String) | ViewGitWeb(repoUrl: String, projectName: String)}) |
            		   GitLFSPullTrait() | 
            		   GitToolSCMSourceTrait(gitTool: String) |
            		   IgnoreOnPushNotificationTrait() | 
            		   LocalBranchTrait() |
            		   OriginPullRequestDiscoveryTrait(strategyId: int) | 
            		   PruneStaleBranchTrait() | 
            		   RefSpecsSCMSourceTrait(templates: RefSpecTemplate(value: String)[]) | 
            		   RegexSCMHeadFilterTrait(regex: String) |
            		   RemoteNameSCMSourceTrait(remoteName: String) |
            		   SSHCheckoutTrait(credentialsId: String) | 
            		   SubmoduleOptionTrait(extension: SubmoduleOption(disableSubmodules: boolean, recursiveSubmodules: boolean, trackingSubmodules: boolean, reference: String, timeout: int, parentCredentials: boolean)) | 
            		   UserIdentityTrait(extension: UserIdentity(name: String, email: String)) | 
            		   WildcardSCMHeadFilterTrait(includes: String, excludes: String) | 
            		   WipeWorkspaceTrait() |
            		   jenkins.plugins.git.traits.BranchDiscoveryTrait~BranchDiscoveryTrait() | 
            		   org.jenkinsci.plugins.github_branch_source.BranchDiscoveryTrait~BranchDiscoveryTrait(strategyId: int)}[])
            

            As one can see, the problem with ForkPullRequestDiscoveryTrait is that the trust property is not supported by structs-plugin, probably because of JENKINS-26535.

            The BranchDiscoveryTrait is not available because there are two classes named BranchDiscoveryTrait in different packages (jenkins.plugins.git.traits and org.jenkinsci.plugins.github_branch_source). Job DSL only uses the class name to determine the class and for BranchDiscoveryTrait that's ambigious, so it's not available. github-branch-source-plugin and/or git-plugin can use a @Symbol annotation to set an unambigious alias.

            IMHO this needs to fixed in github-branch-source-plugin, git-plugin and/or structs-plugin. Pinging Stephen Connolly, Mark Waite and Jesse Glick.

            Show
            daspilker Daniel Spilker added a comment - This bug report was originally about the github-organization-folder-plugin but is actually a problem with the branch sources (see below), so it applies to workflow-multibranch as well. Steffen Gebert I see a lot of traits in my embedded API viewer (Jenkins 2.19.1, latest version of all plugins), but not the org.jenkinsci.plugins.github_ branch source.BranchDiscoveryTrait and org.jenkinsci.plugins.github branch _source.ForkPullRequestDiscoveryTrait traits from your config XML. To debug the problem, I ran the following script in Script Console: import org.jenkinsci.plugins.structs.describable.DescribableModel import org.jenkinsci.plugins.github_branch_source.GitHubSCMSource new DescribableModel(GitHubSCMSource) The output is (formatted for readability): GitHubSCMSource( repoOwner: String , repository: String , apiUri?: String , buildForkPRHead?(deprecated): boolean , buildForkPRMerge?(deprecated): boolean , buildOriginBranch?(deprecated): boolean , buildOriginBranchWithPR?(deprecated): boolean , buildOriginPRHead?(deprecated): boolean , buildOriginPRMerge?(deprecated): boolean , credentialsId?: String , excludes?(deprecated): String , id?: String , includes?(deprecated): String , traits?: SCMSourceTrait{AuthorInChangelogTrait() | CheckoutOptionTrait(extension: CheckoutOption(timeout: int )) | CleanAfterCheckoutTrait() | CleanBeforeCheckoutTrait() | CloneOptionTrait(extension: CloneOption(shallow: boolean , noTags: boolean , reference: String , timeout: int , depth?: int , honorRefspec?: boolean )) | ForkPullRequestDiscoveryTrait(strategyId: int , trust: java.lang.UnsupportedOperationException: do not know how to categorize attributes of type jenkins.scm.api.trait.SCMHeadAuthority<? super org.jenkinsci.plugins.github_branch_source.GitHubSCMSourceRequest, ? extends jenkins.scm.api.mixin.ChangeRequestSCMHead2, ? extends jenkins.scm.api.SCMRevision>) | GitBrowserSCMSourceTrait(browser: GitRepositoryBrowser{AssemblaWeb(repoUrl: String ) | BitbucketWeb(repoUrl: String ) | CGit(repoUrl: String ) | FisheyeGitRepositoryBrowser(repoUrl: String ) | GitBlitRepositoryBrowser(repoUrl: String , projectName: String ) | GitLab(repoUrl: String , version: String ) | GitList(repoUrl: String ) | GitWeb(repoUrl: String ) | GithubWeb(repoUrl: String ) | Gitiles(repoUrl: String ) | GitoriousWeb(repoUrl: String ) | GogsGit(repoUrl: String ) | KilnGit(repoUrl: String ) | Phabricator(repoUrl: String , repo: String ) | RedmineWeb(repoUrl: String ) | RhodeCode(repoUrl: String ) | Stash(repoUrl: String ) | TFS2013GitRepositoryBrowser(repoUrl: String ) | ViewGitWeb(repoUrl: String , projectName: String )}) | GitLFSPullTrait() | GitToolSCMSourceTrait(gitTool: String ) | IgnoreOnPushNotificationTrait() | LocalBranchTrait() | OriginPullRequestDiscoveryTrait(strategyId: int ) | PruneStaleBranchTrait() | RefSpecsSCMSourceTrait(templates: RefSpecTemplate(value: String )[]) | RegexSCMHeadFilterTrait(regex: String ) | RemoteNameSCMSourceTrait(remoteName: String ) | SSHCheckoutTrait(credentialsId: String ) | SubmoduleOptionTrait(extension: SubmoduleOption(disableSubmodules: boolean , recursiveSubmodules: boolean , trackingSubmodules: boolean , reference: String , timeout: int , parentCredentials: boolean )) | UserIdentityTrait(extension: UserIdentity(name: String , email: String )) | WildcardSCMHeadFilterTrait(includes: String , excludes: String ) | WipeWorkspaceTrait() | jenkins.plugins.git.traits.BranchDiscoveryTrait~BranchDiscoveryTrait() | org.jenkinsci.plugins.github_branch_source.BranchDiscoveryTrait~BranchDiscoveryTrait(strategyId: int )}[]) As one can see, the problem with ForkPullRequestDiscoveryTrait is that the trust property is not supported by structs-plugin, probably because of JENKINS-26535 . The BranchDiscoveryTrait is not available because there are two classes named BranchDiscoveryTrait in different packages ( jenkins.plugins.git.traits and org.jenkinsci.plugins.github_branch_source ). Job DSL only uses the class name to determine the class and for BranchDiscoveryTrait that's ambigious, so it's not available. github-branch-source-plugin and/or git-plugin can use a @Symbol annotation to set an unambigious alias. IMHO this needs to fixed in github-branch-source-plugin, git-plugin and/or structs-plugin. Pinging Stephen Connolly , Mark Waite and Jesse Glick .
            Hide
            arichardson Alex Richardson added a comment -

            I ran into this issue today, have there been any changes to make this work?

            Or do I still need to use the workaround of manually adding to the XML in the configure block?

            Show
            arichardson Alex Richardson added a comment - I ran into this issue today, have there been any changes to make this work? Or do I still need to use the workaround of manually adding to the XML in the configure block?
            Hide
            johnlonergan John Lonergan added a comment -

            +1 it would be helpful to include a link to the resolution?

            Show
            johnlonergan John Lonergan added a comment - +1 it would be helpful to include a link to the resolution?

              People

              • Assignee:
                Unassigned
                Reporter:
                spiderfox Christian H
              • Votes:
                1 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: