Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39682

Generated Multibranch Pipeline Job does not index branches

    Details

    • Similar Issues:

      Description

      Using a JobDSL script to generate a multibranch Pipeline job does not trigger a branch index to find Jenkinsfile

      Here is a simple dsl job

      multibranchPipelineJob(repo) {
        branchSources {
          github {
            scanCredentialsId(credentials)
            repoOwner(credentials)
            repository(repo)
          }
        }
      }
      

      Using this created the job fine but did not trigger a branch scan until I manually triggered a branch index. It also works if you open the multibranch job configuration and save it with no changes.

      Creating a multibranch job directly from the UI works fine.

      The only way I can trigger a branch index is to add a triggers section to the command to periodically scan every minute. I then had to create 3 build steps:

      1. JobDSL to create mutlibranch Pipeline job with a trigger set to 1 minute
      2. Shell step to Sleep for 60 seconds
      3. JobDSL to modify the multibranch Pipeline job and turn off the trigger.

        Attachments

          Issue Links

            Activity

            Hide
            ingwar Karol Lassak added a comment -

            Michel Zanini we also found that issue that all our ~ 800 repos was reindexed every night (ofc one day is not enough to reindex all of them with GH API limits)

            But I found way how to fix it..

            See https://github.com/jenkinsci/job-dsl-plugin/blob/master/job-dsl-plugin/src/main/groovy/javaposse/jobdsl/plugin/JenkinsJobManagement.java#L456
            If your configuration is not changed then updateByXml is not called, so no reindexing is done..

             

            In our case it was even harder to catch cause jobs itself did not changed but folders did (we have some portlets there that have random ID by default)
            We changed them to static IDs and problem is gone..

            Show
            ingwar Karol Lassak added a comment - Michel Zanini we also found that issue that all our ~ 800 repos was reindexed every night (ofc one day is not enough to reindex all of them with GH API limits) But I found way how to fix it.. See  https://github.com/jenkinsci/job-dsl-plugin/blob/master/job-dsl-plugin/src/main/groovy/javaposse/jobdsl/plugin/JenkinsJobManagement.java#L456 If your configuration is not changed then  updateByXml is not called, so no reindexing is done..   In our case it was even harder to catch cause jobs itself did not changed but folders did (we have some portlets there that have random ID by default) We changed them to static IDs and problem is gone..
            Hide
            dopingus Thao-Nguyen Do added a comment - - edited

            -I have the same problem as Aaron Franssell despite the suggested fix by greg oire.

            The repository scan fails with: "FATAL: Invalid scan credentials when using <valid credential name>"
            This can be fixed manually by navigating to the job's configuration page and saving the job configuration without any changes.

            However, it is not feasible for the Jenkins instance that I'm managing to do this for every job, as we create a large number of these jobs through Job DSL.
            I tried to looked through the API for some of the involved classes/interfaces, but haven't found anything suitable.-

            def allJobs = Jenkins.instance.getAllItems();
            
            for(item in allJobs) {
              if(item.getClass().equals(org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject)) {
                item.save();
                item.getSCMSources().get(0).afterSave()
              }
            }
            

            -Unfortunately, this did not work.

            Is there any solution or workaround for this?

             -- 

            Used plugins: Branch API Plugin 2.5.8, Github Branch Source Plugin 2.8.3-

            Edit: The APIUri in my Job DSL definition was set incorrectly. When Github Enterprise is used it has to end with "/api/v3"

            Show
            dopingus Thao-Nguyen Do added a comment - - edited - I have the same problem as Aaron Franssell despite the suggested fix by greg oire . The repository scan fails with: "FATAL: Invalid scan credentials when using <valid credential name>" This can be fixed manually by navigating to the job's configuration page and saving the job configuration without any changes. However, it is not feasible for the Jenkins instance that I'm managing to do this for every job, as we create a large number of these jobs through Job DSL. I tried to looked through the API for some of the involved classes/interfaces, but haven't found anything suitable. - def allJobs = Jenkins.instance.getAllItems(); for (item in allJobs) { if (item.getClass().equals(org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject)) { item.save(); item.getSCMSources().get(0).afterSave() } } - Unfortunately, this did not work. Is there any solution or workaround for this?  --  Used plugins: Branch API Plugin 2.5.8, Github Branch Source Plugin 2.8.3 - Edit: The APIUri in my Job DSL definition was set incorrectly. When Github Enterprise is used it has to end with "/api/v3"
            Hide
            michelzanini Michel Zanini added a comment -

            Almost a year after, and I am still locked to branch-api version 2.5.4 to avoid the issues caused by this change.

             

            I discovered my problem is related to folders, but, unlike what has been described by Karol Lassak, it's seems to be related to having Views inside folders.

            Apparently, if you have views inside folder using job-dsl, they always cause the folder XML to change, and this triggers cascading builds for everything inside those folders. 

            That means every time my Seed Job runs it re-runs hundreds of jobs. I tried everything, but with no luck. The only thing I can do is abandon the Views inside folder and give up on having them. The XML generated by my folders always have the same content, and still it's triggering builds.

             

            I think this should be changed to avoid re-building jobs inside folders. It also could somehow be improved, maybe on job-dsl to avoid re-building a project if little has changed. Or at least, avoid re-building all at once.

             

            I am going to try to create an issue on job-dsl to see if someone can do something about this. It's definitely a pain.

             

            Show
            michelzanini Michel Zanini added a comment - Almost a year after, and I am still locked to branch-api version 2.5.4 to avoid the issues caused by this change.   I discovered my problem is related to folders, but, unlike what has been described by Karol Lassak , it's seems to be related to having Views inside folders. Apparently, if you have views inside folder using job-dsl, they always cause the folder XML to change, and this triggers cascading builds for everything inside those folders.  That means every time my Seed Job runs it re-runs hundreds of jobs. I tried everything, but with no luck. The only thing I can do is abandon the Views inside folder and give up on having them. The XML generated by my folders always have the same content, and still it's triggering builds.   I think this should be changed to avoid re-building jobs inside folders. It also could somehow be improved, maybe on job-dsl to avoid re-building a project if little has changed. Or at least, avoid re-building all at once.   I am going to try to create an issue on job-dsl to see if someone can do something about this. It's definitely a pain.  
            Hide
            michelzanini Michel Zanini added a comment -

            I created this issue https://issues.jenkins-ci.org/browse/JENKINS-63344 for my problem on job-dsl.

            Show
            michelzanini Michel Zanini added a comment - I created this issue  https://issues.jenkins-ci.org/browse/JENKINS-63344  for my problem on job-dsl.
            Hide
            bitwiseman Liam Newman added a comment -

            Michel Zanini 

            Thank you for opening a new issue, I've closed this one. 
            I did this only because the scenario you're describing is not the scenario in this issue, it is caused by the change that fixed this issue. 

            I'll look at the new issue and comment over there. 

            Show
            bitwiseman Liam Newman added a comment - Michel Zanini   Thank you for opening a new issue, I've closed this one.  I did this only because the scenario you're describing is not the scenario in this issue, it is caused by the change that fixed this issue.  I'll look at the new issue and comment over there. 

              People

              • Assignee:
                daspilker Daniel Spilker
                Reporter:
                hrmpw Patrick Wolf
              • Votes:
                7 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: