Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58111

Github webhooks triggers a multibranch scan of all refs

    Details

    • Similar Issues:

      Description

      Jenkins handles a monorepo well enough when using git scm with ssh clone. If we have the following repo structure:

      // code placeholder
      api-one/
        Jenkinsfile
      api-two/
        Jenkinsfile
      ...
      api-n/
        Jenkinsfile

      We create one multibranch pipeline per Jenkinsfile in order to have many independent pipelines (CI + CD) for our projects. These projects all share common code through jenkins shared libraries and everything works ok. 

      However, if we want to use the github branch source and want to enable webhooks from github we immediately run into rate limit issues. We have a large repository - thousands of pushes a day. Every time a webhook is fired from github and hits jenkins all N multibranch pipelines begin scanning ALL remote refs for the repository. In our repository there are 10s of thousands of tags.

      A single scan on a single multibranch pipeline eliminates our entire github rate limit budget. 

      I am not entirely sure why jenkins must rescan the entire repo. It seems at the very least the webhook from github contains all information about the ref to be built and the individual jobs could simply take the webhook at face value and build/delete/etc without requiring a full repository scan. 

       

      Is this possible? Why does Jenkins currently scan the entire repo?

       

        Attachments

          Activity

          Hide
          thinkinfinite think infinite added a comment -

          Hi,

          I think we have similar issue that we have a job that triggering on tags with github webhook, but now it can be triggered by almost all Github actions, such as creating/deleting branches, committing codes/pulling request to branches, etc.  However the other jobs are working well even those are defined to be triggered on tags. I've tried to copy the working job's Jenkinsfile onto the issue job but had no luck as I got the same result.

          In the past, this issue happened sometimes, and it can be resolved by temporary disabling the job or unchecking "GitHub hook trigger for GITScm polling" under "Build Triggers", but this time it doesn't work anymore.

          Below is the checkout code:
          checkout([$class: 'GitSCM', branches: [[name: '**/tags/**']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: '$MyCredentialsId', refspec: '+refs/tags/*:refs/remotes/origin/tags/*', url: '$Github_Url']]])

          Show
          thinkinfinite think infinite added a comment - Hi, I think we have similar issue that we have a job that triggering on tags with github webhook, but now it can be triggered by almost all Github actions, such as creating/deleting branches, committing codes/pulling request to branches, etc.  However the other jobs are working well even those are defined to be triggered on tags. I've tried to copy the working job's Jenkinsfile onto the issue job but had no luck as I got the same result. In the past, this issue happened sometimes, and it can be resolved by temporary disabling the job or unchecking "GitHub hook trigger for GITScm polling" under "Build Triggers", but this time it doesn't work anymore. Below is the checkout code: checkout([$class: 'GitSCM', branches: [ [name: '**/tags/**'] ], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [ [credentialsId: '$MyCredentialsId', refspec: '+refs/tags/*:refs/remotes/origin/tags/*', url: '$Github_Url'] ]])

            People

            • Assignee:
              Unassigned
              Reporter:
              cpoole connor poole
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: