Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-51421

limit the number of concurrent request to a workflow library repository

    Details

    • Similar Issues:

      Description

      We have a workflow-lib repository enabled that contains some groovy script utilities commons for all JenkinsFile. We have named it as "workflow-libs" an located on a git repository :

      ssh://bitbucket.server/prj/workflow-lib.git

      and we use it as usual :

      
      @Library ('jworkflow-libs') _
      
       node {
           stage ("Test") {
           builder = new workflow.BuildHelper()
      //  ....
           echo "Hello world"
         }
      }
      

      This can be emulated using :

      stage("Launch all") {
      parallel(
          test1 : {build job: 'test1', wait: true},
          test2 : {build job: 'test2', wait: true},
          test3 : {build job: 'test3', wait: true},
          test4 : {build job: 'test4', wait: true},
          test5 : {build job: 'test5', wait: true},
          test6 : {build job: 'test6', wait: true},
          test7 : {build job: 'test7', wait: true},
          test8 : {build job: 'test8', wait: true},
          test9 : {build job: 'test9', wait: true},
          )
      

      Where job 'testX' contains the previous one commented.

      But we are suffering a problem with the communication between Jenkins and Bitbucket server, when you launch >5 jobs simultaneously, the 6^th^ job fails with an error message .

      hudson.plugins.git.GitException: Failed to fetch from ssh://bitbucket.server/prj/workflow-lib.git
      	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:862)
      	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
      	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.doRetrieve(SCMSourceRetriever.java:112)
      	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:84)
      	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:153)
      	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:134)
      	at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
      	at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
      	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
      	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
      	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
      	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
      	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
      	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
      	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
      	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129)
      	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:517)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:480)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269)
      	at hudson.model.ResourceController.execute(ResourceController.java:97)
      	at hudson.model.Executor.run(Executor.java:421)
      Caused by: hudson.plugins.git.GitException: Command "git fetch --no-tags --progress ssh://bitbucket.server/prj/workflow-lib.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
      stdout: 
      stderr: ssh_exchange_identification: read: Connection reset by peer
      fatal: Could not read from remote repository.
      
      Please make sure you have the correct access rights
      and the repository exists.
      

      If you relaunch the job individually it works.

      We are trying to solve it ( it could be a security restriction between bitbucket - jenkins to avoid DoS ) but would be very useful a configuration parameter for worfklow libs that limit the number of simultaneous request to the same repository (like pooling configuration has).

        Attachments

          Activity

          Hide
          fernando_rosado Fernando Rosado Altamirano added a comment -

          I have detected similar issue if you are using the same repository on different jobs. If the git server limit the number of git requests it will fails with this error.
          It wil be great if we can limit the number of simultaneous requests to the same repository globally. And probably this is a configuration that should be available like the global configuration "Max # of concurrent polling"

          Show
          fernando_rosado Fernando Rosado Altamirano added a comment - I have detected similar issue if you are using the same repository on different jobs. If the git server limit the number of git requests it will fails with this error. It wil be great if we can limit the number of simultaneous requests to the same repository globally. And probably this is a configuration that should be available like the global configuration "Max # of concurrent polling"

            People

            • Assignee:
              Unassigned
              Reporter:
              fernando_rosado Fernando Rosado Altamirano
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: