Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39968

git checkouts are not pipeline-parallel safe

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We have a specific workflow that enables us to discover that bug. However, it could and will probably happen to other people soon.

      When running scm checkouts in parallels in a Jenkins file, the changes discovered are discovered for every git checkout that runs at the same time.

      In our example, we run checkouts with 8 threads in parallel (background: https://roidelapluie.be/blog/2016/11/18/gitslave-jenkins/ ).

      One change is displayed in 8 different repositories, the 8 repositories that are checked out at the same time.

      The next checkouts do not contains the changes, so I guess it is kind of a global variable that is reset after the checkout.

      lock('checkout') {
          def checkouts = [:]
          def threads = 8
      
          stage('Super Repo') {
              checkout([$class: 'GitSCM',
                      branches: [[name: branch]],
                      doGenerateSubmoduleConfigurations: false,
                      extensions: [[$class: 'CleanCheckout'], [$class: 'ScmName', name: 'super']],
                      submoduleCfg: [],
                      userRemoteConfigs: [[url: "${gitBaseUrl}/${superRepo}"]]])
      
              def repos = readFile('.gitslave')
              def reposLines = repos.readLines()
              for (i = 0; i < threads; i++){
                  checkouts["Thread ${i}"] = []
              }
              def idx = 0
              for (line in reposLines) {
                  def repoInfo = line.split(' ')
                  def repoUrl = repoInfo[0]
                  def repoPath = repoInfo[1]
                  def curatedRepoUrl = repoUrl.substring(4, repoUrl.length()-1)
                  def curatedRepoPath = repoPath.substring(1, repoPath.length()-1)
                  echo("Thread ${idx%threads}")
                  checkouts["Thread ${idx%threads}"] << [path: curatedRepoPath, url: "${gitBaseUrl}/${curatedRepoUrl}"]
                  idx++
              }
          }
          stage('GitSlave Repos') {
              def doCheckouts = [:]
              for (i = 0; i < threads; i++){
                  def j = i
                  doCheckouts["Thread ${j}"] = {
                      for (co in checkouts["Thread ${j}"]) {
                          retry(3) {
                              checkout([$class: 'GitSCM',
                                      branches: [[name: branch]],
                                      doGenerateSubmoduleConfigurations: false,
                                      extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: co.path], [$class: 'CleanCheckout'], [$class: 'ScmName', name: co.path]],
                                      submoduleCfg: [],
                                      userRemoteConfigs: [[url: co.url]]])
                          }
                      }
                  }
              }
              parallel doCheckouts
          }
      

        Attachments

          Issue Links

            Activity

            roidelapluie Julien Pivotto created issue -
            roidelapluie Julien Pivotto made changes -
            Field Original Value New Value
            Description We have a specific workflow that enables us to discover that bug. However, it could and will probably happen to other people soon.

            When running scm checkouts in parallels in a Jenkins file, the changes discovered are discovered for *every* git checkout that runs at the same time.

            In our example, we run checkouts with 8 threads in parallel (background: https://roidelapluie.be/blog/2016/11/18/gitslave-jenkins/ ).

            One change is displayed in 8 different repositories, the 8 repositories that are checked out at the same time.
            We have a specific workflow that enables us to discover that bug. However, it could and will probably happen to other people soon.

            When running scm checkouts in parallels in a Jenkins file, the changes discovered are discovered for *every* git checkout that runs at the same time.

            In our example, we run checkouts with 8 threads in parallel (background: https://roidelapluie.be/blog/2016/11/18/gitslave-jenkins/ ).

            One change is displayed in 8 different repositories, the 8 repositories that are checked out at the same time.

            The next checkouts do not contains the changes, so I guess it is kind of a global variable that is reset after the checkout.
            roidelapluie Julien Pivotto made changes -
            Description We have a specific workflow that enables us to discover that bug. However, it could and will probably happen to other people soon.

            When running scm checkouts in parallels in a Jenkins file, the changes discovered are discovered for *every* git checkout that runs at the same time.

            In our example, we run checkouts with 8 threads in parallel (background: https://roidelapluie.be/blog/2016/11/18/gitslave-jenkins/ ).

            One change is displayed in 8 different repositories, the 8 repositories that are checked out at the same time.

            The next checkouts do not contains the changes, so I guess it is kind of a global variable that is reset after the checkout.
            We have a specific workflow that enables us to discover that bug. However, it could and will probably happen to other people soon.

            When running scm checkouts in parallels in a Jenkins file, the changes discovered are discovered for *every* git checkout that runs at the same time.

            In our example, we run checkouts with 8 threads in parallel (background: https://roidelapluie.be/blog/2016/11/18/gitslave-jenkins/ ).

            One change is displayed in 8 different repositories, the 8 repositories that are checked out at the same time.

            The next checkouts do not contains the changes, so I guess it is kind of a global variable that is reset after the checkout.



            {code:java}
            lock('checkout') {
                def checkouts = [:]
                def threads = 8

                stage('Super Repo') {
                    checkout([$class: 'GitSCM',
                            branches: [[name: branch]],
                            doGenerateSubmoduleConfigurations: false,
                            extensions: [[$class: 'CleanCheckout'], [$class: 'ScmName', name: 'super']],
                            submoduleCfg: [],
                            userRemoteConfigs: [[url: "${gitBaseUrl}/${superRepo}"]]])

                    def repos = readFile('.gitslave')
                    def reposLines = repos.readLines()
                    for (i = 0; i < threads; i++){
                        checkouts["Thread ${i}"] = []
                    }
                    def idx = 0
                    for (line in reposLines) {
                        def repoInfo = line.split(' ')
                        def repoUrl = repoInfo[0]
                        def repoPath = repoInfo[1]
                        def curatedRepoUrl = repoUrl.substring(4, repoUrl.length()-1)
                        def curatedRepoPath = repoPath.substring(1, repoPath.length()-1)
                        echo("Thread ${idx%threads}")
                        checkouts["Thread ${idx%threads}"] << [path: curatedRepoPath, url: "${gitBaseUrl}/${curatedRepoUrl}"]
                        idx++
                    }
                }
                stage('GitSlave Repos') {
                    def doCheckouts = [:]
                    for (i = 0; i < threads; i++){
                        def j = i
                        doCheckouts["Thread ${j}"] = {
                            for (co in checkouts["Thread ${j}"]) {
                                retry(3) {
                                    checkout([$class: 'GitSCM',
                                            branches: [[name: branch]],
                                            doGenerateSubmoduleConfigurations: false,
                                            extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: co.path], [$class: 'CleanCheckout'], [$class: 'ScmName', name: co.path]],
                                            submoduleCfg: [],
                                            userRemoteConfigs: [[url: co.url]]])
                                }
                            }
                        }
                    }
                    parallel doCheckouts
                }
            {code}
            markewaite Mark Waite made changes -
            Assignee Mark Waite [ markewaite ]
            leedega Kevin Phillips made changes -
            Link This issue is related to JENKINS-54732 [ JENKINS-54732 ]
            tsantos Tiago Santos made changes -
            Component/s workflow-scm-step-plugin [ 21717 ]
            dnusbaum Devin Nusbaum made changes -
            Link This issue is duplicated by JENKINS-55040 [ JENKINS-55040 ]
            jglick Jesse Glick made changes -
            Link This issue duplicates JENKINS-34313 [ JENKINS-34313 ]
            ext3h Andreas Ringlstetter made changes -
            Link This issue is related to JENKINS-58068 [ JENKINS-58068 ]
            ext3h Andreas Ringlstetter made changes -
            Link This issue is related to JENKINS-58068 [ JENKINS-58068 ]

              People

              • Assignee:
                Unassigned
                Reporter:
                roidelapluie Julien Pivotto
              • Votes:
                4 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated: