Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50975

Concurrent Groovy Shared Library syncs on different jobs use same workspace root

    Details

    • Type: Bug
    • Status: Reopened (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Environment:
    • Similar Issues:

      Description

      We're encountering an issue where multiple jobs syncing a groovy shared library concurrently results in the wrong sync root being used for some of the jobs. For example, job1 and job2 are triggered at the same time and use the p4 plugin for syncing the groovy shared library. job1 will have the correct workspace root (e.g.  /var/lib/jenkins/jobs/job1/workspace%40libs/LIBRARYNAME) for the groovy shared library sync, but job2 will sync the groovy shared library to the same workspace root as job1 (e.g.  /var/lib/jenkins/jobs/job1/workspace%40libs/LIBRARYNAME). When job2 loads the groovy shared library, it references what the workspace root should be for job2 (e.g.  /var/lib/jenkins/jobs/job2/workspace%40libs/LIBRARYNAME), which was not sync'd to, and causes job2 to run based off of an old groovy shared library sync.

        Attachments

        1. error_logs.txt
          2 kB
        2. job1.txt
          4 kB
        3. job2.txt
          4 kB

          Activity

          kroutley Kurt Routley created issue -
          Show
          kroutley Kurt Routley added a comment - Stuart Rowe mentioned it could be due to https://github.com/jenkinsci/workflow-cps-global-lib-plugin/blob/e080b501af942c15325fd17f974bec9cc04142c2/src/main/java/org/jenkinsci/plugins/workflow/libs/SCMSourceRetriever.java  , specifically the doRetrive method. It could also be a knock-on from https://issues.jenkins-ci.org/browse/JENKINS-40408    
          Hide
          p4paul Paul Allen added a comment -

          Resolved in 1.8.12

          Show
          p4paul Paul Allen added a comment - Resolved in 1.8.12
          p4paul Paul Allen made changes -
          Field Original Value New Value
          Status Open [ 1 ] Closed [ 6 ]
          Assignee Paul Allen [ p4paul ]
          Resolution Fixed [ 1 ]
          svschubert Sven Schubert made changes -
          Attachment error_logs.txt [ 44723 ]
          svschubert Sven Schubert made changes -
          Comment [ This bug still exists with version 1.8.14 of the P4 plugin, and Pipeline 2.5:  We have one shared library configured in Jenkins. If the Jenkins tries to start multiple jobs (A, B, C) at the same time (e.g. because an upstream project has been built and multiple jobs depend on this project), only the first job (A) will be able to load the shared library. The other jobs (B, C) get confused about the workspace when trying to load the shared library, e.g. job B tries to find the shared library (pipeline-commons) within the workspace of another job and not within his own workspace ([^error_logs.txt]).

          If the jobs are not started at the same time, everything works fine. 

          Environment:
           * Pipeline: 2.5
           * Pipeline: Groovy (workflow-cps): 2.54 
           * Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.9 ]
          Hide
          svschubert Sven Schubert added a comment -

          This bug still exists with version 1.8.14 of the P4 plugin, and Pipeline 2.5:  We have one shared library configured in Jenkins. If the Jenkins tries to start multiple jobs (A, B, C) at the same time (e.g. because an upstream project has been built and multiple jobs depend on this project), only the first job (A) will be able to load the shared library. The other jobs (B, C) get confused about the workspace when trying to load the shared library, e.g. job B tries to find the shared library (pipeline-commons) within the workspace of another job and not within his own workspace (error_logs.txt).

          If the jobs are not started at the same time, everything works fine. 

          Environment:

          • Pipeline: 2.5
          • Pipeline: Groovy (workflow-cps): 2.54 
          • Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.9
          Show
          svschubert Sven Schubert added a comment - This bug still exists with version 1.8.14 of the P4 plugin, and Pipeline 2.5:  We have one shared library configured in Jenkins. If the Jenkins tries to start multiple jobs (A, B, C) at the same time (e.g. because an upstream project has been built and multiple jobs depend on this project), only the first job (A) will be able to load the shared library. The other jobs (B, C) get confused about the workspace when trying to load the shared library, e.g. job B tries to find the shared library (pipeline-commons) within the workspace of another job and not within his own workspace ( error_logs.txt ). If the jobs are not started at the same time, everything works fine.  Environment: Pipeline: 2.5 Pipeline: Groovy (workflow-cps): 2.54  Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.9
          svschubert Sven Schubert made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Hide
          p4paul Paul Allen added a comment - - edited

          I have recently made changes to Global Library workspaces for multiple instances JENKINS-53922
          https://ci.jenkins.io/job/Plugins/job/p4-plugin/job/master/281/

          I may need to add ${EXECUTOR_NUMBER} to the Global Library Workspace name formatter:

          In file GlobalLibraryScmScurce.java:43

          setFormat("jenkins-lib-${NODE_NAME}-${JOB_NAME}" + id);
          
          setFormat("jenkins-lib-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}" + id);
          

          Did you want to try this out and build it yourself or I can submit a change for you to try on the nightly build?

           

          (Just re-reading your comment and logs - Build 281 might be enough without the EXECUTOR_NUMBER)

          Show
          p4paul Paul Allen added a comment - - edited I have recently made changes to Global Library workspaces for multiple instances JENKINS-53922 https://ci.jenkins.io/job/Plugins/job/p4-plugin/job/master/281/ I may need to add ${EXECUTOR_NUMBER } to the Global Library Workspace name formatter: In file GlobalLibraryScmScurce.java:43 setFormat( "jenkins-lib-${NODE_NAME}-${JOB_NAME}" + id); setFormat( "jenkins-lib-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}" + id); Did you want to try this out and build it yourself or I can submit a change for you to try on the nightly build?   (Just re-reading your comment and logs - Build 281 might be enough without the EXECUTOR_NUMBER)
          Hide
          premsai Prem Gangana added a comment -

          Hi Paul Allen,

          I have tested this with the build 281. It is using 
          "jenkins-lib-${NODE_NAME}-${JOB_NAME}" + id
          as client name now, but i would suggest replacing / with _ in id instead of . (dot). So this is unique across different jobs due to JOB_NAME in there.

          However, the issue will surface if a given job has two builds triggered at / almost at the same time, since the jenkins appends @<number> to the build workspace, and so the client root dirs will be different. I would suggest adding WORKSPACE base name some where in there.

           

          Show
          premsai Prem Gangana added a comment - Hi Paul Allen , I have tested this with the build 281. It is using  "jenkins-lib-${NODE_NAME}-${JOB_NAME}" + id as client name now, but i would suggest replacing / with _ in id instead of . (dot). So this is unique across different jobs due to JOB_NAME in there. However, the issue will surface if a given job has two builds triggered at / almost at the same time, since the jenkins appends @<number> to the build workspace, and so the client root dirs will be different. I would suggest adding WORKSPACE base name some where in there.  
          Hide
          p4paul Paul Allen added a comment -

          Happy to change . to {{_ }}but was curious to know why?  Do you have dots in your JOB_NAME?

          If the same Job is started concurrently then the EXECUTOR_NUMBER should be enough, this is the @1 or @2 that Jenkins adds to the directory. I think WORKSPACE would also work, but we could end up with very long names.  I had considered using a SHA1 of the NODE_NAME + JOB_NAME + depot path + EXECUTOR, but it becomes harder to debug if problems appear later on.

          Show
          p4paul Paul Allen added a comment - Happy to change .  to {{_ }}but was curious to know why?  Do you have dots in your JOB_NAME? If the same Job is started concurrently then the EXECUTOR_NUMBER should be enough, this is the @1 or @2 that Jenkins adds to the directory. I think WORKSPACE would also work, but we could end up with very long names.  I had considered using a SHA1 of the NODE_NAME + JOB_NAME + depot path + EXECUTOR, but it becomes harder to debug if problems appear later on.
          Hide
          p4paul Paul Allen added a comment -

          Turns out EXECUTOR_NUMBER and WORKSPACE are not define when the client workspace is created, Jenkins sets these variables later.

          Plan B is to use a UUID and clean up (delete the client) after syncing the Library.

          Show
          p4paul Paul Allen added a comment - Turns out EXECUTOR_NUMBER and WORKSPACE are not define when the client workspace is created, Jenkins sets these variables later. Plan B is to use a UUID and clean up (delete the client) after syncing the Library.
          Hide
          premsai Prem Gangana added a comment - - edited

          > Having - is for uniformity sake since you were having jenkins-lib${NODE_NAME}-${JOB_ NAME}. 

          -> Deleting the client is actually what i was going to ask you about. Cleaning up the client is needed here, otherwise the clients are going to get stacked up.

          -> How about using BUILD_NUMBER ?

           

          Show
          premsai Prem Gangana added a comment - - edited > Having - is for uniformity sake since you were having jenkins-lib${NODE_NAME}-${JOB_ NAME}.  -> Deleting the client is actually what i was going to ask you about. Cleaning up the client is needed here, otherwise the clients are going to get stacked up. -> How about using BUILD_NUMBER ?  
          Hide
          p4paul Paul Allen added a comment -

          I have tried 'Plan B' a UUID and delete. 

          I have also exposed the delete option for Manual Workspaces (visible in FreeStyle and Pipeline 'checkout' steps).  Not too sure if I will release it like this as deleting the client for normal 'checkout' steps may effect polling.

          https://ci.jenkins.io/job/Plugins/job/p4-plugin/job/master/284/

          Show
          p4paul Paul Allen added a comment - I have tried 'Plan B' a UUID and delete.  I have also exposed the delete option for Manual Workspaces (visible in FreeStyle and Pipeline 'checkout' steps).  Not too sure if I will release it like this as deleting the client for normal 'checkout' steps may effect polling. https://ci.jenkins.io/job/Plugins/job/p4-plugin/job/master/284/
          p4paul Paul Allen made changes -
          Labels root shared-groovy-libraries sync P4_A root shared-groovy-libraries sync
          Hide
          p4paul Paul Allen added a comment -

          Ready for release.

          Show
          p4paul Paul Allen added a comment - Ready for release.
          p4paul Paul Allen made changes -
          Status Reopened [ 4 ] Fixed but Unreleased [ 10203 ]
          Resolution Fixed [ 1 ]
          Hide
          p4paul Paul Allen added a comment -

          Released in 1.9.3

          Show
          p4paul Paul Allen added a comment - Released in 1.9.3
          p4paul Paul Allen made changes -
          Status Fixed but Unreleased [ 10203 ] Closed [ 6 ]
          Hide
          jazzyjayx Jay Spang added a comment - - edited

          I know this issue is old, but I think it's repro'ing for me on Jenkins 2.164.3 with P4 Plugin 1.9.7.

          Here is an example pipeline that repros the error. Note that it DYNAMICALLY loads a pipeline.

          library(
              identifier: 'dynamically-loaded-library@now', retriever: legacySCM(
                  [
                      $class: 'PerforceScm', credential: 'p4creds',
                      populate: [
                          $class: 'AutoCleanImpl', 
                          delete: true, 
                          modtime: false, 
                          parallel: [enable: false, minbytes: '1024', minfiles: '1', threads: '4'], 
                          pin: '', 
                          quiet: true, 
                          replace: true, 
                          tidy: false
                      ], 
                      workspace: [
                          $class: 'ManualWorkspaceImpl', 
                          charset: 'none', 
                          name: "jenkins-${env.JOB_NAME}-dynamically-loaded-library",
                          pinHost: false, 
                          spec: [allwrite: true, clobber: false, compress: false, line: 'LOCAL', locked: false, modtime: false, rmdir: false, streamName: '', 
                              view: "//depot/build-library/... //jenkins-${env.JOB_NAME}-dynamically-loaded-library/..."
                          ]
                      ]
                  ]
              )
          )
          
          node() {
              stage('Echo') {
                  echo("If you hit 'Build Now' rapidly enough, a build wiill fail before it even gets here.")
              }
          }
          
          

          If you hit "Build Now" once, it works great. If you rapidly hit it a few times, several builds will fail with this error.

          P4 Task: reverting all pending and shelved revisions.
          ... p4 revert d:\JenkinsHome\jobs\stream_name\workspac___
           -
          p4 revert d:\JenkinsHome\jobs\stream_name\jobs\stream_name\workspace_libs\dynamically-loaded-library_2/...
          
          ERROR: P4: Task Exception: hudson.AbortException: P4JAVA: Error(s):
          Path 'd:\JenkinsHome\jobs\stream_name\workspace_libs\dynamically-loaded-library_2/...' is not under client's root 'd:\JenkinsHome\jobs\stream_name\workspace_libs\dynamically-loaded-library'.
          

          It seems like when two builds are run in a VERY close timeframe, the plugin will append "_2" to the library's foldername, but re-uses the old p4 workspace.

          EDIT: Edited my original comment to be more concise, with a working example.

          Show
          jazzyjayx Jay Spang added a comment - - edited I know this issue is old, but I think it's repro'ing for me on Jenkins 2.164.3 with P4 Plugin 1.9.7. Here is an example pipeline that repros the error. Note that it DYNAMICALLY loads a pipeline. library( identifier: 'dynamically-loaded-library@now' , retriever: legacySCM( [ $class: 'PerforceScm' , credential: 'p4creds' , populate: [ $class: 'AutoCleanImpl' , delete: true , modtime: false , parallel: [enable: false , minbytes: '1024' , minfiles: '1' , threads: '4' ], pin: '', quiet: true , replace: true , tidy: false ], workspace: [ $class: 'ManualWorkspaceImpl' , charset: 'none' , name: "jenkins-${env.JOB_NAME}-dynamically-loaded-library" , pinHost: false , spec: [allwrite: true , clobber: false , compress: false , line: 'LOCAL' , locked: false , modtime: false , rmdir: false , streamName: '', view: " //depot/build-library/... //jenkins-${env.JOB_NAME}-dynamically-loaded-library/..." ] ] ] ) ) node() { stage( 'Echo' ) { echo( "If you hit 'Build Now' rapidly enough, a build wiill fail before it even gets here." ) } } If you hit "Build Now" once, it works great. If you rapidly hit it a few times, several builds will fail with this error. P4 Task: reverting all pending and shelved revisions. ... p4 revert d:\JenkinsHome\jobs\stream_name\workspac___ - p4 revert d:\JenkinsHome\jobs\stream_name\jobs\stream_name\workspace_libs\dynamically-loaded-library_2/... ERROR: P4: Task Exception: hudson.AbortException: P4JAVA: Error(s): Path 'd:\JenkinsHome\jobs\stream_name\workspace_libs\dynamically-loaded-library_2/...' is not under client 's root ' d:\JenkinsHome\jobs\stream_name\workspace_libs\dynamically-loaded-library'. It seems like when two builds are run in a VERY close timeframe, the plugin will append "_2" to the library's foldername, but re-uses the old p4 workspace. EDIT: Edited my original comment to be more concise, with a working example.
          jazzyjayx Jay Spang made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Hide
          msmeeth Matthew Smeeth added a comment -

          Hi Jay, 

          I'm currently trying to reproduce your issue, so far I have been unable to. If possible could you share your library file?

          Thanks,

          Matthew

          Show
          msmeeth Matthew Smeeth added a comment - Hi Jay,  I'm currently trying to reproduce your issue, so far I have been unable to. If possible could you share your library file? Thanks, Matthew
          Hide
          jazzyjayx Jay Spang added a comment - - edited

          To be extra clear, it only repros for me with dynamically loaded libraries.

          If you import your library with...

          library 'static-global-library@now'

          it doesn't happen. But if you load your library with...

          library(identifier: 'dynamically-loaded-library@now', retriever: legacySCM( // ... ) )

          it happens. Just click "Build Now" as rapidly as you you can 4-5 times.

          I can still send you a library if you'd like, but the failure happens before it even loads, so its contents don't matter.

          Show
          jazzyjayx Jay Spang added a comment - - edited To be extra clear, it only repros for me with dynamically loaded libraries. If you import your library with... library ' static -global-library@now' it doesn't happen. But if you load your library with... library(identifier: 'dynamically-loaded-library@now' , retriever: legacySCM( // ... ) ) it happens. Just click "Build Now" as rapidly as you you can 4-5 times. I can still send you a library if you'd like, but the failure happens before it even loads, so its contents don't matter.
          Hide
          jazzyjayx Jay Spang added a comment -

          Hey Matthew Smeeth, I was wondering if you had any luck repro'ing the error? The key is to use "dynamically loaded" libraries (where you specify the `retriever` property). You also have to hit "Build Now" very rapidly to get it to repro.

          I hypothesize that p4groovy isn't changing the workspace root when it decides to append "_2" to the folder name. Since it works properly when specifying global libraries (that were previously loaded into the Jenkins global config), maybe it uses a slightly different code path.

          Show
          jazzyjayx Jay Spang added a comment - Hey Matthew Smeeth , I was wondering if you had any luck repro'ing the error? The key is to use "dynamically loaded" libraries (where you specify the `retriever` property). You also have to hit "Build Now" very rapidly to get it to repro. I hypothesize that p4groovy isn't changing the workspace root when it decides to append "_2" to the folder name. Since it works properly when specifying global libraries (that were previously loaded into the Jenkins global config), maybe it uses a slightly different code path.

            People

            • Assignee:
              p4paul Paul Allen
              Reporter:
              kroutley Kurt Routley
            • Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated: