Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55248

workspace mixup when 2 jobs running on 2 diff nodes at the same time

    Details

    • Similar Issues:

      Description

      we have 2  pipelines that performs simple checkout on 2 different branches, scheduled to run at the same time on 2 different nodes every day, say Job-1 & Job-2

      sometimes we will get error like this, for example for Job-1

      P4 Task: attempt: 1
      P4 Task: failed: com.perforce.p4java.exception.P4JavaException: com.perforce.p4java.exception.P4JavaException: hudson.AbortException: P4JAVA: Error(s):

      Path 'C:\Jenkins\workspace\Job-2/...' is not under client's root 'C:\Jenkins\workspace\Job-2.

      looks like some sort of race condition going on, it doesn't happen every time.  any ideas?

       

        Attachments

          Activity

          Hide
          p4karl Karl Wirth added a comment -

          Hi Frederico Pratas, Unfortunately the extra workspaces are necessary to ensure data consistency. Perforce workspaces rely on the data being in the same location on the same node. For example - if due to multiple executions the job is running in a different Jenkins workspace location then you are going to get inconsistent results unless the workspace is unique to that location. The same goes for nodes. In general you should never use a Perforce workspace on two different nodes due to the have list that records the files saved to that location.

           

          However if my suggestion above doesn't fix it we can take this offline so I can request more confidential information about your system.

          Show
          p4karl Karl Wirth added a comment - Hi Frederico Pratas , Unfortunately the extra workspaces are necessary to ensure data consistency. Perforce workspaces rely on the data being in the same location on the same node. For example - if due to multiple executions the job is running in a different Jenkins workspace location then you are going to get inconsistent results unless the workspace is unique to that location. The same goes for nodes. In general you should never use a Perforce workspace on two different nodes due to the have list that records the files saved to that location.   However if my suggestion above doesn't fix it we can take this offline so I can request more confidential information about your system.
          Hide
          samica Frederico Pratas added a comment -

          Hi Karl Wirth,

          Thanks for your quick feedback. Unfortunately still no luck

           

          I should clarify that:

          • I'm only using the master node (single node)
          • a given job always uses the same folder/location (unless Jenkins is moving folders around in the background),
          • each p4 workspace is unique to a job (i.e, jobA always does taskA which uses workspace A, jobB always does taskB which uses workspace B), the differences are just incremental syncs with p4 (i.e., update workspace with new changelists)
          • I only allow one instance of each job to run at a given time to avoid clashes
          • a job can run in any executor

          I'v modified my script as so:

          {{
             def p4_ws_name
             def workspace_view
          
             pipeline {
                agent any
                options { timestamps() }
                stages {
                   stage('Checkout') {
                      steps {
                         script {
                            // need to initialize the global variable here because some env vars are only available inside the agent
                            p4_ws_name = 'jenkins_${JOB_NAME}_${NODE_NAME}_${EXECUTOR_NUMBER}'
                            workspace_view """\
                                           //my_depot/... //${p4_ws_name}/...
                                           """.stripIndent()
                         }
                         checkout perforce( 
                            credential: 'xxxx',
                            populate: autoClean(
                               delete: true,
                               modtime: false,
                               parallel: [
                                  enable: false,
                                  minbytes: '1024',
                                  minfiles: '1',
                                  threads: '4'],
                               pin: '',
                               quiet: true,
                               replace: true,
                               tidy: false
                            ),
                            workspace: manualSpec(
                               charset: 'none',
                               cleanup: false,
                               name: p4_ws_name,
                               pinHost: false,
                               spec: clientSpec(
                                  allwrite: false,
                                  backup: true,
                                  changeView: '',
                                  clobber: true,
                                  compress: false,
                                  line: 'LOCAL',
                                  locked: false,
                                  modtime: false,
                                  rmdir: false,
                                  serverID: '',
                                  streamName: '',
                                  type: 'WRITABLE',
                                  view: workspace_view
                               )
                            )
                         )
                      }
                   }
                   stage('Something after') {
                      steps {
                         echo "something something"
                      }
                   }
                }
             }
          }}
          
          Show
          samica Frederico Pratas added a comment - Hi Karl Wirth , Thanks for your quick feedback. Unfortunately still no luck   I should clarify that: I'm only using the master node (single node) a given job always uses the same folder/location (unless Jenkins is moving folders around in the background), each p4 workspace is unique to a job (i.e, jobA always does taskA which uses workspace A, jobB always does taskB which uses workspace B), the differences are just incremental syncs with p4 (i.e., update workspace with new changelists) I only allow one instance of each job to run at a given time to avoid clashes a job can run in any executor I'v modified my script as so: {{ def p4_ws_name def workspace_view pipeline { agent any options { timestamps() } stages { stage( 'Checkout' ) { steps { script { // need to initialize the global variable here because some env vars are only available inside the agent p4_ws_name = 'jenkins_${JOB_NAME}_${NODE_NAME}_${EXECUTOR_NUMBER}' workspace_view """\ //my_depot/... //${p4_ws_name}/... """.stripIndent() } checkout perforce( credential: 'xxxx' , populate: autoClean( delete: true , modtime: false , parallel: [ enable: false , minbytes: '1024' , minfiles: '1' , threads: '4' ], pin: '', quiet: true , replace: true , tidy: false ), workspace: manualSpec( charset: 'none' , cleanup: false , name: p4_ws_name, pinHost: false , spec: clientSpec( allwrite: false , backup: true , changeView: '', clobber: true , compress: false , line: 'LOCAL' , locked: false , modtime: false , rmdir: false , serverID: '', streamName: '', type: 'WRITABLE' , view: workspace_view ) ) ) } } stage( 'Something after' ) { steps { echo "something something" } } } } }}
          Hide
          p4karl Karl Wirth added a comment -

          Hi Frederico Pratas. I have just sent you a private email so I can request more information. Please let me know if you don't receive it.

          Show
          p4karl Karl Wirth added a comment - Hi Frederico Pratas . I have just sent you a private email so I can request more information. Please let me know if you don't receive it.
          Hide
          p4karl Karl Wirth added a comment -

          Solved by using a unique name per job in the workspace name used to get the Jenkinsfile.

          Show
          p4karl Karl Wirth added a comment - Solved by using a unique name per job in the workspace name used to get the Jenkinsfile.
          Hide
          p4karl Karl Wirth added a comment -

          Solved by using a unique name per job in the workspace name used to get the Jenkinsfile.

          Show
          p4karl Karl Wirth added a comment - Solved by using a unique name per job in the workspace name used to get the Jenkinsfile.

            People

            • Assignee:
              Unassigned
              Reporter:
              mei_liu Mei Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: