Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52088

Support for Perforce Buildfarm server

    Details

    • Similar Issues:

      Description

      One of Perforce customers strongly requests Buildfarm server support, specially for the following workflow;

      1. Create new Perforce workspaces
      2. Sync files to the Perforce workspaces
      3. Build
      4. Delete the Perforce workspaces

        Attachments

          Issue Links

            Activity

            s3037560 Kevin Park created issue -
            s3037560 Kevin Park made changes -
            Field Original Value New Value
            Attachment one_time_force_sync.png [ 43202 ]
            s3037560 Kevin Park made changes -
            Attachment p4forcesync.png [ 43203 ]
            s3037560 Kevin Park made changes -
            Attachment one_time_force_sync.png [ 43202 ]
            s3037560 Kevin Park made changes -
            Attachment p4forcesync.png [ 43203 ]
            p4karl Karl Wirth made changes -
            Attachment image-2018-06-27-17-45-09-380.png [ 43229 ]
            Hide
            p4karl Karl Wirth added a comment -

            Note: The above scenario is the only workflow that could work with build farm replicas. Anything that requires 'p4 revert' or  'p4 reconcile' to clean up the workspace or tries to publish a build/results to the Perforce server would fail.

             

            Therefore only the 'Populate options' called 'Sync only' could be supported without checking 'Revert any open or unshelved files'.

             

            Show
            p4karl Karl Wirth added a comment - Note : The above scenario is the only workflow that could work with build farm replicas. Anything that requires 'p4 revert' or  'p4 reconcile' to clean up the workspace or tries to publish a build/results to the Perforce server would fail.   Therefore only the 'Populate options' called 'Sync only' could be supported without checking 'Revert any open or unshelved files'.  
            p4karl Karl Wirth made changes -
            Labels P4_SUPPORT
            p4karl Karl Wirth made changes -
            Link This issue relates to JENKINS-39824 [ JENKINS-39824 ]
            s3037560 Kevin Park made changes -
            Attachment image-2018-06-28-17-02-46-670.png [ 43235 ]
            s3037560 Kevin Park made changes -
            Attachment image-2018-06-28-17-02-46-670.png [ 43235 ]
            s3037560 Kevin Park made changes -
            Attachment image-2018-06-28-17-16-58-850.png [ 43236 ]
            Hide
            s3037560 Kevin Park added a comment -

            I am sorry that I have missed mentioning buildfarm replica support in Multibranch Pipeline project.

             

            With the following multibranch pipeline project configuration, and the following syntax in Jenkinsfile;

            [configuration]

             
            [Jenkinsfile]

            populate: syncOnly(

            force: true,

            have: false,

            modtime: false,

            parallel: [enable: false, minbytes: '1024', minfiles: '1', threads: '4'], pin: '', quiet: false, revert: false),

             

            'p4 revert' is still running. And, 'p4 client -d' is not running, which is running when connected to an edge server.
            P4 Task: remove client: jenkins-master-edge_multi-kpark_bui-kpark_bui-0
            ... p4 client -d jenkins-master-edge_multi-kpark_bui-kpark_bui-0 -
             

            Also there is "ERROR: Could not do lightweight checkout, falling back to heavyweight" error as well.

             

            Full Jenkins log is as below. 
            Console Output
            Started by user jenkins
            ... p4 dirs //test/jenkins/* +
            ... p4 changes -m1 //test/jenkins/DSP/... +
            ERROR: Could not do lightweight checkout, falling back to heavyweight
            com.perforce.p4java.exception.RequestException: //jenkinsTemp-fa13504e-7a22-428a-9e82-84c39cbde4e9/Jenkinsfile - file(s) not in client view.

            at com.perforce.p4java.impl.mapbased.server.cmd.ResultMapParser.handleWarningStr(ResultMapParser.java:164)
            at com.perforce.p4java.impl.mapbased.rpc.OneShotServerImpl.execStreamCmd(OneShotServerImpl.java:545)
            at com.perforce.p4java.impl.mapbased.rpc.OneShotServerImpl.execStreamCmd(OneShotServerImpl.java:461)
            at com.perforce.p4java.impl.mapbased.server.cmd.BaseDelegator.execStreamCmd(BaseDelegator.java:163)
            at com.perforce.p4java.impl.mapbased.server.cmd.PrintDelegator.getFileContents(PrintDelegator.java:56)
            at com.perforce.p4java.impl.mapbased.server.Server.getFileContents(Server.java:2288)
            at org.jenkinsci.plugins.p4.scm.P4SCMFile.content(P4SCMFile.java:145)
            Caused: java.io.IOException
            at org.jenkinsci.plugins.p4.scm.P4SCMFile.content(P4SCMFile.java:147)
            at jenkins.scm.api.SCMFile.contentAsString(SCMFile.java:338)
            at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:104)
            at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:298)
            at hudson.model.ResourceController.execute(ResourceController.java:97)
            at hudson.model.Executor.run(Executor.java:429)
            Checking out p4-buildfarm-//test/jenkins/DSP/Jenkinsfile //${P4_CLIENT}/Jenkinsfile
            //test/jenkins/DSP/... //${P4_CLIENT}/...null into /var/lib/jenkins/workspace/build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3YJGA22D46TEDSF2CDQ@script to read Jenkinsfile... p4 client o jenkins-master-build-DSP-DSP-1 +
            ... p4 info +
            ... p4 client o jenkins-master-build-DSP-DSP-1 +
            ... p4 client -i +
            ... client: jenkins-master-build-DSP-DSP-1... p4 client -o jenkins-master-build-DSP-DSP-1 +
            ... p4 info +
            ... p4 counter change +
            ... p4 changes m1 -ssubmitted //jenkins-master-build-DSP-DSP-1/... +
            ... p4 repos -C +
            Building on Node: master... p4 client o jenkins-master-build-DSP-DSP-1 +
            ... p4 info +
            P4 Task: establishing connection.
            ... server: 192.168.56.47:1666
            ... node: jenkins

            P4 Task: syncing files at change: 12057... p4 sync -p /var/lib/jenkins/workspace/build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3Y___ +
            duration: (60ms)

            P4 Task: saving built changes.
            Found last change 12056 on syncID jenkins-NODE_NAME-build-DSP-DSP-EXECUTOR_NUMBER... p4 client o jenkins-master-build-DSP-DSP-1 +
            ... p4 info +
            ... p4 changes m20 //jenkins-master-build-DSP-DSP-1/...@12056,12057 +
            ... p4 changes -l -m1 @12057 +
            ... p4 user -o kpark +
            ... p4 describe -s -m51 12057 +
            ... p4 fixes -c12057 +
            ... done

            Running in Durability level: MAX_SURVIVABILITY[Pipeline] nodeRunning on Jenkins
            in /var/lib/jenkins/workspace/build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3YJGA22D46TEDSF2CDQ[Pipeline] {[Pipeline] stage[Pipeline]

            { (Sync)[Pipeline] checkout... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 info + ... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 client -i + ... client: jenkins-master-build-DSP-DSP-0... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 info + ... p4 counter change + ... p4 changes -m1 -ssubmitted //jenkins-master-build-DSP-DSP-0/... + P4: no revisions under //jenkins-master-build-DSP-DSP-0/... using latest change: 12057... p4 repos -C + Found last change 12056 on syncID jenkins-NODE_NAME-build-DSP-DSP-EXECUTOR_NUMBER... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 info + P4 Task: establishing connection. ... server: 192.168.56.47:1666 ... node: jenkins P4: Polling with range: 12056,now... p4 changes -m20 //jenkins-master-build-DSP-DSP-0/...@12056,now + ... p4 repos -C + Building on Node: master... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 info + P4 Task: establishing connection. ... server: 192.168.56.47:1666 ... node: jenkins P4 Task: syncing files at change: 12057... p4 sync -p /var/lib/jenkins/workspace/build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3Y___ + duration: (10ms) P4 Task: saving built changes. Found last change 12056 on syncID jenkins-NODE_NAME-build-DSP-DSP-EXECUTOR_NUMBER... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 info + ... p4 changes -m20 //jenkins-master-build-DSP-DSP-0/...@12056,12057 + ... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 info + ... done[Pipeline] }

            [Pipeline] // stage[Pipeline] stage[Pipeline]

            { (Verify_File)[Pipeline] sh[build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3YJGA22D46TEDSF2CDQ] Running shell script + ls -la test total 20 drwxr-xr-x 2 jenkins jenkins 4096 Jun 23 07:17 . drwxr-xr-x 4 jenkins jenkins 4096 Jun 23 07:17 .. -r--r--r-- 1 jenkins jenkins 5 Jun 23 07:17 a.txt -r--r--r-- 1 jenkins jenkins 5 Jun 23 07:17 b.txt -r--r--r-- 1 jenkins jenkins 5 Jun 23 07:17 c.txt[Pipeline] }

            [Pipeline] // stage[Pipeline] stage[Pipeline]

            { (Verify_Content)[Pipeline] sh[build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3YJGA22D46TEDSF2CDQ] Running shell script + cat test/a.txt test/b.txt test/c.txt test test test[Pipeline] sh[build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3YJGA22D46TEDSF2CDQ] Running shell script + echo cleanup true cleanup true[Pipeline] }

            [Pipeline] // stage[Pipeline] stage[Pipeline]

            { (Declarative: Post Actions)[Pipeline] cleanup... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 info + ... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 client -i + ... client: jenkins-master-build-DSP-DSP-0... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 info + P4 Task: establishing connection. ... server: 192.168.56.47:1666 ... node: jenkins P4 Task: cleanup client: jenkins-master-build-DSP-DSP-0... p4 client -o jenkins-master-build-DSP-DSP-0 - p4 client -o jenkins-master-build-DSP-DSP-0 ... p4 revert /var/lib/jenkins/workspace/build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3YJ___ - p4 revert /var/lib/jenkins/workspace/build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3YJGA22D46TEDSF2CDQ/... *Replica does not support this command.* [Pipeline] }

            [Pipeline] // stage[Pipeline] }[Pipeline] // node[Pipeline] End of PipelineFinished: SUCCESS

            Show
            s3037560 Kevin Park added a comment - I am sorry that I have missed mentioning buildfarm replica support in Multibranch Pipeline project.   With the following multibranch pipeline project configuration, and the following syntax in Jenkinsfile; [configuration]   [Jenkinsfile] populate: syncOnly( force: true, have: false, modtime: false, parallel: [enable: false, minbytes: '1024', minfiles: '1', threads: '4'] , pin: '', quiet: false, revert: false),   'p4 revert' is still running. And, 'p4 client -d' is not running, which is running when connected to an edge server. P4 Task: remove client: jenkins-master-edge_multi-kpark_bui-kpark_bui-0 ... p4 client -d jenkins-master-edge_multi-kpark_bui-kpark_bui-0 -   Also there is "ERROR: Could not do lightweight checkout, falling back to heavyweight" error as well.   Full Jenkins log is as below.  Console Output Started by user jenkins ... p4 dirs //test/jenkins/* + ... p4 changes -m1 //test/jenkins/DSP/... + ERROR: Could not do lightweight checkout, falling back to heavyweight com.perforce.p4java.exception.RequestException: //jenkinsTemp-fa13504e-7a22-428a-9e82-84c39cbde4e9/Jenkinsfile - file(s) not in client view. at com.perforce.p4java.impl.mapbased.server.cmd.ResultMapParser.handleWarningStr(ResultMapParser.java:164) at com.perforce.p4java.impl.mapbased.rpc.OneShotServerImpl.execStreamCmd(OneShotServerImpl.java:545) at com.perforce.p4java.impl.mapbased.rpc.OneShotServerImpl.execStreamCmd(OneShotServerImpl.java:461) at com.perforce.p4java.impl.mapbased.server.cmd.BaseDelegator.execStreamCmd(BaseDelegator.java:163) at com.perforce.p4java.impl.mapbased.server.cmd.PrintDelegator.getFileContents(PrintDelegator.java:56) at com.perforce.p4java.impl.mapbased.server.Server.getFileContents(Server.java:2288) at org.jenkinsci.plugins.p4.scm.P4SCMFile.content(P4SCMFile.java:145) Caused: java.io.IOException at org.jenkinsci.plugins.p4.scm.P4SCMFile.content(P4SCMFile.java:147) at jenkins.scm.api.SCMFile.contentAsString(SCMFile.java:338) at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:104) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:298) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Checking out p4-buildfarm-//test/jenkins/DSP/Jenkinsfile //${P4_CLIENT}/Jenkinsfile //test/jenkins/DSP/... //${P4_CLIENT}/...null into /var/lib/jenkins/workspace/build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3YJGA22D46TEDSF2CDQ@script to read Jenkinsfile... p4 client o jenkins-master-build-DSP-DSP -1 + ... p4 info + ... p4 client o jenkins-master-build-DSP-DSP -1 + ... p4 client -i + ... client: jenkins-master-build-DSP-DSP- 1... p4 client -o jenkins-master-build-DSP-DSP -1 + ... p4 info + ... p4 counter change + ... p4 changes m1 -ssubmitted //jenkins-master-build-DSP-DSP -1/... + ... p4 repos -C + Building on Node: master... p4 client o jenkins-master-build-DSP-DSP -1 + ... p4 info + P4 Task: establishing connection. ... server: 192.168.56.47:1666 ... node: jenkins P4 Task: syncing files at change: 12057... p4 sync -p /var/lib/jenkins/workspace/build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3Y___ + duration: (60ms) P4 Task: saving built changes. Found last change 12056 on syncID jenkins-NODE_NAME-build-DSP-DSP-EXECUTOR_NUMBER... p4 client o jenkins-master-build-DSP-DSP -1 + ... p4 info + ... p4 changes m20 //jenkins-master-build-DSP-DSP -1/...@12056,12057 + ... p4 changes -l -m1 @12057 + ... p4 user -o kpark + ... p4 describe -s -m51 12057 + ... p4 fixes -c12057 + ... done Running in Durability level: MAX_SURVIVABILITY [Pipeline] nodeRunning on Jenkins in /var/lib/jenkins/workspace/build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3YJGA22D46TEDSF2CDQ [Pipeline] { [Pipeline] stage [Pipeline] { (Sync)[Pipeline] checkout... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 info + ... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 client -i + ... client: jenkins-master-build-DSP-DSP-0... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 info + ... p4 counter change + ... p4 changes -m1 -ssubmitted //jenkins-master-build-DSP-DSP-0/... + P4: no revisions under //jenkins-master-build-DSP-DSP-0/... using latest change: 12057... p4 repos -C + Found last change 12056 on syncID jenkins-NODE_NAME-build-DSP-DSP-EXECUTOR_NUMBER... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 info + P4 Task: establishing connection. ... server: 192.168.56.47:1666 ... node: jenkins P4: Polling with range: 12056,now... p4 changes -m20 //jenkins-master-build-DSP-DSP-0/...@12056,now + ... p4 repos -C + Building on Node: master... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 info + P4 Task: establishing connection. ... server: 192.168.56.47:1666 ... node: jenkins P4 Task: syncing files at change: 12057... p4 sync -p /var/lib/jenkins/workspace/build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3Y___ + duration: (10ms) P4 Task: saving built changes. Found last change 12056 on syncID jenkins-NODE_NAME-build-DSP-DSP-EXECUTOR_NUMBER... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 info + ... p4 changes -m20 //jenkins-master-build-DSP-DSP-0/...@12056,12057 + ... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 info + ... done[Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Verify_File)[Pipeline] sh[build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3YJGA22D46TEDSF2CDQ] Running shell script + ls -la test total 20 drwxr-xr-x 2 jenkins jenkins 4096 Jun 23 07:17 . drwxr-xr-x 4 jenkins jenkins 4096 Jun 23 07:17 .. -r--r--r-- 1 jenkins jenkins 5 Jun 23 07:17 a.txt -r--r--r-- 1 jenkins jenkins 5 Jun 23 07:17 b.txt -r--r--r-- 1 jenkins jenkins 5 Jun 23 07:17 c.txt[Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Verify_Content)[Pipeline] sh[build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3YJGA22D46TEDSF2CDQ] Running shell script + cat test/a.txt test/b.txt test/c.txt test test test[Pipeline] sh[build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3YJGA22D46TEDSF2CDQ] Running shell script + echo cleanup true cleanup true[Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Declarative: Post Actions)[Pipeline] cleanup... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 info + ... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 client -i + ... client: jenkins-master-build-DSP-DSP-0... p4 client -o jenkins-master-build-DSP-DSP-0 + ... p4 info + P4 Task: establishing connection. ... server: 192.168.56.47:1666 ... node: jenkins P4 Task: cleanup client: jenkins-master-build-DSP-DSP-0... p4 client -o jenkins-master-build-DSP-DSP-0 - p4 client -o jenkins-master-build-DSP-DSP-0 ... p4 revert /var/lib/jenkins/workspace/build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3YJ___ - p4 revert /var/lib/jenkins/workspace/build_DSP-3XHSNXIDZZ35NUQ6WQN4R5QHUQRJU23ZE3YJGA22D46TEDSF2CDQ/... *Replica does not support this command.* [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] End of PipelineFinished: SUCCESS

              People

              • Assignee:
                Unassigned
                Reporter:
                s3037560 Kevin Park
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: