Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-7436

Add support for perforce shelve builds

    Details

    • Type: New Feature
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: p4-plugin
    • Labels:
      None
    • Similar Issues:

      Description

      It would be really useful if the perforce plugin could add support for shelving – Functionality added perforce in version 2009.2.

      Here is a good blog writeup --http://blog.perforce.com/blog/?p=1872

      Many things hudson is great for is finding out if anyone "broke" the build by polling source repositories looking for commits and kicking off builds. But breaking builds and backing code is a pain sometimes. It would be even better if you could run a build in hudson BEFORE checking in your changelist to see if your changelist WOULD break the build and fix problems before they occur. That is exactly what perforce shelving does – It allows you to shelve your changelist – saving all your modifications to the server (without committing them), and then allowing others to pull down the shelved modified code and perform a build.

      I'm not sure if the underlying tek42 perforce client library you use supports shelving, but if so this would be really useful functionality.

      Thanks.

      Doug

        Attachments

          Activity

          Hide
          rpetti Rob Petti added a comment -

          The version of the tek42 library we use probably doesn't support this at all, but we had branched it ourselves quite a while ago because the maintainer wasn't answering our calls regarding outstanding bugs. It's currently sitting with the rest of our source code, so we can modify it as we please.

          I imagine we would need to allow a user to specify a shelved changelist or maybe a list of shelved changelists to pull down, probably through a job parameter. Does that sound like what you had in mind?

          Show
          rpetti Rob Petti added a comment - The version of the tek42 library we use probably doesn't support this at all, but we had branched it ourselves quite a while ago because the maintainer wasn't answering our calls regarding outstanding bugs. It's currently sitting with the rest of our source code, so we can modify it as we please. I imagine we would need to allow a user to specify a shelved changelist or maybe a list of shelved changelists to pull down, probably through a job parameter. Does that sound like what you had in mind?
          Hide
          vbuzzsaw vbuzzsaw added a comment -

          Yes, that's pretty much what I had envisioned. I would think min requirement would be to support one shelved changelist in the p4 configuration, and ensure this could be expressed as a token to be resolved from a build job parameter. I hadn't through through the idea of supporting multiple changelists. I suppose that would be possible. My concern would be that one might need to add extra logic to support the edge case that is a conflict in the multiple changelists, as the plugin would have no idea how to resolve it.

          The main use case I would be trying to solve is where a developer in their local workspace has one changelist they are ready to check in, but in order to sanity test it, they could shelve that changelist and manually run a hudson build referring to the shelved changelist to verify it works outside of their local environment before committing it.

          Quite frankly, I only have very cursory experience with this functionality, but could envision taking advantage of it and integrating it into our reg dev process if could be easily encorporated this plugin and as easy to use as the current plugin functionality (For example, I love how you make specifying workspaces so easy, creating them if they don't exist...)

          As I get more experience with this functionality, I will add any additional comments I might have from my learning.

          Thanks.

          Show
          vbuzzsaw vbuzzsaw added a comment - Yes, that's pretty much what I had envisioned. I would think min requirement would be to support one shelved changelist in the p4 configuration, and ensure this could be expressed as a token to be resolved from a build job parameter. I hadn't through through the idea of supporting multiple changelists. I suppose that would be possible. My concern would be that one might need to add extra logic to support the edge case that is a conflict in the multiple changelists, as the plugin would have no idea how to resolve it. The main use case I would be trying to solve is where a developer in their local workspace has one changelist they are ready to check in, but in order to sanity test it, they could shelve that changelist and manually run a hudson build referring to the shelved changelist to verify it works outside of their local environment before committing it. Quite frankly, I only have very cursory experience with this functionality, but could envision taking advantage of it and integrating it into our reg dev process if could be easily encorporated this plugin and as easy to use as the current plugin functionality (For example, I love how you make specifying workspaces so easy, creating them if they don't exist...) As I get more experience with this functionality, I will add any additional comments I might have from my learning. Thanks.
          Hide
          rpetti Rob Petti added a comment -

          Sounds good. When I can find the time, I'll start looking into adding shelve functionality to the tek42 libraries we're using, then we can go from there.

          Show
          rpetti Rob Petti added a comment - Sounds good. When I can find the time, I'll start looking into adding shelve functionality to the tek42 libraries we're using, then we can go from there.
          Hide
          vbuzzsaw vbuzzsaw added a comment -

          OK, had a bit more time to work out the use case with some colleagues. What we are looking for is for the perforce-plugin to support the following command line equivalent:

          p4 unshelve -s changelist [-f]

          Details on the command can be found here: http://www.perforce.com/perforce/doc.current/manuals/cmdref/unshelve.html

          The perforce plugin should behave as it always does, specifying a perforce client workspace and ensuring a p4 sync of that workspace prior to executing the build.

          The additional work would be to support an additional option to specify a shelved changelist, and if specified, after syncing the workspace, the p4 unshelve -s changelist command would be run to bring down the pending changes and the perform the build. The -f option to that command would force an overwrite of any files with write access – not certain if that also needs to be an option to specify, or if we want to default the behavior.

          A couple of additional thoughts on this:

          1. the perforce-plugin is responsible for listing the changes since the last build – Might make sense to include in that set of changes in pending shelved changelist, and somehow indicate or highlight the changelist is pending/shelved.

          2. upon completion of the build, there will be open files in the active perforce workspace. Need to determine the correct behavior to clean that up. Might make sense at the end of the build to discard those changes, and revert the workspace back prior to picking up the shelved changelist. Optionally, there was a suggestion that a user might want to indicate in the plugin confguration to commit the pending changlist if the build succeeds. That sounds cool but might be an V2 of the implementation as it might require a bit more work.

          Show
          vbuzzsaw vbuzzsaw added a comment - OK, had a bit more time to work out the use case with some colleagues. What we are looking for is for the perforce-plugin to support the following command line equivalent: p4 unshelve -s changelist [-f] Details on the command can be found here: http://www.perforce.com/perforce/doc.current/manuals/cmdref/unshelve.html The perforce plugin should behave as it always does, specifying a perforce client workspace and ensuring a p4 sync of that workspace prior to executing the build. The additional work would be to support an additional option to specify a shelved changelist, and if specified, after syncing the workspace, the p4 unshelve -s changelist command would be run to bring down the pending changes and the perform the build. The -f option to that command would force an overwrite of any files with write access – not certain if that also needs to be an option to specify, or if we want to default the behavior. A couple of additional thoughts on this: 1. the perforce-plugin is responsible for listing the changes since the last build – Might make sense to include in that set of changes in pending shelved changelist, and somehow indicate or highlight the changelist is pending/shelved. 2. upon completion of the build, there will be open files in the active perforce workspace. Need to determine the correct behavior to clean that up. Might make sense at the end of the build to discard those changes, and revert the workspace back prior to picking up the shelved changelist. Optionally, there was a suggestion that a user might want to indicate in the plugin confguration to commit the pending changlist if the build succeeds. That sounds cool but might be an V2 of the implementation as it might require a bit more work.
          Hide
          jw7mitchell jw7mitchell added a comment -

          Regarding vbuzzsaw's comment:
          "2. upon completion of the build, there will be open files in the active perforce workspace. Need to determine the correct behavior to clean that up. Might make sense at the end of the build to discard those changes, and revert the workspace back prior to picking up the shelved changelist. Optionally, there was a suggestion that a user might want to indicate in the plugin confguration to commit the pending changlist if the build succeeds."

          The main use case we expect this functionality to be used for would be for a user to test their code changes by building and / or testing them on a central build / test engine prior to checking their code in.

          If this is so, we'd expect the default behavior is to leave the shelved files checked out so that the user can determine based upon the result of the build/test whether to check them into the codeline or not. Reverting the codeline would not be the thing to do in this case, since the changes would be lost. An option to check-in automatically might be useful.

          Show
          jw7mitchell jw7mitchell added a comment - Regarding vbuzzsaw's comment: "2. upon completion of the build, there will be open files in the active perforce workspace. Need to determine the correct behavior to clean that up. Might make sense at the end of the build to discard those changes, and revert the workspace back prior to picking up the shelved changelist. Optionally, there was a suggestion that a user might want to indicate in the plugin confguration to commit the pending changlist if the build succeeds." The main use case we expect this functionality to be used for would be for a user to test their code changes by building and / or testing them on a central build / test engine prior to checking their code in. If this is so, we'd expect the default behavior is to leave the shelved files checked out so that the user can determine based upon the result of the build/test whether to check them into the codeline or not. Reverting the codeline would not be the thing to do in this case, since the changes would be lost. An option to check-in automatically might be useful.
          Hide
          rpetti Rob Petti added a comment -

          The user would then need direct access to the build machine, and the perforce user that hudson is running under. Reverting the hudson workspace is necessary so as not to pollute subsequent builds. Any commits (that aren't automatic) should be done by the user in their own workspace.

          As I understand it, unshelving a changelist, then reverting the open files does not delete the shelved changelist. It will only return the workspace to the state it was at before the unshelving occurred, which is exactly what we want.

          Show
          rpetti Rob Petti added a comment - The user would then need direct access to the build machine, and the perforce user that hudson is running under. Reverting the hudson workspace is necessary so as not to pollute subsequent builds. Any commits (that aren't automatic) should be done by the user in their own workspace. As I understand it, unshelving a changelist, then reverting the open files does not delete the shelved changelist. It will only return the workspace to the state it was at before the unshelving occurred, which is exactly what we want.
          Hide
          jw7mitchell jw7mitchell added a comment -

          Regarding rpetti's comment - You are correct. Reversion on the build machine is the correct default behavior.

          Show
          jw7mitchell jw7mitchell added a comment - Regarding rpetti's comment - You are correct. Reversion on the build machine is the correct default behavior.
          Hide
          pehu Per Hugoson added a comment -

          Has anyone looked in to this yet?

          Show
          pehu Per Hugoson added a comment - Has anyone looked in to this yet?
          Hide
          rpetti Rob Petti added a comment -

          Not yet, unfortunately.

          Show
          rpetti Rob Petti added a comment - Not yet, unfortunately.
          Hide
          bbala_anna balasubramaniam bodeddula added a comment -

          Yes, really unfortunate. Anyways, do you have a high level idea on how this can be achieved in some hacky way. I haven't looked at the plugin code yet but how easy or tough it is for some new person to understand the code and add this feature in?

          Show
          bbala_anna balasubramaniam bodeddula added a comment - Yes, really unfortunate. Anyways, do you have a high level idea on how this can be achieved in some hacky way. I haven't looked at the plugin code yet but how easy or tough it is for some new person to understand the code and add this feature in?
          Hide
          rpetti Rob Petti added a comment -

          It would be very difficult, especially for a new person. The underlying api would need to be extended to support shelving, you'd need to make new UI elements, and then you'd need to sort through the tangle of checkout code to figure out where it should live. And that's just for /unshelving/. There's still autocommitting, merging, and reverting that needs to be implemented as well, which would likely take the form of post-build plugins that will need to be written from scratch.

          The easiest, hackiest way would be to simply add the shelving commands to your job as a build step. It should only require a few lines of bash or batch.

          Show
          rpetti Rob Petti added a comment - It would be very difficult, especially for a new person. The underlying api would need to be extended to support shelving, you'd need to make new UI elements, and then you'd need to sort through the tangle of checkout code to figure out where it should live. And that's just for /unshelving/. There's still autocommitting, merging, and reverting that needs to be implemented as well, which would likely take the form of post-build plugins that will need to be written from scratch. The easiest, hackiest way would be to simply add the shelving commands to your job as a build step. It should only require a few lines of bash or batch.
          Hide
          aribiross Ross Aribi added a comment -

          I've implemented a solution to enable a user to test builds using a "shelved" changelist if desired. This isn't a plugin, but rather a process incorporated within the build request form in Jenkins (Parameterized build).

          The Shelving process itself involves the following steps:
          p4 unshelve -f -s ${SHELVED_CHANGELIST} ${sourceCodePath}
          p4 revert -k ${sourceCodePath}

          Where:
          The -f flag forces the clobbering of any writeable but unopened files that are being unshelved. The -s flag specifies the number of the pending changelist that contains the shelved files.

          So, my implementation in Jenkins is as follows:
          1) Setup a parameterized build template
          2) Add a string parameter to the build template and name it something like: SHELEVED_CHANGELIST
          3) In the build section (I execute shell), put the following steps in the beginning of your build steps:

          1. Revert previous shelving activities
            p4 revert ${sourceCodePath}
          1. Unshelve if true
            if [ $SHELVED_CHANGELIST != "" ] ; then ( echo "User specified a shelved changelist, unshelving changelist ($SHELVED_CHANGELIST)";
            p4 unshelve -f -s $SHELVED_CHANGELIST ${sourceCodePath};
            )
            else (
            echo "Not running with a shelved changelist"; ); fi
          1. Now revert unshelve but KEEP the files locally
            p4 revert -k ${sourceCodePath}

          That's it. We've been using this solution successfully for about a year now and developers like it. All you need to do is to educate your developers on how to do shelving and create that changelist

          Show
          aribiross Ross Aribi added a comment - I've implemented a solution to enable a user to test builds using a "shelved" changelist if desired. This isn't a plugin, but rather a process incorporated within the build request form in Jenkins (Parameterized build). The Shelving process itself involves the following steps: p4 unshelve -f -s ${SHELVED_CHANGELIST} ${sourceCodePath} p4 revert -k ${sourceCodePath} Where: The -f flag forces the clobbering of any writeable but unopened files that are being unshelved. The -s flag specifies the number of the pending changelist that contains the shelved files. So, my implementation in Jenkins is as follows: 1) Setup a parameterized build template 2) Add a string parameter to the build template and name it something like: SHELEVED_CHANGELIST 3) In the build section (I execute shell), put the following steps in the beginning of your build steps: Revert previous shelving activities p4 revert ${sourceCodePath} Unshelve if true if [ $SHELVED_CHANGELIST != "" ] ; then ( echo "User specified a shelved changelist, unshelving changelist ($SHELVED_CHANGELIST)"; p4 unshelve -f -s $SHELVED_CHANGELIST ${sourceCodePath}; ) else ( echo "Not running with a shelved changelist"; ); fi Now revert unshelve but KEEP the files locally p4 revert -k ${sourceCodePath} That's it. We've been using this solution successfully for about a year now and developers like it. All you need to do is to educate your developers on how to do shelving and create that changelist
          Hide
          rpetti Rob Petti added a comment -

          Wait, if you are leaving the unshelved files behind, doesn't that pollute the build environment for the next build? Or are you cleaning and force syncing on every build?

          Show
          rpetti Rob Petti added a comment - Wait, if you are leaving the unshelved files behind, doesn't that pollute the build environment for the next build? Or are you cleaning and force syncing on every build?
          Hide
          richardtaylor Richard Taylor added a comment -

          Hi. We've had a solution like this in mind for a while now. I've got a few questions.

          p4 revert -k ${sourceCodePath}

          When the next builds starts how do you ensure that the workspace does not contain any modified files? Even if the -k flag was not specified any files which where 'added' in the change list would still be left on the disc (and this would cause a problem for our build system). Our planned approach is to write a script which does the unshelve and also writes out a cleanup script which handles the adds and other cases not covered by p4 revert. Is there a simplere approach.

          Many thanks
          Rich

          Show
          richardtaylor Richard Taylor added a comment - Hi. We've had a solution like this in mind for a while now. I've got a few questions. p4 revert -k ${sourceCodePath} When the next builds starts how do you ensure that the workspace does not contain any modified files? Even if the -k flag was not specified any files which where 'added' in the change list would still be left on the disc (and this would cause a problem for our build system). Our planned approach is to write a script which does the unshelve and also writes out a cleanup script which handles the adds and other cases not covered by p4 revert. Is there a simplere approach. Many thanks Rich
          Hide
          fmerrow Frank Merrow added a comment -

          FWIW: This kind of a "pre-flight" feature has been on our To Do List for awhile now (just like this CR) . . . It would be great if you did it for us. <wink> Issue "voted" and "watched"

          Show
          fmerrow Frank Merrow added a comment - FWIW: This kind of a "pre-flight" feature has been on our To Do List for awhile now (just like this CR) . . . It would be great if you did it for us. <wink> Issue "voted" and "watched"
          Hide
          rpetti Rob Petti added a comment -

          Richard,
          This response is really late, but this is basically what I had in mind:

          -If there is a changeset already unshelved from last build:
           --Issue 'p4 opened' to get list of added files
           --Issue 'p4 revert'
           --Cleanup added files, since they would be left behind otherwise
          -Sync
          -Unshelve
          -Build
          

          Note that nowhere would we use '-k'. That would put the workspace into an inconsistent state, and we can't have that.

          Show
          rpetti Rob Petti added a comment - Richard, This response is really late, but this is basically what I had in mind: -If there is a changeset already unshelved from last build: --Issue 'p4 opened' to get list of added files --Issue 'p4 revert' --Cleanup added files, since they would be left behind otherwise -Sync -Unshelve -Build Note that nowhere would we use '-k'. That would put the workspace into an inconsistent state, and we can't have that.
          Hide
          abigos Andy Bigos added a comment -

          Just bumping this issue as wondering if there has been any progress. It sounds like the way to get something going (at the moment) is as a build step, however it does really seem to fall in the domain of the scm plugin.

          The complications seem to come from trying to be consistent when the workspace isn't cleaned, which is something I guess most people don't do as it's generally considered best practice to clean the workspace between builds. I would think this was especially true when dealing the pre-commit testing.

          With that in mind, I wonder if it's possible to do an initial implementation that relies on having a clean workspace? E.g. if a shelved CL is provided (via a parameter) this overrides the clean workspace setting so that it always starts from a known state? That would remove some of the complexity and I guess meet most peoples use cases. Just an idea.

          I'd be interesting to hear of any progress on the plugin side before we go ahead and implement this as a build step.

          Ta
          Andy

          Show
          abigos Andy Bigos added a comment - Just bumping this issue as wondering if there has been any progress. It sounds like the way to get something going (at the moment) is as a build step, however it does really seem to fall in the domain of the scm plugin. The complications seem to come from trying to be consistent when the workspace isn't cleaned, which is something I guess most people don't do as it's generally considered best practice to clean the workspace between builds. I would think this was especially true when dealing the pre-commit testing. With that in mind, I wonder if it's possible to do an initial implementation that relies on having a clean workspace? E.g. if a shelved CL is provided (via a parameter) this overrides the clean workspace setting so that it always starts from a known state? That would remove some of the complexity and I guess meet most peoples use cases. Just an idea. I'd be interesting to hear of any progress on the plugin side before we go ahead and implement this as a build step. Ta Andy
          Hide
          rpetti Rob Petti added a comment -

          As far as I know, nobody has been working on this. We don't use shelving at my organization, so I can't justify using company time to implement it.

          Show
          rpetti Rob Petti added a comment - As far as I know, nobody has been working on this. We don't use shelving at my organization, so I can't justify using company time to implement it.
          Hide
          abigos Andy Bigos added a comment -

          >.. I can't justify using company time to implement it.

          Thanks for update Rob, we will look at implementing this via build steps in that case.

          Show
          abigos Andy Bigos added a comment - >.. I can't justify using company time to implement it. Thanks for update Rob, we will look at implementing this via build steps in that case.
          Hide
          jm0221 John McGowan added a comment - - edited

          We've been doing precommit/shelved builds for a good while in a build step prior to the main job. Parameterized build with the p4 shelve number as input. I run "Run buildstep before SCM":

          #!/bin/bash -ex

          p4 revert ...

          Then before the main build task I run this (this is on a clean and new workspace everytime btw):

          if [ $SHELVE != 0 ]; then

          1. Unshelve the p4 shelve list supplied in the parameter 'SHELVE'
            p4 unshelve -s $SHELVE
            if [ $? -ne 0 ]; then
            echo "Unshelving change $SHELVE hasn't worked, aborting"
            exit 1
            fi
            else
            echo "Input a value for the P4 shelve command or else you're just building the latest nightly code, which is covered in another job"
            exit 1
            fi
          Show
          jm0221 John McGowan added a comment - - edited We've been doing precommit/shelved builds for a good while in a build step prior to the main job. Parameterized build with the p4 shelve number as input. I run "Run buildstep before SCM": #!/bin/bash -ex p4 revert ... Then before the main build task I run this (this is on a clean and new workspace everytime btw): if [ $SHELVE != 0 ]; then Unshelve the p4 shelve list supplied in the parameter 'SHELVE' p4 unshelve -s $SHELVE if [ $? -ne 0 ]; then echo "Unshelving change $SHELVE hasn't worked, aborting" exit 1 fi else echo "Input a value for the P4 shelve command or else you're just building the latest nightly code, which is covered in another job" exit 1 fi

            People

            • Assignee:
              Unassigned
              Reporter:
              vbuzzsaw vbuzzsaw
            • Votes:
              22 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

              • Created:
                Updated: