Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-4603

Build based on a specific Perforce Changelist number.

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: perforce-plugin
    • Labels:
      None
    • Environment:
      Platform: All, OS: All
    • Similar Issues:

      Description

      We have a need to trigger Hudson to build a project tree based on a known
      Perforce Changelist number. This may seem backwards from the Hudson paradigm,
      but our existing build system checks out a build ID file and increments a build
      counter along with saving the Changelist number being used to submit the
      change. This Changelist number is then used to synch the entire branch to a
      known moment in time. This approach was chosen over applying a label to avoid
      the overhead of applying 4-5000 labels to our source. Using the known
      Changelist eleminates mistakenly including any changes that occur after the
      know point in time.

      Is this possible using the Perforce plugin, or with any combination of existing
      plugins?

      Would this be a feasable/desirable enhancement for the existing Perforce plugin?

        Attachments

          Activity

          Hide
          tdrury tdrury added a comment -

          Here's further explanation of Scott's (and my) issue:

          We have a large, enterprise project that consists of about 100 maven modules.
          We would create a Hudson project from each module. The modules have
          inter-dependencies. We want to make sure all the modules are build from the
          same "snapshot in time". We could do this be labeling the code, but that's a
          lot of overhead. Perforce allows you to build from a changelist - presumably
          this is done via the time the changelist was submitted.

          Since this needs to apply to 100 or so Hudson projects simultaneously, setting
          this per-project is not feasible. What we need is a way to set this either
          globally, or for a group of projects.

          Can anyone think of any other solution to build a large number of maven projects
          in Hudson from the same point in time (or Perforce changelist)?

          Show
          tdrury tdrury added a comment - Here's further explanation of Scott's (and my) issue: We have a large, enterprise project that consists of about 100 maven modules. We would create a Hudson project from each module. The modules have inter-dependencies. We want to make sure all the modules are build from the same "snapshot in time". We could do this be labeling the code, but that's a lot of overhead. Perforce allows you to build from a changelist - presumably this is done via the time the changelist was submitted. Since this needs to apply to 100 or so Hudson projects simultaneously, setting this per-project is not feasible. What we need is a way to set this either globally, or for a group of projects. Can anyone think of any other solution to build a large number of maven projects in Hudson from the same point in time (or Perforce changelist)?
          Hide
          mdonohue mdonohue added a comment -

          It would be nice if this feature could be leveraged to solve issue 2975 as well.

          Show
          mdonohue mdonohue added a comment - It would be nice if this feature could be leveraged to solve issue 2975 as well.
          Hide
          rpetti Rob Petti added a comment -

          A work-around would be to have a single job do the syncing for all workspaces (if they are all on the same node, of course), then have that job trigger the rest to build from the pre-synced workspaces without syncing again.

          I'll take a look at the possibility of passing a perforce changeset as a parameter into and out of builds. This should allow you to set up a build stream that is locked to a single point in time.

          Show
          rpetti Rob Petti added a comment - A work-around would be to have a single job do the syncing for all workspaces (if they are all on the same node, of course), then have that job trigger the rest to build from the pre-synced workspaces without syncing again. I'll take a look at the possibility of passing a perforce changeset as a parameter into and out of builds. This should allow you to set up a build stream that is locked to a single point in time.
          Hide
          torbent torbent added a comment -

          I have a somewhat similar issue (in JENKINS-4928) where I also want to build two projects (in an up/downstream relationship) from the same changelist.
          My solution has been to use an "automatic label" (i.e. a label that refers specifically to a changelist number), but unfortunately the perforce plugin does not expect labels to change so its use is a bit "challenged" in that respect. Also, changes are not tracked.
          Our builds do work, and do sync to the correct changelist, but we miss the list of changes, and triggering cannot be done by SCM changes. We use up/downstream triggers instead.
          So while such a label would nicely solve your problem of building everything from the same changelist, it currently won't work ideally with Hudson ... You could have a "monitor job" which does nothing but update the label and trigger downstream jobs, which could then sync to the label (the plugin will do that). But the list of changes will only be in the monitor job.

          For the record, an automatic label looks a lot like this:
          Label: its-name
          Revison: @10000
          View: //whatever

          Show
          torbent torbent added a comment - I have a somewhat similar issue (in JENKINS-4928 ) where I also want to build two projects (in an up/downstream relationship) from the same changelist. My solution has been to use an "automatic label" (i.e. a label that refers specifically to a changelist number), but unfortunately the perforce plugin does not expect labels to change so its use is a bit "challenged" in that respect. Also, changes are not tracked. Our builds do work, and do sync to the correct changelist, but we miss the list of changes, and triggering cannot be done by SCM changes. We use up/downstream triggers instead. So while such a label would nicely solve your problem of building everything from the same changelist, it currently won't work ideally with Hudson ... You could have a "monitor job" which does nothing but update the label and trigger downstream jobs, which could then sync to the label (the plugin will do that). But the list of changes will only be in the monitor job. For the record, an automatic label looks a lot like this: Label: its-name Revison: @10000 View: //whatever
          Hide
          avolanis avolanis added a comment -

          We have a very similar environment to what scott and tdrury describe with upstream/downstream builds requiring to sync to a known changelist number. In our current home grown Maven base CI system we have been using quite successfully the perforce counters idea to manage the builds.

          I created a patch for the trunk version 1.18 of the plugin that works as follows and satisfies this need for our continuous integration builds.

          A new field "P4 Counter" is available to set the name of the counter. A checkbox controls if this is an upstream build which will update the value of the counter when it performs a successful sync or a downstream build which uses the value of the counter to perform a sync to the given changelist. It works very nicely for our needs.

          I also added an unrelated feature that I found I needed on occasion. The P4PASSWD value is set by the plugin in the environment as well as the other Perforce variables that are currently set. This is optional since most folks would consider a password in the environment a VERY bad idea but for us it was already the way we had setup our build environment and it only made sense to continue for now. This allows perforce commands embedded in script steps to succeed without requiring a "p4 login" on the slaves. I would prefer that this is not done but could not think of a way to make it happen transparently. Perhaps an option would be for the plugin to perform a "p4 login" on the slaves? But this could have unintended side-effects since the "p4 login" session would be shared across all jobs running on the slave concurrently and not all jobs are necessarily using the same Perforce credentials.

          I can post a patch diff if this is of interest to others or to the submitters so that this can be included in a future version of the plugin.

          Show
          avolanis avolanis added a comment - We have a very similar environment to what scott and tdrury describe with upstream/downstream builds requiring to sync to a known changelist number. In our current home grown Maven base CI system we have been using quite successfully the perforce counters idea to manage the builds. I created a patch for the trunk version 1.18 of the plugin that works as follows and satisfies this need for our continuous integration builds. A new field "P4 Counter" is available to set the name of the counter. A checkbox controls if this is an upstream build which will update the value of the counter when it performs a successful sync or a downstream build which uses the value of the counter to perform a sync to the given changelist. It works very nicely for our needs. I also added an unrelated feature that I found I needed on occasion. The P4PASSWD value is set by the plugin in the environment as well as the other Perforce variables that are currently set. This is optional since most folks would consider a password in the environment a VERY bad idea but for us it was already the way we had setup our build environment and it only made sense to continue for now. This allows perforce commands embedded in script steps to succeed without requiring a "p4 login" on the slaves. I would prefer that this is not done but could not think of a way to make it happen transparently. Perhaps an option would be for the plugin to perform a "p4 login" on the slaves? But this could have unintended side-effects since the "p4 login" session would be shared across all jobs running on the slave concurrently and not all jobs are necessarily using the same Perforce credentials. I can post a patch diff if this is of interest to others or to the submitters so that this can be included in a future version of the plugin.
          Hide
          rpetti Rob Petti added a comment -

          A patch for the solution to this issue will be much appreciated. I haven't had any time to really work on this, so I'll take all the help I can get.

          I would suggest opening a new issue for the P4PASSWD environment variable, and posting the diff for it there. It'll help keep everything organized.

          Show
          rpetti Rob Petti added a comment - A patch for the solution to this issue will be much appreciated. I haven't had any time to really work on this, so I'll take all the help I can get. I would suggest opening a new issue for the P4PASSWD environment variable, and posting the diff for it there. It'll help keep everything organized.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in hudson
          User: : rpetti
          Path:
          trunk/hudson/plugins/perforce/src/main/java/com/tek42/perforce/parse/AbstractPerforceTemplate.java
          trunk/hudson/plugins/perforce/src/main/java/hudson/plugins/perforce/PerforcePasswordEncryptor.java
          trunk/hudson/plugins/perforce/src/main/java/hudson/plugins/perforce/PerforceSCM.java
          trunk/hudson/plugins/perforce/src/main/resources/hudson/plugins/perforce/PerforceSCM/config.jelly
          trunk/hudson/plugins/perforce/src/main/webapp/help/exposeP4Passwd.html
          trunk/hudson/plugins/perforce/src/main/webapp/help/p4counter.html
          trunk/hudson/plugins/perforce/src/main/webapp/help/updateCounterValue.html
          trunk/hudson/plugins/perforce/src/test/java/hudson/plugins/perforce/PerforceSCMTest.java
          http://jenkins-ci.org/commit/26712
          Log:
          [FIXED JENKINS-4603] Applying patch provided by avolanis

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in hudson User: : rpetti Path: trunk/hudson/plugins/perforce/src/main/java/com/tek42/perforce/parse/AbstractPerforceTemplate.java trunk/hudson/plugins/perforce/src/main/java/hudson/plugins/perforce/PerforcePasswordEncryptor.java trunk/hudson/plugins/perforce/src/main/java/hudson/plugins/perforce/PerforceSCM.java trunk/hudson/plugins/perforce/src/main/resources/hudson/plugins/perforce/PerforceSCM/config.jelly trunk/hudson/plugins/perforce/src/main/webapp/help/exposeP4Passwd.html trunk/hudson/plugins/perforce/src/main/webapp/help/p4counter.html trunk/hudson/plugins/perforce/src/main/webapp/help/updateCounterValue.html trunk/hudson/plugins/perforce/src/test/java/hudson/plugins/perforce/PerforceSCMTest.java http://jenkins-ci.org/commit/26712 Log: [FIXED JENKINS-4603] Applying patch provided by avolanis
          Hide
          torbent torbent added a comment -

          Just a quick note (to self and others): The use of "p4 counter" requires that the Hudson Perforce user has "review" permissions. I'll see if I can grow one of those

          Show
          torbent torbent added a comment - Just a quick note (to self and others): The use of "p4 counter" requires that the Hudson Perforce user has "review" permissions. I'll see if I can grow one of those
          Hide
          rpetti Rob Petti added a comment -

          Another quick note: This can also be accomplished by parameter substitution in the Label field of the perforce configuration. That will cause the plugin to sync to whatever changeset or label is passed into the build. That way you don't need any additional permissions on the perforce server.

          Show
          rpetti Rob Petti added a comment - Another quick note: This can also be accomplished by parameter substitution in the Label field of the perforce configuration. That will cause the plugin to sync to whatever changeset or label is passed into the build. That way you don't need any additional permissions on the perforce server.
          Hide
          torbent torbent added a comment -

          Syncing to a label is not the problem; I think we do that for more than half of our jobs
          The problem is rather actually labeling a build so a downstream job can sync to the same point in "time".
          Right now, our build script runs a 'p4 label' command after the actual build. The label is always called the same, but moves forward in the changelist numbers. But as I've written in JENKINS-4928 this means that we lose changeset information downstream.
          I might try one or both of these:

          • Using a new label for all builds, and passing its name as a parameter downstream. A downside is the number of new labels this will result in. But will it give us changesets downstream?
          • Using this counter-idea will eliminate the labels. But will it give us changesets downstream?
            A question (although it sort of feels like hijacking the issue): When are the parameters expanded? It seems slightly magic. Can I e.g. use JOB_NAME in client name?
          Show
          torbent torbent added a comment - Syncing to a label is not the problem; I think we do that for more than half of our jobs The problem is rather actually labeling a build so a downstream job can sync to the same point in "time". Right now, our build script runs a 'p4 label' command after the actual build. The label is always called the same, but moves forward in the changelist numbers. But as I've written in JENKINS-4928 this means that we lose changeset information downstream. I might try one or both of these: Using a new label for all builds, and passing its name as a parameter downstream. A downside is the number of new labels this will result in. But will it give us changesets downstream? Using this counter-idea will eliminate the labels. But will it give us changesets downstream? A question (although it sort of feels like hijacking the issue): When are the parameters expanded? It seems slightly magic. Can I e.g. use JOB_NAME in client name?
          Hide
          rpetti Rob Petti added a comment -

          I didn't mean actually using labels, I meant passing the changeset in as a parameter in the Label configuration field. When syncing, it'll use p4 sync //workspace/...@${label} where ${label} is expanded appropriately. So it doesn't matter if label is set to a changeset, or an actual label.

          I don't think either of these methods will give you changelogs downstream, unfortunately. There's simply no way to accurately describe the state of the last sync when using labels or changesets. There might be a plugin for propagating changelogs downstream, however.

          Parameters are expanded in the View and the Label configuration fields only. The parameters that are expanded are the ones defined by the "Parameterized Build" functionality. This doesn't include the built-in ones (eg, JOB_NAME) as far as I know.

          Show
          rpetti Rob Petti added a comment - I didn't mean actually using labels, I meant passing the changeset in as a parameter in the Label configuration field. When syncing, it'll use p4 sync //workspace/...@${label} where ${label} is expanded appropriately. So it doesn't matter if label is set to a changeset, or an actual label. I don't think either of these methods will give you changelogs downstream, unfortunately. There's simply no way to accurately describe the state of the last sync when using labels or changesets. There might be a plugin for propagating changelogs downstream, however. Parameters are expanded in the View and the Label configuration fields only. The parameters that are expanded are the ones defined by the "Parameterized Build" functionality. This doesn't include the built-in ones (eg, JOB_NAME) as far as I know.
          Hide
          torbent torbent added a comment -

          Ah yes, I see. Passing changesets that way would probably be OK. I should try.

          I thought the 'p4 counter' solution might be able to provide changelogs. The job would know the value of the counter (same as changelist number) for every build and should be able to find all changes between those. Precisely like it does for the polling.
          Of course, if the plugin wants to be very defensive, I must admit that of course it's possible to play games with the counters and update them behind the scenes (even back in time). Although you technically can do that with changelist numbers as well, that series of numbers is always increasing.

          Show
          torbent torbent added a comment - Ah yes, I see. Passing changesets that way would probably be OK. I should try. I thought the 'p4 counter' solution might be able to provide changelogs. The job would know the value of the counter (same as changelist number) for every build and should be able to find all changes between those. Precisely like it does for the polling. Of course, if the plugin wants to be very defensive, I must admit that of course it's possible to play games with the counters and update them behind the scenes (even back in time). Although you technically can do that with changelist numbers as well, that series of numbers is always increasing.

            People

            • Assignee:
              rpetti Rob Petti
              Reporter:
              scottlavender scottlavender
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: