Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-13879

Builds failing to trigger due to changelist number being set incorrectly

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: perforce-plugin
    • Labels:
      None
    • Environment:
      Jenkins 1.462, Perforce plugin 1.3.13
    • Similar Issues:

      Description

      When a job is run, the Perforce plugin for Jenkins records a changelist number for the build instance. From that point forward, triggering additional builds via Perforce only happens when new changelists are submitted with numbers higher than the recorded changelist number. Great, except that the recorded changelist number may not be correct.

      For example, if the most recent changelists look like this (from "p4 changelists -m 2"):
      Change 24083 on 2012/05/23 by rmercer@rmercer-DcsMain pending 'The adapter layer now returns D'
      Change 24082 on 2012/05/23 by reladmin@jenkins-aci-RILCinterion-QA-2 'Check in server build result fo'

      and a build starts (either via the Build Now button, or because of 24082 being submitted), the Perforce plugin runs "p4 counter change" and records 24083 as the changelist number. Note that 24083 is a pending changelist.

      Further polling will fail to detect that 24083 was submitted. No build will result. Oops, the latest "reserved" changelist number was recorded, rather than the latest "submitted" changelist number.

      Often this problem is masked because new changelist numbers get generated upon a checkin, but not always. Perforce only renumbers a changelist upon checkin if there are intervening changelists.

      Suggested solution: Rather than using "p4 counter change" to detect the current changelist number, please use "p4 changelists -m 1 -s submitted".

        Attachments

          Activity

          Hide
          rpetti Rob Petti added a comment -

          It already uses 'p4 changes -m 1 -s'. 'p4 counter change' is only used when looking for the actual latest change in the workspace. Here's an example from one of my own build logs:

          [workspace] $ p4 counter change
          [workspace] $ p4 -s changes -s submitted //installer/...@260018,@260111
          [workspace] $ p4 describe -s 260103
          ...
          Sync'ing workspace to changelist 260103.
          [workspace] $ p4 -s sync //installer/...@260103
          

          'p4 counter change' returns 260111, but as you can see, it's using the last changeset that was submitted in the view for syncing.

          In the cases where you've seen different behavior, were there any changes made at all? I suspect it may be falling back onto the value provided by the counter when there are no changes available in the view, which is of course incorrect behavior.

          Show
          rpetti Rob Petti added a comment - It already uses 'p4 changes -m 1 -s'. 'p4 counter change' is only used when looking for the actual latest change in the workspace. Here's an example from one of my own build logs: [workspace] $ p4 counter change [workspace] $ p4 -s changes -s submitted //installer/...@260018,@260111 [workspace] $ p4 describe -s 260103 ... Sync'ing workspace to changelist 260103. [workspace] $ p4 -s sync //installer/...@260103 'p4 counter change' returns 260111, but as you can see, it's using the last changeset that was submitted in the view for syncing. In the cases where you've seen different behavior, were there any changes made at all? I suspect it may be falling back onto the value provided by the counter when there are no changes available in the view, which is of course incorrect behavior.
          Hide
          jfrank Jeremy Frank added a comment -

          Hi Rob,

          Thank you for the quick response.

          Your suspicion about the fallback is correct. Please consider these two situations:

          1. In the case that prompted this ticket yesterday, there was a build that failed due to a file-locking issue. Our release administrator fixed the problem and pressed Build Now to restart the build. "p4 counter change" reported 24075, but no new submitted changelists were seen, so 24075 was picked as the official changelist number for the build. Ten minutes later, developer Kevin checked in changelist 24075. You see, it was pending before, and now it is submitted. Since the perforce plugin thinks it already built 24075, it never finds Kevin's checkin.
          2. Even when submitted changelists are found, there is still a very subtle race condition here. If I may extend your example, say you had two view mappings: //installer/one/... and //installer/two/.... Now let's say changelist 260201 is pending on a file in //installer/one, and changelist 260202 is pending on a file in //installer/two. A build starts. "p4 counter change" says 260202. "p4 changes -s submitted //installer/one/...@260104,@260202" reports nothing. Then before the next call to "p4 changes" both changelists are checked in, and they are not renumbered. Now your "p4 changes -s submitted //installer/two/...@260104,@260202" finds changelist 260202. The build will contain both changes, yet the Jenkins build log only shows 260202. It missed the details of 260201 because of the race condition.

          Bottom line is, "p4 counter change" is completely useless because it contains pending changelists that may or may not become submitted changelists. If you use "p4 changelists -m 1 -s submitted" (with no path!) in place of "p4 counter change" I believe these problems go away. You will get the latest submitted changelist number, and you are guaranteed to never have any new changelist submitted with a lower number.

          I look forward to your thoughts on this.

          Show
          jfrank Jeremy Frank added a comment - Hi Rob, Thank you for the quick response. Your suspicion about the fallback is correct. Please consider these two situations: In the case that prompted this ticket yesterday, there was a build that failed due to a file-locking issue. Our release administrator fixed the problem and pressed Build Now to restart the build. " p4 counter change " reported 24075, but no new submitted changelists were seen, so 24075 was picked as the official changelist number for the build. Ten minutes later, developer Kevin checked in changelist 24075. You see, it was pending before, and now it is submitted. Since the perforce plugin thinks it already built 24075, it never finds Kevin's checkin.   Even when submitted changelists are found, there is still a very subtle race condition here. If I may extend your example, say you had two view mappings: //installer/one/... and //installer/two/... . Now let's say changelist 260201 is pending on a file in //installer/one , and changelist 260202 is pending on a file in //installer/two . A build starts. " p4 counter change " says 260202. " p4 changes -s submitted //installer/one/...@260104,@260202 " reports nothing. Then before the next call to " p4 changes " both changelists are checked in, and they are not renumbered. Now your " p4 changes -s submitted //installer/two/...@260104,@260202 " finds changelist 260202. The build will contain both changes, yet the Jenkins build log only shows 260202. It missed the details of 260201 because of the race condition. Bottom line is, " p4 counter change " is completely useless because it contains pending changelists that may or may not become submitted changelists. If you use " p4 changelists -m 1 -s submitted " (with no path!) in place of " p4 counter change " I believe these problems go away. You will get the latest submitted changelist number, and you are guaranteed to never have any new changelist submitted with a lower number. I look forward to your thoughts on this.
          Hide
          rpetti Rob Petti added a comment -

          The API doesn't currently support fetching changelists without specifying a path, so I was going to use "p4 changelists -m 1 -s submitted //..." instead. Does that sound good?

          Show
          rpetti Rob Petti added a comment - The API doesn't currently support fetching changelists without specifying a path, so I was going to use "p4 changelists -m 1 -s submitted //..." instead. Does that sound good?
          Hide
          jfrank Jeremy Frank added a comment -

          Interesting, the API must have changed somewhere along the way. (It is accepted by my Perforce setup.)

          Anyway, yes, using //... sounds the same to me.

          Thank you again!
          --jsf

          Show
          jfrank Jeremy Frank added a comment - Interesting, the API must have changed somewhere along the way. (It is accepted by my Perforce setup.) Anyway, yes, using //... sounds the same to me. Thank you again! --jsf
          Hide
          rpetti Rob Petti added a comment -

          We're using the tek42 perforce api. The raw command itself still works, but there's currently no way to run it using the api.

          I'll check in the fix after I've tested it a bit.

          Show
          rpetti Rob Petti added a comment - We're using the tek42 perforce api. The raw command itself still works, but there's currently no way to run it using the api. I'll check in the fix after I've tested it a bit.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Rob Petti
          Path:
          src/main/java/hudson/plugins/perforce/PerforceSCM.java
          http://jenkins-ci.org/commit/perforce-plugin/33720a12967b605ca3ebdcf39e735b17b642a174
          Log:
          JENKINS-13879 use the latest submitted depot change instead of counter when there are no changes to record

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Rob Petti Path: src/main/java/hudson/plugins/perforce/PerforceSCM.java http://jenkins-ci.org/commit/perforce-plugin/33720a12967b605ca3ebdcf39e735b17b642a174 Log: JENKINS-13879 use the latest submitted depot change instead of counter when there are no changes to record
          Hide
          rpetti Rob Petti added a comment -

          Released in 1.3.15.

          Show
          rpetti Rob Petti added a comment - Released in 1.3.15.
          Hide
          jasonwang Jian Wang added a comment -

          I tried to use multiple SCM plugin to access to two different perforce servers in one jenkins job, wrote script code in powershell. Every time I build my job, it will automatically sync the work space, since my job contains two perforce clients, it need to sync from two servers. But the changeset number is always the first server's, so when I came to the second server, there will be an issue, the second server has a much larger change set, it will take a long time to describe the change because the changeset number is from the first server, at last I met the OutOfMemory error:

          Building in workspace C:\jenkins\jobs\VCOPS_PLATFORM_AUTOTEST\workspace
          Using master perforce client: VCOPSPLAT_STATS_SANDBOX
          [workspace] $ C:\perforce-r10.2\p4.exe workspace -o VCOPSPLAT_STATS_SANDBOX

          Last build changeset: 129940

          [workspace] $ C:\perforce-r10.2\p4.exe changes -s submitted -m 1 //...
          [workspace] $ C:\perforce-r10.2\p4.exe -s changes -s submitted //VCOPSPLAT_STATS_SANDBOX/...@129941,@129982
          Sync'ing workspace to changelist 129982.
          [workspace] $ C:\perforce-r10.2\p4.exe -s sync //VCOPSPLAT_STATS_SANDBOX/...@129982
          Sync complete, took 364 ms
          Using master perforce client: VCOPSPLAT_CLOUDVM_SANDBOX
          [workspace] $ C:\perforce-r10.2\p4.exe workspace -o VCOPSPLAT_CLOUDVM_SANDBOX

          Last build changeset: 129940

          [workspace] $ C:\perforce-r10.2\p4.exe changes -s submitted -m 1 //...
          [workspace] $ C:\perforce-r10.2\p4.exe -s changes -s submitted //VCOPSPLAT_CLOUDVM_SANDBOX/...@129941,@1934948
          [workspace] $ C:\perforce-r10.2\p4.exe describe -s 1934844
          [workspace] $ C:\perforce-r10.2\p4.exe -G where //...
          [workspace] $ C:\perforce-r10.2\p4.exe describe -s 1934765
          ...

          Is this due to perforce plugin or multiple SCM plugin?

          Show
          jasonwang Jian Wang added a comment - I tried to use multiple SCM plugin to access to two different perforce servers in one jenkins job, wrote script code in powershell. Every time I build my job, it will automatically sync the work space, since my job contains two perforce clients, it need to sync from two servers. But the changeset number is always the first server's, so when I came to the second server, there will be an issue, the second server has a much larger change set, it will take a long time to describe the change because the changeset number is from the first server, at last I met the OutOfMemory error: Building in workspace C:\jenkins\jobs\VCOPS_PLATFORM_AUTOTEST\workspace Using master perforce client: VCOPSPLAT_STATS_SANDBOX [workspace] $ C:\perforce-r10.2\p4.exe workspace -o VCOPSPLAT_STATS_SANDBOX Last build changeset: 129940 [workspace] $ C:\perforce-r10.2\p4.exe changes -s submitted -m 1 //... [workspace] $ C:\perforce-r10.2\p4.exe -s changes -s submitted //VCOPSPLAT_STATS_SANDBOX/...@129941,@129982 Sync'ing workspace to changelist 129982. [workspace] $ C:\perforce-r10.2\p4.exe -s sync //VCOPSPLAT_STATS_SANDBOX/...@129982 Sync complete, took 364 ms Using master perforce client: VCOPSPLAT_CLOUDVM_SANDBOX [workspace] $ C:\perforce-r10.2\p4.exe workspace -o VCOPSPLAT_CLOUDVM_SANDBOX Last build changeset: 129940 [workspace] $ C:\perforce-r10.2\p4.exe changes -s submitted -m 1 //... [workspace] $ C:\perforce-r10.2\p4.exe -s changes -s submitted //VCOPSPLAT_CLOUDVM_SANDBOX/...@129941,@1934948 [workspace] $ C:\perforce-r10.2\p4.exe describe -s 1934844 [workspace] $ C:\perforce-r10.2\p4.exe -G where //... [workspace] $ C:\perforce-r10.2\p4.exe describe -s 1934765 ... Is this due to perforce plugin or multiple SCM plugin?

            People

            • Assignee:
              rpetti Rob Petti
              Reporter:
              jfrank Jeremy Frank
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: