Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36883

Jobs with “Exclude Changes” Polling Filter Not Syncing Latest at Build Time

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: p4-plugin
    • Labels:
      None
    • Environment:
      P4 2015.2 (Win 2008 R@), Jenkins 2.14 (Win 2008 R2), P4-Plugin 1.4.3
    • Similar Issues:

      Description

      I’ve found that the latest versions of files that are mapped in a workspace and also in an “Exclude changes from Depot path” polling filter, are not being synced at build time.
      Here’s the scenario. I’m currently set Jobs to use Manual workspaces set in Jenkins, to do a Force Sync, and have the “Populate have list” flag set.

      The Workspace includes an example path:
      PathA/…
      PathB/…

      Also configured with the Polling Filter - “Exclude changes from Depot path” is set to a path under PathB (actually in my test it was the same path).

      What happens:

      1. A check-in under Path A is recognized from polling and triggers a build which has a quiet period associated with it.
      2. During the quiet period, one or more check-ins occur under Path B. These files naturally have a higher changelist number than what triggered the build.
      3. When the quiet period expires, the build starts and syncs Path A’s files to latest available changelist for that Path, whatever it may be – All is good!
      4. For the files in Path B, however, the build syncs to the changelist number from Path A that triggered the build, NOT the latest available changelist. That causes the files under Path B, to be sync’ed to non-latest state.
      5. Again, if check-ins occur in the quiet period under Path A – the latest versions of those files ARE being synced.

      For me, I have files under Path B that must be at latest when the build kicks off. I don’t, however, want to trigger builds with these changes, because unrelated changes occur here all day long. This behavior of sync’ing old files in that path is actually breaking our builds and forcing us to stay at version 1.3.8 which prevents us from using the latest version of the plugin (1.3.9 is where this issues was first noticed).

      Btw – Polling does recognize the changelists in Path B (See below - in my test 4938865 and 4938866 are under Path B), but they are not being synced to the workspace. 4938864 is the change in Path A that triggered the build – that is being synced AND the files in Path B are being synced to the 4938864 level also. The two Path B changes are also not listed/described on the Changes page for the job.

      I can see this maybe being by design, but the Exclude Filter specifically says it’s for polling – not syncing files, plus syncing the latest files from all paths at build time and using an Exclude in Polling path worked up through 1.3.8.

      Started on Jul 22, 2016 9:47:01 AM
      Polling SCM changes on master
      P4: Polling on: master with:Test-Perforce-Plugin
      ... p4 client -o Test-Perforce-Plugin
       +
      ... p4 info
       +
      
      P4 Task: establishing connection.
      ... server: atl-p4-test:1818
      ... node: atl-build-test
      P4: Polling with range: 4938863,now
      ... p4 changes -m100 //Test-Perforce-Plugin/...@4938863,now
       +
      ... p4 change -o 4938866
       +
      ... p4 describe -s 4938866
       +
      ... p4 change -o 4938865
       +
      ... p4 describe -s 4938865
       +
      ... p4 change -o 4938864
       +
      ... p4 describe -s 4938864
       +
      ... found change: 4938864
      Done. Took 0.18 sec
      Changes found
      

        Attachments

          Activity

          Hide
          p4paul Paul Allen added a comment -

          Think I am misunderstanding something, or your environment is different to mine. All files should get sync'ed to @now (the latest change). If you look in the log output you should see p4 sync //<workspace>/...@now.

          "Exclude Changes from Depot Path" is only a polling filter to prevent build events from occurring on changes in the specified path.

          Show
          p4paul Paul Allen added a comment - Think I am misunderstanding something, or your environment is different to mine. All files should get sync'ed to @now (the latest change). If you look in the log output you should see p4 sync //<workspace>/...@now. "Exclude Changes from Depot Path" is only a polling filter to prevent build events from occurring on changes in the specified path.
          Hide
          jedavis Jason Davis added a comment -

          Hi Paul - I agree with you that the polling filter should only prevent builds from occurring on the changes in the filter path! That feature is working wonderfully, but there’s something else going on that’s causing files in the workspace spec to be sync’ed to _different _changelist levels after the Jenkins Quiet Period - and that started happening with v1.3.9 of the plugin. Up through 1.3.8, ALL files in a workspace spec did sync to the latest CL after a Quiet Period at build start time (without @now) – but that’s not happening any longer.

          Please see the above example. I explain there that, right now and without the use of @now in the workspace spec or P4 label setting, non-filtered files are currently being sync'ed to latest (CL 105), while filtered files are being sync'ed to trigger level (CL 100 instead of CL 103). I’ve confirmed this by testing with the above spec in a simple Jenkins job. Just make some checkins to files both in and out of a filtered path during a Quiet Period to see that they’re not synced to the latest CL at build start time.

          You suggested above in the Sept. 8 comment, that the build should sync to the CL at trigger time - not the CL at start time, so that other changes don't sneak in to the changes report. Also that @now should be used as a workaround. I’m further suggesting that @now shouldn’t be required to make the filtered files sync to latest at build start, because Jenkins has a configurable Quiet Period setting, which by design is to enable Devs to submit other changes after a trigger as an "oops" measure. If @now becomes required in the P4 Plugin label field or workspace spec to sync to latest after a Quiet Period, then that prevents use of other P4 labels I might like to use, essentially states that more configuration is required to use the Jenkins Quiet Period setting, would make the P4 plugin work by differently than all other SCM plugins I’m familiar with since I started using Jenkins around 8-9 years ago (TFS, SVN, Community Perforce), and the P4 plugin itself did not have this requirement until v1.3.9.

          In the end, I'm first hoping that ALL the files in the workspace spec are synced to the same and latest CL at build start time like it once did (and that’s the same CL that the changes also report uses) – and second, that the build should not require @now to deal with those changes checked in during the Jenkins Quiet Period because I believe that alters what one would expect from SCM plugins when using a Quiet Period (as well as other reasons mentioned in the above comments).

          Show
          jedavis Jason Davis added a comment - Hi Paul - I agree with you that the polling filter should only prevent builds from occurring on the changes in the filter path! That feature is working wonderfully, but there’s something else going on that’s causing files in the workspace spec to be sync’ed to _different _changelist levels after the Jenkins Quiet Period - and that started happening with v1.3.9 of the plugin. Up through 1.3.8, ALL files in a workspace spec did sync to the latest CL after a Quiet Period at build start time (without @now) – but that’s not happening any longer. Please see the above example. I explain there that, right now and without the use of @now in the workspace spec or P4 label setting, non-filtered files are currently being sync'ed to latest (CL 105), while filtered files are being sync'ed to trigger level (CL 100 instead of CL 103). I’ve confirmed this by testing with the above spec in a simple Jenkins job. Just make some checkins to files both in and out of a filtered path during a Quiet Period to see that they’re not synced to the latest CL at build start time. You suggested above in the Sept. 8 comment, that the build should sync to the CL at trigger time - not the CL at start time, so that other changes don't sneak in to the changes report. Also that @now should be used as a workaround. I’m further suggesting that @now shouldn’t be required to make the filtered files sync to latest at build start, because Jenkins has a configurable Quiet Period setting, which by design is to enable Devs to submit other changes after a trigger as an "oops" measure. If @now becomes required in the P4 Plugin label field or workspace spec to sync to latest after a Quiet Period, then that prevents use of other P4 labels I might like to use, essentially states that more configuration is required to use the Jenkins Quiet Period setting, would make the P4 plugin work by differently than all other SCM plugins I’m familiar with since I started using Jenkins around 8-9 years ago (TFS, SVN, Community Perforce), and the P4 plugin itself did not have this requirement until v1.3.9. In the end, I'm first hoping that ALL the files in the workspace spec are synced to the same and latest CL at build start time like it once did (and that’s the same CL that the changes also report uses) – and second, that the build should not require @now to deal with those changes checked in during the Jenkins Quiet Period because I believe that alters what one would expect from SCM plugins when using a Quiet Period (as well as other reasons mentioned in the above comments).
          Hide
          p4karl Karl Wirth added a comment -

          Have reproduced this (ignoring the 'now' label for this example). Setup a polling period of 2 minutes but a Quiet setting on the job of 3 minutes. The job will run with the latest polled changelist of the 2 polls. For example:

          • Poll 1 determines changelist 256 is the latest - triggers job#36 with a quiet period of 3 mins.
          • 3 more changes made after quiet period starts and before next poll period. CL=260 on included branch, 261 and 262 on excluded from poll branch.
          • Poll 2 determines that changelist 260 is the latest.
          • Job#36 runs with changelist 260 as the latest changelist.

          So the `bug` is that the running job picks up the latest poll CL not the poll CL that was available when the job was started. Note that this is not the latest CL in the workspace because there were later CLs on the branch that was excluded from the polling filter. Attaching JENKINS-36883-repro.txt full log output.

          The desired behavior of putting 'now' in the 'Pin build at Perforce label' field would get up to 262 which is the desired behavior.

          Show
          p4karl Karl Wirth added a comment - Have reproduced this (ignoring the 'now' label for this example). Setup a polling period of 2 minutes but a Quiet setting on the job of 3 minutes. The job will run with the latest polled changelist of the 2 polls. For example: Poll 1 determines changelist 256 is the latest - triggers job#36 with a quiet period of 3 mins. 3 more changes made after quiet period starts and before next poll period. CL=260 on included branch, 261 and 262 on excluded from poll branch. Poll 2 determines that changelist 260 is the latest. Job#36 runs with changelist 260 as the latest changelist. So the `bug` is that the running job picks up the latest poll CL not the poll CL that was available when the job was started. Note that this is not the latest CL in the workspace because there were later CLs on the branch that was excluded from the polling filter. Attaching JENKINS-36883-repro.txt full log output. The desired behavior of putting 'now' in the 'Pin build at Perforce label' field would get up to 262 which is the desired behavior.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Paul Allen
          Path:
          src/main/java/org/jenkinsci/plugins/p4/PerforceScm.java
          src/main/java/org/jenkinsci/plugins/p4/changes/P4Revision.java
          src/main/java/org/jenkinsci/plugins/p4/client/ClientHelper.java
          src/main/java/org/jenkinsci/plugins/p4/tasks/AbstractTask.java
          src/main/java/org/jenkinsci/plugins/p4/tasks/CheckoutTask.java
          src/main/java/org/jenkinsci/plugins/p4/tasks/PollTask.java
          src/test/java/org/jenkinsci/plugins/p4/client/ConnectionTest.java
          http://jenkins-ci.org/commit/p4-plugin/f4c24f7964bd1de847ace7b43546be81e51d2ccc
          Log:
          Polling Fix for use with quiet period.

          Switched all uses of change/label to P4Revision object and implemented
          Comparable. The changes to build are now calculated at build time
          (after the quiet period) not during the polling phase.

          JENKINS-36883

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Paul Allen Path: src/main/java/org/jenkinsci/plugins/p4/PerforceScm.java src/main/java/org/jenkinsci/plugins/p4/changes/P4Revision.java src/main/java/org/jenkinsci/plugins/p4/client/ClientHelper.java src/main/java/org/jenkinsci/plugins/p4/tasks/AbstractTask.java src/main/java/org/jenkinsci/plugins/p4/tasks/CheckoutTask.java src/main/java/org/jenkinsci/plugins/p4/tasks/PollTask.java src/test/java/org/jenkinsci/plugins/p4/client/ConnectionTest.java http://jenkins-ci.org/commit/p4-plugin/f4c24f7964bd1de847ace7b43546be81e51d2ccc Log: Polling Fix for use with quiet period. Switched all uses of change/label to P4Revision object and implemented Comparable. The changes to build are now calculated at build time (after the quiet period) not during the polling phase. JENKINS-36883
          Hide
          jedavis Jason Davis added a comment - - edited

          Verified fixed in 1.4.8. Thank you!

          Show
          jedavis Jason Davis added a comment - - edited Verified fixed in 1.4.8. Thank you!

            People

            • Assignee:
              p4paul Paul Allen
              Reporter:
              jedavis Jason Davis
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: