Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17652

Poll Exclude Files Too Aggressive on Blocking Builds?

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: perforce-plugin
    • Labels:
      None
    • Environment:
      Jenkins: 1.511
      Perforce Server Version: 2012.1
      Java 1.7.0_17
      Perforce Plugin: 1.3.21
      Server: Windows 2008 R2
    • Similar Issues:

      Description

      Caveat - It's definitely possible that I don't understand how Poll Exclude Files should work. It seems like changes that should trigger a build when Poll Exclude Files are provided are not. I currently have this in my Poll Exclude Files section (all paths are in the Client Spec):

      //<afilepath>/VersionInfo.cs
      //<afilepath>/VersionInfo.rc2
      "//<asecondfilepath>/config/buildinfo/..."

      Below is the polling log for one of my builds. I would have expected that CS 184776 would have triggered the build, however, polling reports that it is an excluded file.

      Exclude file(s) found:
      //<asecondfilepath>/UTandCover.fbp7
      Changelist 184776 is composed of file(s) and/or user(s) that are excluded.

      I thought I had excluded all "buildinfo/..." files which are in a different subdirectory, but maybe I can't exclude files when they are in the same root dir?

      Thanks

      --------------------------------------------------------------

      Started on Apr 17, 2013 8:04:53 AM
      Looking for changes...
      Using node: <blah>
      Using remote perforce client: <blah>
      [jenkins] $ "C:\Program Files\Perforce\p4.exe" workspace -o <blah>
      'Don't update client' is set. Not saving the client changes.
      [jenkins] $ "C:\Program Files\Perforce\p4.exe" counter change
      [jenkins] $ "C:\Program Files\Perforce\p4.exe" -s changes -s submitted //<blah>/...@184734,@184829
      Latest submitted change selected by workspace is 184807
      [jenkins] $ "C:\Program Files\Perforce\p4.exe" describe -s 184807
      [jenkins] $ "C:\Program Files\Perforce\p4.exe" -G where //...
      Exclude file(s) found:
      //<afilepath>/VersionInfo.rc2
      Changelist 184807 is composed of file(s) and/or user(s) that are excluded.
      [jenkins] $ "C:\Program Files\Perforce\p4.exe" describe -s 184806
      Exclude file(s) found:
      //<afilepath>/VersionInfo.cs
      Changelist 184806 is composed of file(s) and/or user(s) that are excluded.
      [jenkins] $ "C:\Program Files\Perforce\p4.exe" describe -s 184776
      Exclude file(s) found:
      //<asecondfilepath>/UTandCover.fbp7
      Changelist 184776 is composed of file(s) and/or user(s) that are excluded.
      [jenkins] $ "C:\Program Files\Perforce\p4.exe" describe -s 184754
      Exclude file(s) found:
      //<asecondfilepath>/config/buildinfo/Item3.17.0.0.buildinfo
      Changelist 184754 is composed of file(s) and/or user(s) that are excluded.
      [jenkins] $ "C:\Program Files\Perforce\p4.exe" describe -s 184751
      Exclude file(s) found:
      //<asecondfilepath>/config/buildinfo/Item2.17.0.0.buildinfo
      Changelist 184751 is composed of file(s) and/or user(s) that are excluded.
      [jenkins] $ "C:\Program Files\Perforce\p4.exe" describe -s 184748
      Exclude file(s) found:
      //<asecondfilepath>/config/buildinfo/Item1.17.0.0.buildinfo
      Changelist 184748 is composed of file(s) and/or user(s) that are excluded.
      [jenkins] $ "C:\Program Files\Perforce\p4.exe" describe -s 184746
      Exclude file(s) found:
      //<afilepath>/VersionInfo.rc2
      Changelist 184746 is composed of file(s) and/or user(s) that are excluded.
      [jenkins] $ "C:\Program Files\Perforce\p4.exe" describe -s 184745
      Exclude file(s) found:
      //<afilepath>/VersionInfo.cs
      Changelist 184745 is composed of file(s) and/or user(s) that are excluded.
      Done. Took 5.9 sec
      No changes

        Attachments

          Activity

          Hide
          rpetti Rob Petti added a comment -

          I believe your understanding is correct, and that definitely doesn't look right to me either.

          Show
          rpetti Rob Petti added a comment - I believe your understanding is correct, and that definitely doesn't look right to me either.
          Hide
          rpetti Rob Petti added a comment -

          Can you try without the quotes?

          Show
          rpetti Rob Petti added a comment - Can you try without the quotes?
          Hide
          jedavis Jason Davis added a comment - - edited

          Yes - I just now did and unfortunately the files I'm trying to exclude have spaces in the Perforce path. When I take away the quotes, I'm warned about the path not being a valid Perforce path and it looks like Polling stops altogether.

          I tested this with another project that was only excluding files in this path:

          "//<asecondfilepath>/config/buildinfo/..."

          To possibly test if quoted Perforce paths are what's causing the problem, I also just now took the above path out of another build and left only:

          //<afilepath>/VersionInfo.cs

          in the exclude files section. I'll have to wait for awhile to see if the exclusions start allowing builds through again.

          Show
          jedavis Jason Davis added a comment - - edited Yes - I just now did and unfortunately the files I'm trying to exclude have spaces in the Perforce path. When I take away the quotes, I'm warned about the path not being a valid Perforce path and it looks like Polling stops altogether. I tested this with another project that was only excluding files in this path: "//<asecondfilepath>/config/buildinfo/..." To possibly test if quoted Perforce paths are what's causing the problem, I also just now took the above path out of another build and left only: //<afilepath>/VersionInfo.cs in the exclude files section. I'll have to wait for awhile to see if the exclusions start allowing builds through again.
          Hide
          jedavis Jason Davis added a comment -

          Just checked polling on a build that has this single row in the exclude files section:

          //<afilepath>/VersionInfo.cs

          However the build is not being triggered when it finds changes in other paths:

          Exclude file(s) found:
          	//<asecondfilepath>/ScriptBox/Help.fbp7	
                  //<asecondfilepath>/LaunchChecker.fbp7
          Changelist 185127 is composed of file(s) and/or user(s) that are excluded.
          

          Is it correct that even if VersionInfo.cs was in the list of excluded files, these changes should still trigger a build? btw - In this actual build, VersionInfo.cs isn't even in the list of changes that the build is currently aware of.

          Show
          jedavis Jason Davis added a comment - Just checked polling on a build that has this single row in the exclude files section: //<afilepath>/VersionInfo.cs However the build is not being triggered when it finds changes in other paths: Exclude file(s) found: //<asecondfilepath>/ScriptBox/Help.fbp7 //<asecondfilepath>/LaunchChecker.fbp7 Changelist 185127 is composed of file(s) and/or user(s) that are excluded. Is it correct that even if VersionInfo.cs was in the list of excluded files, these changes should still trigger a build? btw - In this actual build, VersionInfo.cs isn't even in the list of changes that the build is currently aware of.
          Hide
          rpetti Rob Petti added a comment -

          To clarify, it should only ever skip a changeset if every file in it is excluded. If it contains a file that is not excluded, then it will not skip the change.

          In this case it appears to be mistakenly excluding every single file as if you had put //... as your excluded path.

          Does <asecondfilepath> or <afilepath> contain any non-alphanumeric characters?

          Show
          rpetti Rob Petti added a comment - To clarify, it should only ever skip a changeset if every file in it is excluded. If it contains a file that is not excluded, then it will not skip the change. In this case it appears to be mistakenly excluding every single file as if you had put //... as your excluded path. Does <asecondfilepath> or <afilepath> contain any non-alphanumeric characters?
          Hide
          jedavis Jason Davis added a comment -

          Thank you for the clarification. Neither <asecondfilepath> nor <afilepath> include non-alphanumeric characters. The paths are in completely separate locations in the source tree sharing only the topmost level of the path. <asecondfilepath> does contain a space so I have to quote the path in the exclude list.

          Thanks for looking into this!

          Show
          jedavis Jason Davis added a comment - Thank you for the clarification. Neither <asecondfilepath> nor <afilepath> include non-alphanumeric characters. The paths are in completely separate locations in the source tree sharing only the topmost level of the path. <asecondfilepath> does contain a space so I have to quote the path in the exclude list. Thanks for looking into this!
          Hide
          rpetti Rob Petti added a comment - - edited

          So a couple other things to check:

          -The exclude paths are depot paths, not workspace paths. I suspect you are using depot paths already, but I just wanted to double check.

          -Are you using anything odd in your view spec? It should look like:
          //depot/project/a/... //workspace/a/...
          //depot/project/b/*.java //workspace/b/*.java

          Only ... and * are currently supported as matchers. Named matchers (%%1, %%2, etc) don't work yet and will break the file exclusion code, since it will think all files are outside of the current view.

          Show
          rpetti Rob Petti added a comment - - edited So a couple other things to check: -The exclude paths are depot paths, not workspace paths. I suspect you are using depot paths already, but I just wanted to double check. -Are you using anything odd in your view spec? It should look like: //depot/project/a/... //workspace/a/... //depot/project/b/*.java //workspace/b/*.java Only ... and * are currently supported as matchers. Named matchers (%%1, %%2, etc) don't work yet and will break the file exclusion code, since it will think all files are outside of the current view.
          Hide
          jedavis Jason Davis added a comment -

          Thanks again for checking this out. Yes - I'm just using depot paths. Also the standard matchers - no %1 or %2 type matching. The paths I'm using are just text paths. ex. //depot/item/path/item/... I just replaced them above in the examples to avoid showing some of the product information. In one case it's a full path to a file, in the other, it's all files in the directory using /...

          Thanks

          Show
          jedavis Jason Davis added a comment - Thanks again for checking this out. Yes - I'm just using depot paths. Also the standard matchers - no %1 or %2 type matching. The paths I'm using are just text paths. ex. //depot/item/path/item/... I just replaced them above in the examples to avoid showing some of the product information. In one case it's a full path to a file, in the other, it's all files in the directory using /... Thanks
          Hide
          rpetti Rob Petti added a comment -

          To clarify, I'm asking about the view, not the exclude paths.

          Show
          rpetti Rob Petti added a comment - To clarify, I'm asking about the view, not the exclude paths.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Rob Petti
          Path:
          src/main/java/hudson/plugins/perforce/PerforceSCM.java
          http://jenkins-ci.org/commit/perforce-plugin/9f69472ee3b54f44ab23a42dd9d28c42ebda2262
          Log:
          JENKINS-17652 adding pattern matching support for views containing %%1 style tokens

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Rob Petti Path: src/main/java/hudson/plugins/perforce/PerforceSCM.java http://jenkins-ci.org/commit/perforce-plugin/9f69472ee3b54f44ab23a42dd9d28c42ebda2262 Log: JENKINS-17652 adding pattern matching support for views containing %%1 style tokens
          Hide
          jedavis Jason Davis added a comment -

          I see. Yup - I definitely thought you meant the spec area in Jenkins for the exclude paths.

          There are no replacements in any of the Workspace specs that I've setup. One of them has a few eliminated mappings: -//apath/item/... and it looks like all of them have at least one quoted mapping "//a path/item/..." The mappings all also include files from more than one depot.

          The mappings in 95% of the builds exhibiting the problem look like this:

          //depot/item/item/... //SpecName/...
          "//depot2/path item/item1/..." "//SpecName/path item/item1/..."
          //depot2/path2/item2/... //SpecName/path2/item2/...

          I think those are the only slightly unusual thing that I have going on. Maybe this is helpful... It's files in the second depot that I'm excluding. However, I'm also making other checkins in that second depot that I would think would trigger the build, because they're not being excluded. The plugin appears to be determining them to be excluded anyway.

          Show
          jedavis Jason Davis added a comment - I see. Yup - I definitely thought you meant the spec area in Jenkins for the exclude paths. There are no replacements in any of the Workspace specs that I've setup. One of them has a few eliminated mappings: -//apath/item/... and it looks like all of them have at least one quoted mapping "//a path/item/..." The mappings all also include files from more than one depot. The mappings in 95% of the builds exhibiting the problem look like this: //depot/item/item/... //SpecName/... "//depot2/path item/item1/..." "//SpecName/path item/item1/..." //depot2/path2/item2/... //SpecName/path2/item2/... I think those are the only slightly unusual thing that I have going on. Maybe this is helpful... It's files in the second depot that I'm excluding. However, I'm also making other checkins in that second depot that I would think would trigger the build, because they're not being excluded. The plugin appears to be determining them to be excluded anyway.
          Hide
          rpetti Rob Petti added a comment -

          The files in the second depot you are referring to, are those in "//depot/path item/item1/..." by chance?

          Show
          rpetti Rob Petti added a comment - The files in the second depot you are referring to, are those in "//depot/path item/item1/..." by chance?
          Hide
          jedavis Jason Davis added a comment - - edited

          The excluded files and subdirs themselves only exist in the second depot. The first depot DOES have a similar upper portion of the path, however, we don't overlap the two in our Workspace specs.

          So yes, depot1 contains a similar set of upper paths and they diverge, but we don't map them together in the workspace. If we had mapped the two together, the actual excluded subdirs and files only exist in depot2 - there wouldn't be any collisions.

          So for ex. this exists in depot1 (the "ItemsX" share the same names in our Perforce):

          "//depot1/ITEMA/Item B/..."

          and depot2:

          "//ITEMA/Item B/<newfoldernotinfirstdepot>/config/buildinfo/..."

          ITEMA is the name of our second depot.

          Show
          jedavis Jason Davis added a comment - - edited The excluded files and subdirs themselves only exist in the second depot. The first depot DOES have a similar upper portion of the path, however, we don't overlap the two in our Workspace specs. So yes, depot1 contains a similar set of upper paths and they diverge, but we don't map them together in the workspace. If we had mapped the two together, the actual excluded subdirs and files only exist in depot2 - there wouldn't be any collisions. So for ex. this exists in depot1 (the "ItemsX" share the same names in our Perforce): "//depot1/ITEMA/Item B/..." and depot2: "//ITEMA/Item B/<newfoldernotinfirstdepot>/config/buildinfo/..." ITEMA is the name of our second depot.
          Hide
          rpetti Rob Petti added a comment -

          So to clarify, all the files that were mistakenly marked as excluded are in paths that contain a space?

          Show
          rpetti Rob Petti added a comment - So to clarify, all the files that were mistakenly marked as excluded are in paths that contain a space?
          Hide
          jedavis Jason Davis added a comment -

          Hmmm... I can't say for certain, because I didn't think to keep a history of the past polling issues, but for the 4 or so builds that should have built at this very moment - that is true. All the files have been mistakenly marked as being excluded are currently in a path with spaces.

          The files happen to share a common parent folder and I'm exluding files in the "buildinfo" folder.

          //depot2/Item Path1/path2/UTandCover.fbp7 (A checkin occurred on this file)

          The exclude spec has this:

          //depot2/Item Path1/path2/config/buildinfo/...

          Show
          jedavis Jason Davis added a comment - Hmmm... I can't say for certain, because I didn't think to keep a history of the past polling issues, but for the 4 or so builds that should have built at this very moment - that is true. All the files have been mistakenly marked as being excluded are currently in a path with spaces. The files happen to share a common parent folder and I'm exluding files in the "buildinfo" folder. //depot2/Item Path1/path2/UTandCover.fbp7 (A checkin occurred on this file) The exclude spec has this: //depot2/Item Path1/path2/config/buildinfo/...
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Rob Petti
          Path:
          src/main/java/hudson/plugins/perforce/PerforceSCM.java
          src/test/java/hudson/plugins/perforce/PerforceSCMTest.java
          http://jenkins-ci.org/commit/perforce-plugin/6ed6b7083c65c4a1a7c81075b6858c444b0ebe57
          Log:
          JENKINS-17652 trim quotes from paths before comparing file paths to the view paths

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Rob Petti Path: src/main/java/hudson/plugins/perforce/PerforceSCM.java src/test/java/hudson/plugins/perforce/PerforceSCMTest.java http://jenkins-ci.org/commit/perforce-plugin/6ed6b7083c65c4a1a7c81075b6858c444b0ebe57 Log: JENKINS-17652 trim quotes from paths before comparing file paths to the view paths
          Hide
          rpetti Rob Petti added a comment -

          Ok, I suspect that's the cause then. Perforce reports it's view as a quoted depot path if there is a space, but it reports file paths in changelogs without quotations at all. One of the many inconsistencies with perforce, unfortunately... I've checked in a change that should resolve the issue.

          Show
          rpetti Rob Petti added a comment - Ok, I suspect that's the cause then. Perforce reports it's view as a quoted depot path if there is a space, but it reports file paths in changelogs without quotations at all. One of the many inconsistencies with perforce, unfortunately... I've checked in a change that should resolve the issue.
          Hide
          jedavis Jason Davis added a comment -

          Thank you much! I'll try it out and update the ticket when you release the next update. Thanks for a great plugin.

          Show
          jedavis Jason Davis added a comment - Thank you much! I'll try it out and update the ticket when you release the next update. Thanks for a great plugin.
          Hide
          jedavis Jason Davis added a comment -

          Looks like v1.3.22 has solved the issue. Files that were once NOT triggering the build - it looks like because of the spaces - are now triggering builds. Thank you much for your help!

          Show
          jedavis Jason Davis added a comment - Looks like v1.3.22 has solved the issue. Files that were once NOT triggering the build - it looks like because of the spaces - are now triggering builds. Thank you much for your help!

            People

            • Assignee:
              rpetti Rob Petti
              Reporter:
              jedavis Jason Davis
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: