Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-5675

cleartool lshistory problems with ignored load rules

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: clearcase-plugin
    • Labels:
      None
    • Environment:
      LINUX

      Description

      Hi,

      below is the thread as discussed on the hudson users list.
      Please don't take them as criticism. Hudson is an excellent tool and it was a breeze to set it up. I would note in a lifetime want to switch back to cruisecontrol. Please see the below as constructive input ...

      1) I have tried both trailing and no trailing slash in the load rule for the root directory to search. No difference. The only way I can get hudson to pick up my changes are via specifying every file directly that is supposed to be monitored for changes. Also I have done some tests (as outlined in the mailthread below) to check if some of the hardlinks cause this (they don't but cleartool lsh -all seems to affect the behaviour .

      I just did some more comparisons between what hudson is doing to get the changes as opposed to cruisecontrol.
      IMO the clearcase plugin suffers from the following issues (this is also reflected by some of the other open tickets).

      2) cleartool lsh -all is used. This brings up all sorts of changes that users are not interested in and is in contradiction to the design of the load rules. If I specify a loadrule as a root path to search for changes then it is confusing to get output from all other elements not under the specified path (but in the same vob).

      3) I fell on my nose when trying to change config.xml by hand and then the changes done were overwritten by changes done to via the gui. This is probably a feature but nevertheless a noob like me ends up falling into this trap.

      4) Setting up a new project by copying an existing configuration will trigger a new build in that half finished configuration (which will lead to a first failed build).

      ________________________________________________________________________________

      ok - symptomes were same so I thought I'd better check. I will open a new bug.

      Also all my projects were created by copying an existing project rather than starting from scratch.

      Now I just checked the config.xml of my project and the follwing is still refering to the old existing project which the copy was created from:
      <unixDynStorageDir>/view/ci_TEST</unixDynStorageDir>

      Could this have any effect.
      I'll change this by hand maybe it makes a difference.

      --------------------------------------------------------------------------------
      From: ext Andrew Bayer andrew.bayer@gmail.com
      Sent: Wednesday, February 17, 2010 6:32 PM
      To: users@hudson.dev.java.net
      Subject: Re: RE: Re: Re: cleartool lshistory and modification to hardlinks

      The problem there seemed to be that no branch was specified, so I don't think it's the same issue. You should open a new bug.

      A.

      On Wed, Feb 17, 2010 at 9:20 AM, Bauernberger, Joachim (EXT-Other - DE/Ulm) <joachim.bauernberger.ext@nsn.com> wrote:

      Hi,

      Just went through the list of open bugs and found this:
      http://issues.jenkins-ci.org/browse/JENKINS-4800

      Looks like I'm having the same problem as in JENKINS-4800 (though I've specified the main branch as my branch to look for changes).

      This ticket is still open. So should I add a comment to this existing one or create a new ticket?

      Cheers,
      Joachim

      --------------------------------------------------------------------------------
      From: ext Bauernberger, Joachim (EXT-Other - DE/Ulm) joachim.bauernberger.ext@nsn.com
      Sent: Wednesday, February 17, 2010 6:04 PM

      To: users@hudson.dev.java.net

      Subject: RE: Re: Re: cleartool lshistory and modification to hardlinks

      Hi,

      thanks. Yes I will do so.

      Cheers,
      Joachim

      --------------------------------------------------------------------------------
      From: ext Andrew Bayer andrew.bayer@gmail.com
      Sent: Wednesday, February 17, 2010 6:02 PM
      To: users@hudson.dev.java.net
      Subject: Re: Re: cleartool lshistory and modification to hardlinks

      Could you open a bug for this at http://issues.jenkins-ci.org and include your job's config.xml and the log from a polling session?

      A.

      On Wed, Feb 17, 2010 at 8:55 AM, Bauernberger, Joachim (EXT-Other - DE/Ulm) <joachim.bauernberger.ext@nsn.com> wrote:

      Hi Andrew,

      thanks, though removing the trailing / does not solve this problem.

      I have now done some further tests to proof my initial assumption that:
      ------------------------------------
      The plugin relies on lshistory -all to supply a valid list of changes. But hardlinks fall through this list, since changed hardlinks are reported as change under their original destination (not neccessarily part of the path supplied by the load rules).
      ------------------------------------

      It turns out that unless I explicitly select files in the load rules they get ignored.
      So using a load rule like "/view/ci_FOO/foo/Application/BAR/myfile.txt" would trigger a build whenever "myfile.txt" is modified.

      But no build is triggered when using a directory as a load rule. E.g. load rules /view/ci_FOO/foo/Application/BAR/ or /view/ci_FOO/foo/Application/BAR
      do not trigger a build when any files beneath that directory change (regardless whether the loadrule ends with a trailing slash).

      Thanks for any help.

      Cheers,
      Joachim

      --------------------------------------------------------------------------------
      From: ext Andrew Bayer andrew.bayer@gmail.com
      Sent: Wednesday, February 17, 2010 4:55 PM
      To: users@hudson.dev.java.net
      Subject: Re: cleartool lshistory and modification to hardlinks

      Remove that trailing / from your load rules - that's fixed in trunk, but in 1.1.1, the trailing / breaks lshistory.

      A.

      On Wed, Feb 17, 2010 at 7:47 AM, Bauernberger, Joachim (EXT-Other - DE/Ulm) <joachim.bauernberger.ext@nsn.com> wrote:

      Hi,

      I've spent some time browsing the wiki and archive of this list and am
      very new to hudson.
      Therefore my apologies if what I'm asking seems obvious, and apologies
      for this lengthy mail!

      We are just migrating from cruisecontrol to hudson using base-clearcase
      on unix with dynamic views.
      I'm really excited with how easy it was to configure hudson. Move our
      exisitng process to hudson was a breeze.
      Hudson simply rules and I'm now a real fan

      I have some questions regarding my setup. I think I'm either doing
      something wrong or maybe hudson is using lshistory in a way we didn't
      expect it to).

      Our clearcase plugin version is the latest "1.1.1" along with the
      lastest stable hudson.

      Our project is set up so that anytime we have modifications below these
      directories on the /main branch then the build should be triggered:
      /view/ci_FOO/foo/Application/FOO/
      /view/ci_FOO/foo/Application/BAR/

      From what I understood sofar, I have to specify load rules to match the
      root path of where I want to look for changes.

      My load rules are therefore set to:
      /foo/Application/FOO/
      /foo/Application/BAR/

      Now, inside these directories there are some hardlinks (clearcase not
      filesystem) to other files not specified in the load rules (e.g.
      /foo/Application/ModuleTest/). For some reason modifications to the
      files pointed to by the hardlinks are not being picked up and so builds
      do not get triggered.

      The lshistory -all option as utilized by the plugin gives a list of all
      changes in the "Application/" vob since last builds (and there are
      usually several hundred). And from the "Base Clearcase Polling Log" I
      can see output similar to the follwoing:

      8<---------SNIP------8<--------SNIP------8<--------SNIP-------
      Started on Feb 17, 2010 3:50:25 PM
      [ci_FOO] $ cleartool pwv -root
      /view/ci_FOO
      [workspace] $ cleartool startview ci_FOO
      [workspace] $ cleartool mount -all
      [ci_FOO] $ cleartool lshistory -all -since 17-feb-10.12:34:34utc+0000
      -fmt '\"%Nd\" \"%u\" \"%En\" \"%Vn\" \"%e\" \"%o\" \n%c\n' -branch
      brtype:main -nco foo/Application/FOO/ foo/Application/BAR/

      ... and then hundreds of changes here but nothing that would trigger my
      build!!
      8<---------SNIP------8<--------SNIP------8<--------SNIP-------

      Any time one of the hardlinks (or the destination it points to) is being
      changed then this change does not get recognized as a valid change.

      The only workaround for this I can think of is either replace the
      hardlinks by the actual version and instead of the actual version have a
      hardlink pointing to it.

      The other alternative would be to hack together a cleartool "wrapper"
      which executes cleartool find for every load rule (without the -all
      option) which is then used instead of the real cleartool and put into
      the hudson's $PATH.

      Another way how this might be solved is by omitting the -all option in
      lshistory and instead run the command for every load rule found. This is
      more expensive but it would eliminate all non relevant modifications in
      other parts of the vob and ensure hard links are avaluated.

      I am very new so I might not make sense at all.
      Please let me know if that's the case

      Thanks for reading this far &
      Thanks for any pointers,
      Joachim

        Activity

        Hide
        abayer abayer added a comment -

        For what it's worth - on point #2, we do an lshistory -all for speed reasons. It's dramatically faster to do that than to do lshistory -recurse - we're talking an order of magnitude in many cases. It does increase the noise in the logs in some cases, but the speed improvements are very, very much worth it.

        On point #3 - if you edit the configuration files for anything in Hudson directly, you're not actually editing anything in memory. If you want to do that, after you've changed the file(s), you need to go to Manage Hudson->Reload Configuration Files.

        On point #4 - this is another general Hudson concern. I think I may have seen an issue opened requesting that copied jobs be disabled when created - you can search for that issue and vote for it, or create a new issue if it doesn't already exist. You can also work around this by disabling polling on the origin job before copying it.

        Could you attach your job's config.xml and the polling log so that I can debug this?

        Show
        abayer abayer added a comment - For what it's worth - on point #2, we do an lshistory -all for speed reasons. It's dramatically faster to do that than to do lshistory -recurse - we're talking an order of magnitude in many cases. It does increase the noise in the logs in some cases, but the speed improvements are very, very much worth it. On point #3 - if you edit the configuration files for anything in Hudson directly, you're not actually editing anything in memory. If you want to do that, after you've changed the file(s), you need to go to Manage Hudson->Reload Configuration Files. On point #4 - this is another general Hudson concern. I think I may have seen an issue opened requesting that copied jobs be disabled when created - you can search for that issue and vote for it, or create a new issue if it doesn't already exist. You can also work around this by disabling polling on the origin job before copying it. Could you attach your job's config.xml and the polling log so that I can debug this?
        Hide
        abayer abayer added a comment -

        Attaching file sent in email

        Show
        abayer abayer added a comment - Attaching file sent in email
        Hide
        abayer abayer added a comment -

        Could you please attach the polling log for the job in question?

        Show
        abayer abayer added a comment - Could you please attach the polling log for the job in question?
        Show
        anb0s anb0s added a comment - #2 can be solved, see: http://issues.jenkins-ci.org/browse/JENKINS-7481 http://issues.jenkins-ci.org/browse/JENKINS-8949 http://issues.jenkins-ci.org/browse/JENKINS-5394

          People

          • Assignee:
            abayer abayer
            Reporter:
            jbauernberger jbauernberger
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated: