Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21014

Include verify lifecycle to calculate the more relevant build for "Build whenever a SNAPSHOT dependency is built" when there is more than one upstream candidate

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: maven-plugin
    • Labels:
      None
    • Similar Issues:

      Description

      Currently the upstream candidate resolution mechanism for "Build whenever a SNAPSHOT dependency is built" only considers Maven install and deploy lifecycle when there is more than one possible upstream build for the same artifact. In this case the name tiebreaker kicks in, which then determines the upstream build comparing by the job name. This is a reproducible behaviour but doesn't cater for the following situation as then the "wrong" build is selected depending on your naming strategy.

      If you want to split your job into two steps where the first step verifies that your maven project compiles and all test are successful and in the second step one then upload the new artefacts to your remote repository (if the first step works fine) this works fine if you run the first step with "mvn clean verify" and the second step with "mvn deploy" and there is no other upstream candidate. It doesn't work when there are other possible candidates (e.g. a static code analysis build of the same artefact) as then the tiebreaker kicks in and the the "wrong" build is selected (see above).

      Example
      Artefact A has the following two builds

      • A_1 (Static code analysis, which only compiles)
      • A_2 (CI, step1: mvn clean verify, step2: mvn deploy)

      Artifact B depends on artifact A
      There should be an automatic upstream build from B to A_2 and not A_1.
      With the current implementation, A_1 would be considered as upstream candidate as it is preferred by the name sorting tie breaker

      In order to make the "Build whenever a SNAPSHOT dependency is built" feature working for the above situation the algorithm to determine the upstream build should not only consider the install and deploy lifecycle phase, but also the verify phase in preference over the name sorting tie breaker.

        Attachments

          Activity

          christiangalsterer Christian Galsterer created issue -
          christiangalsterer Christian Galsterer made changes -
          Field Original Value New Value
          Summary Calculate Maven dependency graph also in verify lifecycle to deploy to local remote repository only when maven verify is succesfull and to allow proper working of "Build whenever a SNAPSHOT dependency is built" Include verify lifecycle to calculate the more relevant build for "Build whenever a SNAPSHOT dependency is built"
          Description Currently the dependency graph for "Build whenever a SNAPSHOT dependency is built" maven jobs works only for Maven install and deploy lifecycle.

          If you want to split your job into two steps where the first step verifies that your maven project compiles and all test are successful and in the second step one then upload the new artefacts to your remote repository (if the first step works fine) this currently works of you run the first step with mvn clean install but not with clean verify. The problem with that when your build fails is (e.g. compilation problems, broken tests) that one has a set of artefacts in your local repository which is incomplete and in the worst case also incompatible as some parts already new and same parts are old and they are not compatible, e.g. by an API change.

          So solve this the artefacts shall only be deployed to the local and remote depository when the verify phase was successful. In order to make the "Build whenever a SNAPSHOT dependency is built" feature working, the dependency graph needs to be created not only for the install and verify lifecycle phase, but already for the verify phase.

          Please find attached a patch which contains these changes.

          If you think the patch is correct, please let me know and I create a proper hit pull request.
          Currently the upstream candidate resolution mechanism for "Build whenever a SNAPSHOT dependency is built" only considers Maven install and deploy lifecycle when there is more than one possible upstream build. In this case the tiebreaker kicks in, which then determines the upstream build comparing by the job name. This is a reproducible behaviour but doesn't cater for the following situation as then the "wrong" build is selected depending on your naming strategy.

          If you want to split your job into two steps where the first step verifies that your maven project compiles and all test are successful and in the second step one then upload the new artefacts to your remote repository (if the first step works fine) this works fine if you run the first step with "mvn clean verify" and the second step with "deploy" and there is no other upstream candidate. It doesn't work when there are other possible candidates (e.g. a static code analysis build of the same artefact) as then the tiebreaker kicks in and the the "wrong" build is selected (see above).

          In order to make the "Build whenever a SNAPSHOT dependency is built" feature working for the above sitatuation the algorithm to determine the upstream build should not only consider the install and deploy lifecycle phase, but also the verify phase.
          christiangalsterer Christian Galsterer made changes -
          Component/s core [ 15593 ]
          christiangalsterer Christian Galsterer made changes -
          Summary Include verify lifecycle to calculate the more relevant build for "Build whenever a SNAPSHOT dependency is built" Include verify lifecycle to calculate the more relevant build for "Build whenever a SNAPSHOT dependency is built" when there are more than one candidate
          christiangalsterer Christian Galsterer made changes -
          Summary Include verify lifecycle to calculate the more relevant build for "Build whenever a SNAPSHOT dependency is built" when there are more than one candidate Include verify lifecycle to calculate the more relevant build for "Build whenever a SNAPSHOT dependency is built" when there is more than one upstream candidate
          Hide
          christiangalsterer Christian Galsterer added a comment -

          Committed change, see pull request https://github.com/jenkinsci/maven-plugin/pull/15.

          Show
          christiangalsterer Christian Galsterer added a comment - Committed change, see pull request https://github.com/jenkinsci/maven-plugin/pull/15 .
          christiangalsterer Christian Galsterer made changes -
          Assignee Jesse Glick [ jglick ]
          christiangalsterer Christian Galsterer made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Hide
          jglick Jesse Glick added a comment -

          Not sure why this was assigned to me.

          Show
          jglick Jesse Glick added a comment - Not sure why this was assigned to me.
          jglick Jesse Glick made changes -
          Assignee Jesse Glick [ jglick ]
          christiangalsterer Christian Galsterer made changes -
          Description Currently the upstream candidate resolution mechanism for "Build whenever a SNAPSHOT dependency is built" only considers Maven install and deploy lifecycle when there is more than one possible upstream build. In this case the tiebreaker kicks in, which then determines the upstream build comparing by the job name. This is a reproducible behaviour but doesn't cater for the following situation as then the "wrong" build is selected depending on your naming strategy.

          If you want to split your job into two steps where the first step verifies that your maven project compiles and all test are successful and in the second step one then upload the new artefacts to your remote repository (if the first step works fine) this works fine if you run the first step with "mvn clean verify" and the second step with "deploy" and there is no other upstream candidate. It doesn't work when there are other possible candidates (e.g. a static code analysis build of the same artefact) as then the tiebreaker kicks in and the the "wrong" build is selected (see above).

          In order to make the "Build whenever a SNAPSHOT dependency is built" feature working for the above sitatuation the algorithm to determine the upstream build should not only consider the install and deploy lifecycle phase, but also the verify phase.
          Currently the upstream candidate resolution mechanism for "Build whenever a SNAPSHOT dependency is built" only considers Maven install and deploy lifecycle when there is more than one possible upstream build for the *same* artifact. In this case the name tiebreaker kicks in, which then determines the upstream build comparing by the job name. This is a reproducible behaviour but doesn't cater for the following situation as then the "wrong" build is selected depending on your naming strategy.

          If you want to split your job into two steps where the first step verifies that your maven project compiles and all test are successful and in the second step one then upload the new artefacts to your remote repository (if the first step works fine) this works fine if you run the first step with "mvn clean verify" and the second step with "deploy" and there is no other upstream candidate. It doesn't work when there are other possible candidates (e.g. a static code analysis build of the *same* artefact) as then the tiebreaker kicks in and the the "wrong" build is selected (see above).

          Example
          Artifact A has the following two builds
          - A_1 (CI, verify->deploy)
          - A_2 (Static code analysis, which only compiles)

          Artifact B depends on artifact
          There should be an automatic upstream build from B to A_1 and not A_2.

          In order to make the "Build whenever a SNAPSHOT dependency is built" feature working for the above sitatuation the algorithm to determine the upstream build should not only consider the install and deploy lifecycle phase, but also the verify phase.
          Hide
          kutzi kutzi added a comment - - edited

          It seems kind of wrong to me me to include 'verify' into the calculation as verify doesn't install or deploy any artifacts.

          How are you deploying the artifacts in job A_1? The root error seems to me that the calculation doesn't see that the 1st job is actually deploying something.

          Show
          kutzi kutzi added a comment - - edited It seems kind of wrong to me me to include 'verify' into the calculation as verify doesn't install or deploy any artifacts. How are you deploying the artifacts in job A_1? The root error seems to me that the calculation doesn't see that the 1st job is actually deploying something.
          Hide
          christiangalsterer Christian Galsterer added a comment - - edited

          The use case is that you want to split your job into two steps, where the first step does a "mvn clean verify" and the second step is then a "mvn deploy". By this one makes sure that you only deploy to your remote repository when the complete build was successful and don't deploy after each maven module which lead to incomplete deployments and all sort of problems, e.g. incompatible APIs, incompatible implementations, etc.

          The other problem is that if there are two jobs as potential upstream candidates the job name tie breaker implementation kicks-in which chooses the upstream build based on the sorting of the job name. Although this is a deterministic behaviour IMO it would be better if also "verify" would be considered and takes precedence over the naming tie breaker. The change is completely backwards compatible but would allow support for jobs configuration which separate building and deploying into 2 steps as described above.

          Hope this clarifies the reasoning behind the change. I think it is useful as a general feature to allow to split a job in two steps to make sure that only working builds are deployed.

          As some time has passed I actually created a new pull request to update to more recent Jenkins version and also include a small bugfix, see https://github.com/jenkinsci/maven-plugin/pull/19.

          I have the change running on a larger scale setup (> 2000 jobs) and am quite happy with it and also the feedback by developer teams that they only get "complete and consistent" artefact in the remote repository is positive.

          Would be cool if this makes it into the maven plugin. Let me know if there is anything to improve in the pull request.

          Show
          christiangalsterer Christian Galsterer added a comment - - edited The use case is that you want to split your job into two steps, where the first step does a "mvn clean verify" and the second step is then a "mvn deploy". By this one makes sure that you only deploy to your remote repository when the complete build was successful and don't deploy after each maven module which lead to incomplete deployments and all sort of problems, e.g. incompatible APIs, incompatible implementations, etc. The other problem is that if there are two jobs as potential upstream candidates the job name tie breaker implementation kicks-in which chooses the upstream build based on the sorting of the job name. Although this is a deterministic behaviour IMO it would be better if also "verify" would be considered and takes precedence over the naming tie breaker. The change is completely backwards compatible but would allow support for jobs configuration which separate building and deploying into 2 steps as described above. Hope this clarifies the reasoning behind the change. I think it is useful as a general feature to allow to split a job in two steps to make sure that only working builds are deployed. As some time has passed I actually created a new pull request to update to more recent Jenkins version and also include a small bugfix, see https://github.com/jenkinsci/maven-plugin/pull/19 . I have the change running on a larger scale setup (> 2000 jobs) and am quite happy with it and also the feedback by developer teams that they only get "complete and consistent" artefact in the remote repository is positive. Would be cool if this makes it into the maven plugin. Let me know if there is anything to improve in the pull request.
          christiangalsterer Christian Galsterer made changes -
          Description Currently the upstream candidate resolution mechanism for "Build whenever a SNAPSHOT dependency is built" only considers Maven install and deploy lifecycle when there is more than one possible upstream build for the *same* artifact. In this case the name tiebreaker kicks in, which then determines the upstream build comparing by the job name. This is a reproducible behaviour but doesn't cater for the following situation as then the "wrong" build is selected depending on your naming strategy.

          If you want to split your job into two steps where the first step verifies that your maven project compiles and all test are successful and in the second step one then upload the new artefacts to your remote repository (if the first step works fine) this works fine if you run the first step with "mvn clean verify" and the second step with "deploy" and there is no other upstream candidate. It doesn't work when there are other possible candidates (e.g. a static code analysis build of the *same* artefact) as then the tiebreaker kicks in and the the "wrong" build is selected (see above).

          Example
          Artifact A has the following two builds
          - A_1 (CI, verify->deploy)
          - A_2 (Static code analysis, which only compiles)

          Artifact B depends on artifact
          There should be an automatic upstream build from B to A_1 and not A_2.

          In order to make the "Build whenever a SNAPSHOT dependency is built" feature working for the above sitatuation the algorithm to determine the upstream build should not only consider the install and deploy lifecycle phase, but also the verify phase.
          Currently the upstream candidate resolution mechanism for "Build whenever a SNAPSHOT dependency is built" only considers Maven install and deploy lifecycle when there is more than one possible upstream build for the *same* artifact. In this case the name tiebreaker kicks in, which then determines the upstream build comparing by the job name. This is a reproducible behaviour but doesn't cater for the following situation as then the "wrong" build is selected depending on your naming strategy.

          If you want to split your job into two steps where the first step verifies that your maven project compiles and all test are successful and in the second step one then upload the new artefacts to your remote repository (if the first step works fine) this works fine if you run the first step with "mvn clean verify" and the second step with "mvn deploy" and there is no other upstream candidate. It doesn't work when there are other possible candidates (e.g. a static code analysis build of the *same* artefact) as then the tiebreaker kicks in and the the "wrong" build is selected (see above).

          Example
          Artefact A has the following two builds
          - A_1 (Static code analysis, which only compiles)
          - A_2 (CI, step1: mvn clean verify, step2: mvn deploy)

          Artifact B depends on artifact A
          There should be an automatic upstream build from B to A_2 and not A_1.
          With the current implementation, A_1 would be considered as upstream candidate as it is preferred by the name sorting tie breaker

          In order to make the "Build whenever a SNAPSHOT dependency is built" feature working for the above situation the algorithm to determine the upstream build should not only consider the install and deploy lifecycle phase, but also the verify phase in preference over the name sorting tie breaker.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Christian Galsterer
          Path:
          src/main/java/hudson/maven/MavenModule.java
          src/test/java/hudson/maven/MavenSnapshotTriggerTest.java
          http://jenkins-ci.org/commit/maven-plugin/f8d2b8499231b13be418d96907d3d52dc3951954
          Log:
          JENKINS-21014 include verify in upstream candidate calculation

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christian Galsterer Path: src/main/java/hudson/maven/MavenModule.java src/test/java/hudson/maven/MavenSnapshotTriggerTest.java http://jenkins-ci.org/commit/maven-plugin/f8d2b8499231b13be418d96907d3d52dc3951954 Log: JENKINS-21014 include verify in upstream candidate calculation
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Christian Galsterer
          Path:
          src/test/java/hudson/maven/MavenSnapshotTriggerTest.java
          http://jenkins-ci.org/commit/maven-plugin/9687602edf2ae8ee82a6a0810ee7007d8d86226f
          Log:
          JENKINS-21014 include verify in upstream candidate calculation

          • changed test to make sure that dependencies for test projects are available in repository
          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christian Galsterer Path: src/test/java/hudson/maven/MavenSnapshotTriggerTest.java http://jenkins-ci.org/commit/maven-plugin/9687602edf2ae8ee82a6a0810ee7007d8d86226f Log: JENKINS-21014 include verify in upstream candidate calculation changed test to make sure that dependencies for test projects are available in repository
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          src/main/java/hudson/maven/MavenModule.java
          src/test/java/hudson/maven/MavenSnapshotTriggerTest.java
          http://jenkins-ci.org/commit/maven-plugin/9a6885cc87a33505b5083337eb61731ec3803f67
          Log:
          Merge pull request #19 from christiangalsterer/master

          JENKINS-21014 include verify in upstream candidate calculation

          Compare: https://github.com/jenkinsci/maven-plugin/compare/0a22e26f66bd...9a6885cc87a3

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: src/main/java/hudson/maven/MavenModule.java src/test/java/hudson/maven/MavenSnapshotTriggerTest.java http://jenkins-ci.org/commit/maven-plugin/9a6885cc87a33505b5083337eb61731ec3803f67 Log: Merge pull request #19 from christiangalsterer/master JENKINS-21014 include verify in upstream candidate calculation Compare: https://github.com/jenkinsci/maven-plugin/compare/0a22e26f66bd...9a6885cc87a3
          Hide
          kutzi kutzi added a comment -

          Just for the record:

          • the use case you describe is exactly covered by the 'deploy artifacts' checkbox (don't know if that's the name in the UI from the top of my head) for Jenkins maven builds. However, I had some bad experience with this in the past (didn't work always as expected) and others are generally thinking that thie kind of Jenkins 'magic' on top of Maven is bad (e.g. Stephen Connolly . So your use case is surely a valid one.
          • in time we might include the full standard Maven lifecycle into the dependency calculation and priorise later lifecycle entries higher than previous one. Originally, I've implemented this only because the previous solution seemed completely 'random' and having any kind of predictability was better than that
          Show
          kutzi kutzi added a comment - Just for the record: the use case you describe is exactly covered by the 'deploy artifacts' checkbox (don't know if that's the name in the UI from the top of my head) for Jenkins maven builds. However, I had some bad experience with this in the past (didn't work always as expected) and others are generally thinking that thie kind of Jenkins 'magic' on top of Maven is bad (e.g. Stephen Connolly . So your use case is surely a valid one. in time we might include the full standard Maven lifecycle into the dependency calculation and priorise later lifecycle entries higher than previous one. Originally, I've implemented this only because the previous solution seemed completely 'random' and having any kind of predictability was better than that
          kutzi kutzi made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 152855 ] JNJira + In-Review [ 194372 ]
          Hide
          bondolo Mike Duigou added a comment - - edited

          In our environment we frequently have multiple projects building the same artifacts. Most of these projects are either smoketest builds or builds for the code review validation (gerrit). We want only the projects which run off vcs updates and install artifacts to be used for dependency tracking. The availability of new artifacts is the reasonable trigger for downstream builds, no? The result of this change is that one of several projects which are building a particular set of artifacts being chosen as upstream. For our purposes this is very wrong–only projects publishing artifacts should be considered as upstream candidates.

          Show
          bondolo Mike Duigou added a comment - - edited In our environment we frequently have multiple projects building the same artifacts. Most of these projects are either smoketest builds or builds for the code review validation (gerrit). We want only the projects which run off vcs updates and install artifacts to be used for dependency tracking. The availability of new artifacts is the reasonable trigger for downstream builds, no? The result of this change is that one of several projects which are building a particular set of artifacts being chosen as upstream. For our purposes this is very wrong–only projects publishing artifacts should be considered as upstream candidates.

            People

            • Assignee:
              Unassigned
              Reporter:
              christiangalsterer Christian Galsterer
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: