Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50099

Downstream pipelines are not triggered when the artifact is a a WAR and consumed by war-overlay

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: pipeline-maven-plugin
    • Labels:
      None
    • Environment:
      Jenkins ver. 2.89.4, Slave on ubuntu, freshly updated plugins; Pipeline Maven Integration 3.4.2, maven 3.5.3
    • Similar Issues:

      Description

      Problem

      When a SNAPSHOT artifact is built (packaging = 'war'), its downstream project building a WAR-overlay depending on the upstream SNAPSHOT is not triggered.

      How to reproduce

      A test project (maven_war_overlay_project) with the described behaviour is available in the test_maven_projects directory on the fork in https://github.com/benzht/pipeline-maven-plugin/tree/JENKINS-50099 under the test_maven_projects (due to time constraints I could not integrate it into the test suite....).

      The new test project, maven_war_overlay_project, depends on the existing maven_war_project, which in turn depends on maven_jar_project.
      After manually building all of them, building the first correctly triggers the second, but maven_war_overlay_project is not triggered.
      Looking at the logging, it becomes clear that in the war_overlay_project no dependency is detected.

      Output from maven_war_project correctly records its dependency:

      ...
      [withMaven] pipelineGraphPublisher - recordParentProject - filter: versions[snapshot: true, release: false]
      [withMaven] pipelineGraphPublisher - recordDependencies - filter: versions[snapshot: true, release: false], scopes:[compile, provided, runtime]
      [withMaven] pipelineGraphPublisher - Record dependency: jenkins.mvn.test:mono-module-maven-app:0.1-SNAPSHOT, ignoreUpstreamTriggers: false
      [withMaven] pipelineGraphPublisher - Skip recording release dependency: javax.servlet:javax.servlet-api:3.1.0
      [withMaven] pipelineGraphPublisher - Skip recording release dependency: junit:junit:4.12
      [withMaven] pipelineGraphPublisher - Skip recording release dependency: org.hamcrest:hamcrest-core:1.3
      [withMaven] pipelineGraphPublisher - recordGeneratedArtifacts...
      [withMaven] pipelineGraphPublisher - Record generated artifact: jenkins.mvn.test:maven-war-app:0.1-SNAPSHOT, version: 0.1-SNAPSHOT, executedLifecyclePhases: [clean, process-resources, compile, process-test-resources, test-compile, test, package, install, deploy], skipDownstreamTriggers: false, lifecycleThreshold:deploy, file: /home/jenkins/slave/workspace/maven_war_project/pom.xml
      [withMaven] pipelineGraphPublisher - Record generated artifact: jenkins.mvn.test:maven-war-app:0.1-SNAPSHOT, version: 0.1-20180311.164646-4, executedLifecyclePhases: [clean, process-resources, compile, process-test-resources, test-compile, test, package, install, deploy], skipDownstreamTriggers: false, lifecycleThreshold:deploy, file: /home/jenkins/slave/workspace/maven_war_project/target/maven-war-app-0.1-SNAPSHOT.war
      ...
      

      Output from maven_war_overlay_project does not record any dependeny:

      ...
      [withMaven] pipelineGraphPublisher - recordParentProject - filter: versions[snapshot: true, release: false]
      [withMaven] pipelineGraphPublisher - recordDependencies - filter: versions[snapshot: true, release: false], scopes:[compile, provided, runtime]
      [withMaven] pipelineGraphPublisher - recordGeneratedArtifacts...
      [withMaven] pipelineGraphPublisher - Record generated artifact: jenkins.mvn.test:maven-war-overlay-app:0.1-SNAPSHOT, version: 0.1-SNAPSHOT, executedLifecyclePhases: [clean, process-resources, compile, process-test-resources, test-compile, test, package, install, deploy], skipDownstreamTriggers: false, lifecycleThreshold:deploy, file: /home/jenkins/slave/workspace/maven_war_overlay_project/pom.xml
      [withMaven] pipelineGraphPublisher - Record generated artifact: jenkins.mvn.test:maven-war-overlay-app:0.1-SNAPSHOT, version: 0.1-20180311.164534-3, executedLifecyclePhases: [clean, process-resources, compile, process-test-resources, test-compile, test, package, install, deploy], skipDownstreamTriggers: false, lifecycleThreshold:deploy, file: /home/jenkins/slave/workspace/maven_war_overlay_project/target/maven-war-overlay-app.war
      

      Jenkinsfile of maven_war_overlay_project:

      node {
         stage('Preparation') { // for display purposes
            git changelog: false, credentialsId: '3360632d-fb8d-4f68-be83-b650ec243e94', poll: false, url: 'http://........git'
         }
         stage('Build') {
              withMaven(jdk: 'JDK8', maven: 'M3',
                      globalMavenSettingsConfig: 'globalMavenSettings', options: [dependenciesFingerprintPublisher(), pipelineGraphPublisher(), artifactsPublisher(disabled: true)]) {
                  writeFile file: '.archive-jenkins-maven-event-spy-logs', text: ''
                  sh "mvn clean deploy"
              }
         }
      }
      

        Attachments

          Issue Links

            Activity

            Hide
            cleclerc Cyrille Le Clerc added a comment - - edited

            Reproducing a test case with

            The runtime dependency of the overlay does NOT appear in the list of resolved dependencies of the maven project. Probably because it is flagged as a runtime dependency.

            https://gist.github.com/cyrille-leclerc/bcb8e7d078570b7f7867eb55d0ed647c

            The maven-war-plugin overlay docs clearly recommends to use a "runtime" scope

            https://maven.apache.org/plugins/maven-war-plugin/overlays.html

            <dependencies>
                <dependency>
                  <groupId>com.example.projects</groupId>
                  <artifactId>documentedprojectdependency</artifactId>
                  <version>1.0-SNAPSHOT</version>
                  <type>war</type>
                  <scope>runtime</scope>
                </dependency>
                ...
              </dependencies>
            
            Show
            cleclerc Cyrille Le Clerc added a comment - - edited Reproducing a test case with https://github.com/cyrille-leclerc/my-war-overlay.git https://github.com/cyrille-leclerc/my-war.git The runtime dependency of the overlay does NOT appear in the list of resolved dependencies of the maven project. Probably because it is flagged as a runtime dependency. https://gist.github.com/cyrille-leclerc/bcb8e7d078570b7f7867eb55d0ed647c The maven-war-plugin overlay docs clearly recommends to use a "runtime" scope https://maven.apache.org/plugins/maven-war-plugin/overlays.html <dependencies> <dependency> <groupId> com.example.projects </groupId> <artifactId> documentedprojectdependency </artifactId> <version> 1.0-SNAPSHOT </version> <type> war </type> <scope> runtime </scope> </dependency> ... </dependencies>
            Hide
            cleclerc Cyrille Le Clerc added a comment - - edited

            Hartmut Benz problem identified ^^. 

            Workaround: declare the war dependency of your overlay project with a scope "compile" instead of a scope "runtime". could this be acceptable for you?

             

            Solving JENKINS-50205 will solve this issue.

            Show
            cleclerc Cyrille Le Clerc added a comment - - edited Hartmut Benz problem identified ^^.  Workaround: declare the war dependency of your overlay project with a scope "compile" instead of a scope "runtime". could this be acceptable for you?   Solving  JENKINS-50205 will solve this issue.
            Hide
            benzht Hartmut Benz added a comment -

             

            The work-around certainly makes the war-overlay being triggered after the war.jar has been built - thank you for that already.
            During the weekend, I will study which other effects the change from 'runtime' to 'compile' has on the generated artifact and post the results here.

            Thanks again for your support

            Show
            benzht Hartmut Benz added a comment -   The work-around certainly makes the war-overlay being triggered after the war.jar has been built - thank you for that already. During the weekend, I will study which other effects the change from 'runtime' to 'compile' has on the generated artifact and post the results here. Thanks again for your support
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            Thanks for the kind word Hartmut Benz. I don't know yet how we will be able to discover these runtime dependencies as they seem to not be catch by the org.apache.maven.project.DependencyResolutionResult event we are tracking.

            Show
            cleclerc Cyrille Le Clerc added a comment - Thanks for the kind word Hartmut Benz . I don't know yet how we will be able to discover these runtime dependencies as they seem to not be catch by the org.apache.maven.project.DependencyResolutionResult event we are tracking.
            Hide
            benzht Hartmut Benz added a comment -

            The work-around for the WAR works: the WAR versions built with compile and runtime scope are identical.

            Sadly, the build-trigger is still not working but that seems to be related to JENKINS-46313 and may require another ticket and some time to build a test scenario. My upstream project is multi-branch, multi-(maven)-module. The downstream WAR overlay basically depends on all branches with the right version number. But rom the point of view of the dependency-relationships, the branches all depend upon one another (but don't trigger each other).

            Effectively, the downstream project depends on the upstream (branch) project directly (a–>b) and at the same time indirectly via other upstream branch projects (a–>a'–>b). And that seems to be the trigger-exclusion criteria of JENKINS-46313.

            Cyrille Le Clerc, do you see here a misconfiguration on my side or should I build a ticket with demo projects for it?

            Show
            benzht Hartmut Benz added a comment - The work-around for the WAR works: the WAR versions built with compile and runtime scope are identical. Sadly, the build-trigger is still not working but that seems to be related to  JENKINS-46313 and may require another ticket and some time to build a test scenario. My upstream project is multi-branch, multi-(maven)-module. The downstream WAR overlay basically depends on all branches with the right version number. But rom the point of view of the dependency-relationships, the branches all depend upon one another (but don't trigger each other). Effectively, the downstream project depends on the upstream (branch) project directly (a–>b) and at the same time indirectly via other upstream branch projects (a–>a'–>b). And that seems to be the trigger-exclusion criteria of  JENKINS-46313 . Cyrille Le Clerc , do you see here a misconfiguration on my side or should I build a ticket with demo projects for it?
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            I may have reproduced the problem. I'll add details to the recently introduced maven dashboard for build results (in master branch but not released yet) to help troubleshooting.

            Show
            cleclerc Cyrille Le Clerc added a comment - I may have reproduced the problem. I'll add details to the recently introduced maven dashboard for build results (in master branch but not released yet) to help troubleshooting.
            Hide
            cleclerc Cyrille Le Clerc added a comment - - edited

            Hartmut Benz can you install v3.5.0 and look at the "Maven" build details screen for your war pipeline and your war-overlay pipeline?

            Please look at the "dependencies" of the war-overlay pipeline and at the "downstream pipelines" of the war pipeline

            Show
            cleclerc Cyrille Le Clerc added a comment - - edited Hartmut Benz can you install v3.5.0 and look at the "Maven" build details screen for your war pipeline and your war-overlay pipeline? Please look at the "dependencies" of the war-overlay pipeline and at the "downstream pipelines" of the war pipeline
            Hide
            benzht Hartmut Benz added a comment - - edited

            Cyrille Le Clerc, I have upgraded everything to 'current' including the 3.5 version.

            The upstream project (on matrix level) list the following
            Downstream Jobs

            • Downstream1 » develop
            • Downstream2 » develop
            • upstream » develop
            • upstream » feature/jenkinsWork
            • upstream » feature/t6482_keycloak
            • upstream » feature/testServer

            The downstream projects Downstream1.develop and Downstream2.develop list the following
            Upstream Builds

            • upstream » feature/testNewServer #78
            • upstream » feature/jenkinsWork #24
            • upstream » develop #40        <—this is the most recent build number
            • upstream » feature/t6482_keycloak #3

            The downstream matrix project config has Build on Snapshot dependency set.

            But Upstream-develop Build #40 did not trigger either of the Downstream projects. This is the end of the logs:

            [withMaven] Not triggering Downsteam1 » develop because it has dependencies in the downstream project list
            [withMaven] Not triggering Downsteam2 » develop because it has dependencies in the downstream project list
            Finished: SUCCESS
            

            Is there anything else I can test, log, or try to help you further?

            Show
            benzht Hartmut Benz added a comment - - edited Cyrille Le Clerc , I have upgraded everything to 'current' including the 3.5 version. The upstream project (on matrix level) list the following Downstream Jobs Downstream1 » develop Downstream2 » develop upstream » develop upstream » feature/jenkinsWork upstream » feature/t6482_keycloak upstream » feature/testServer The downstream projects Downstream1.develop and Downstream2.develop list the following Upstream Builds upstream » feature/testNewServer #78 upstream » feature/jenkinsWork #24 upstream » develop #40        <—this is the most recent build number upstream » feature/t6482_keycloak #3 The downstream matrix project config has Build on Snapshot dependency set. But Upstream-develop Build #40 did not trigger either of the Downstream projects. This is the end of the logs: [withMaven] Not triggering Downsteam1 » develop because it has dependencies in the downstream project list [withMaven] Not triggering Downsteam2 » develop because it has dependencies in the downstream project list Finished: SUCCESS Is there anything else I can test, log, or try to help you further?
            Hide
            benzht Hartmut Benz added a comment -

            My best guess would be to exclude all sister-branches in a multi-branch project from being considered as being downstream. That way they would not prevent the real downstream projects from being triggered.

            Show
            benzht Hartmut Benz added a comment - My best guess would be to exclude all sister-branches in a multi-branch project from being considered as being downstream . That way they would not prevent the real downstream projects from being triggered.
            Hide
            mmichel Maxime Michel added a comment - - edited

            Hartmut Benz I haven't had time to look at the setup you described and compare it with ours, however, I am seeing the exact same message with a non-multibranch setup:

            Basically, a module pipeline is included in a webapp pipeline, which is then overlaid in another pipeline. Building the module results in:

            [withMaven] Not triggering webapp because it has dependencies in the downstream project list
            [withMaven] Not triggering overlaid-webapp because it has dependencies in the downstream project list
            

            To me the issue seems to be that both webapps are listed twice as dependency to themselves in the Maven dashboard. That doesn't look right, shouldn't self be excluded from being a possible dependency to any job?

            Show
            mmichel Maxime Michel added a comment - - edited Hartmut Benz I haven't had time to look at the setup you described and compare it with ours, however, I am seeing the exact same message with a non-multibranch setup: Basically, a module pipeline is included in a webapp pipeline, which is then overlaid in another pipeline. Building the module results in: [withMaven] Not triggering webapp because it has dependencies in the downstream project list [withMaven] Not triggering overlaid-webapp because it has dependencies in the downstream project list To me the issue seems to be that both webapps are listed twice as dependency to themselves in the Maven dashboard. That doesn't look right, shouldn't self be excluded from being a possible dependency to any job?
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            Note: as war-overlay "dependency" is not a "standard Maven dependency", we may look at solving this problem in conjunction with solving JENKINS-50475 (Maven BOM Dependency) which is not either a "standard Maven dependency".

            Show
            cleclerc Cyrille Le Clerc added a comment - Note: as war-overlay "dependency" is not a "standard Maven dependency", we may look at solving this problem in conjunction with solving  JENKINS-50475 (Maven BOM Dependency) which is not either a "standard Maven dependency".
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            war-overlay and maven-bom dependencies are not "standard Maven dependencies", we may look at solving these 2 problems simultaneously.

            Show
            cleclerc Cyrille Le Clerc added a comment - war-overlay and maven-bom dependencies are not "standard Maven dependencies", we may look at solving these 2 problems simultaneously.

              People

              • Assignee:
                cleclerc Cyrille Le Clerc
                Reporter:
                benzht Hartmut Benz
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: