Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46313

Avoid excessive triggering of downstream jobs

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      JENKINS-44721 added the ability to trigger pipelines according to their SNAPSHOT dependencies. This can lead to the problem that sometimes many builds are triggered. JENKINS-46152 avoids this by introducing a flag to disable triggering altogether, but then the benefit of the triggers is lost.

      The old Maven project type had a feature called "Block downstream trigger when building" which would be helpful in this situation. That feature doesn't skip all the downstream triggers, but only the triggers of those downstream dependencies which will later on be triggered by other downstream dependencies.

      To explain it with an example: imagine Maven projects A, B, and C, where B depends on A, and C depends on both A and B. Currently, when A is built, C gets triggered twice, once by A and once by B (which is also triggered by A). But with the proposed feature, C only gets triggered by B, because A knows that B will trigger C later on and will therefore skip that trigger.

      Of course, this feature should be introduced with a new flag to preserve the current behavior.

        Attachments

          Issue Links

            Activity

            Hide
            markusdlugi Markus Dlugi added a comment -

            Cyrille Le Clerc:

            I investigated this issue a little bit further because we required this feature for our own Jenkins instance. As it turns out, there are two mechanisms regarding this topic in the old Maven project. The first one is the aforementioned "Block downstream trigger when building". The code for this can be seen here:

            https://github.com/jenkinsci/maven-plugin/blob/master/src/main/java/hudson/maven/AbstractMavenProject.java#L87

            This results in the following message at the end of the build:

            "Not triggering SOME_PROJECT because it has a dependency OTHER_PROJECT already building or in queue"

            So this is only related to upstream projects which are already building or in queue. A second feature, which is the one I explained in the initial ticket description, can be seen here:

            https://github.com/jenkinsci/maven-plugin/blob/master/src/main/java/hudson/maven/AbstractMavenProject.java#L95

            This checks whether any of the downstream job's upstreams are also in the downstream project list. In that case, triggering is also skipped, the message is:

            "Not triggering SOME_PROJECT because it has dependencies in the downstream project list"

            I have implemented both features in the pipeline-maven-plugin and will create a pull request for this.

            Show
            markusdlugi Markus Dlugi added a comment - Cyrille Le Clerc : I investigated this issue a little bit further because we required this feature for our own Jenkins instance. As it turns out, there are two mechanisms regarding this topic in the old Maven project. The first one is the aforementioned "Block downstream trigger when building". The code for this can be seen here: https://github.com/jenkinsci/maven-plugin/blob/master/src/main/java/hudson/maven/AbstractMavenProject.java#L87 This results in the following message at the end of the build: "Not triggering SOME_PROJECT because it has a dependency OTHER_PROJECT already building or in queue" So this is only related to upstream projects which are already building or in queue. A second feature, which is the one I explained in the initial ticket description, can be seen here: https://github.com/jenkinsci/maven-plugin/blob/master/src/main/java/hudson/maven/AbstractMavenProject.java#L95 This checks whether any of the downstream job's upstreams are also in the downstream project list. In that case, triggering is also skipped, the message is: "Not triggering SOME_PROJECT because it has dependencies in the downstream project list" I have implemented both features in the pipeline-maven-plugin and will create a pull request for this.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Markus Dlugi
            Path:
            jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/dao/PipelineMavenPluginDao.java
            jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/dao/PipelineMavenPluginH2Dao.java
            jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/dao/PipelineMavenPluginNullDao.java
            jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/listeners/DownstreamPipelineTriggerRunListener.java
            jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/dao/PipelineMavenPluginH2DaoTest.java
            http://jenkins-ci.org/commit/pipeline-maven-plugin/d68ce7bba62f6881ca1ada84ee1cf8d5a851f7a8
            Log:
            JENKINS-46313 Avoid excessive triggering of downstream jobs #109

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Markus Dlugi Path: jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/dao/PipelineMavenPluginDao.java jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/dao/PipelineMavenPluginH2Dao.java jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/dao/PipelineMavenPluginNullDao.java jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/listeners/DownstreamPipelineTriggerRunListener.java jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/dao/PipelineMavenPluginH2DaoTest.java http://jenkins-ci.org/commit/pipeline-maven-plugin/d68ce7bba62f6881ca1ada84ee1cf8d5a851f7a8 Log: JENKINS-46313 Avoid excessive triggering of downstream jobs #109
            Show
            cleclerc Cyrille Le Clerc added a comment - Released in 3.2.0-alpha-1, feedback welcome. https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/pipeline-maven/3.2.0-alpha-1/pipeline-maven-3.2.0-alpha-1.hpi
            Hide
            markusdlugi Markus Dlugi added a comment - - edited

            Cyrille Le Clerc After some testing, the only issue I sometimes notice is that the exclusions will not work when a build fails. It's not a severe issue, because then it basically falls back to the previous behaviour and triggers more builds for the affected project, but it's a little nuisance. After the project returns to normal, the exclusions start to work again. I haven't analyzed this in depth, but my guess would be that the culprit for this is the DownstreamPipelineTriggerRunListener, which uses the actual last build number instead of querying the last build number from the DAO.

            I'm not sure when I will get the time to have a closer look at this bug. Since it's not a critical bug, I think you could also release a new version and ship a bugfix in a subsequent release if you want to release soon.

            Show
            markusdlugi Markus Dlugi added a comment - - edited Cyrille Le Clerc  After some testing, the only issue I sometimes notice is that the exclusions will not work when a build fails. It's not a severe issue, because then it basically falls back to the previous behaviour and triggers more builds for the affected project, but it's a little nuisance. After the project returns to normal, the exclusions start to work again. I haven't analyzed this in depth, but my guess would be that the culprit for this is the DownstreamPipelineTriggerRunListener , which uses the actual last build number instead of querying the last build number from the DAO. I'm not sure when I will get the time to have a closer look at this bug. Since it's not a critical bug, I think you could also release a new version and ship a bugfix in a subsequent release if you want to release soon.
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            Thanks Markus Dlugi, can we say that it's not a release blocker, can we say that it will be an improvement?

            Show
            cleclerc Cyrille Le Clerc added a comment - Thanks Markus Dlugi , can we say that it's not a release blocker, can we say that it will be an improvement?
            Hide
            markusdlugi Markus Dlugi added a comment -

            Cyrille Le Clerc I think we can do that, because it only affects a new feature and doesn't introduce any regressions for existing behaviour. So in my opinion, go ahead.

            Show
            markusdlugi Markus Dlugi added a comment - Cyrille Le Clerc I think we can do that, because it only affects a new feature and doesn't introduce any regressions for existing behaviour. So in my opinion, go ahead.
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            released in 3.2.0

            Show
            cleclerc Cyrille Le Clerc added a comment - released in 3.2.0

              People

              • Assignee:
                cleclerc Cyrille Le Clerc
                Reporter:
                markusdlugi Markus Dlugi
              • Votes:
                6 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: