Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60479

Building a matrix with exclusions in a declarative pipeline using a pipeline library finds no branches to run

    Details

    • Similar Issues:

      Description

      I have run into an issue when attempting to build a matrix in a declarative pipeline using a pipeline library through p4.

      If the excludes section is used, the pipeline is run successfully, but it reports that there are no branches to run. Without the excludes section, all targets in the matrix are run.

       

      To reproduce, I created a Perforce instance with Jenkins in a docker network and installed the p4-plugin 1.10.7. I then created a multibranch pipeline with a source of //testDepot/... from perforce.

      I created two streams and a library stream.

       

      In the first stream, I created a Jenkinsfile:

      @Library('test-pipeline') _
      lib()
      

      Where the test-pipeline is a pipeline library at //testDepot/libstream/... with the pipeline code in //testDepot/libstream/vars/lib.groovy

      def call()
      {
        pipeline
        {
          agent any
          stages
          {
            stage('Build')
            {
              matrix
              {
                axes
                {
                  axis
                  {
                    name 'PLATFORM'
                    values 'Linux', 'Windows', 'OSX'
                  }
                  axis
                  {
                    name 'CONFIGURATION'
                    values 'Debug', 'DebugMT', 'DebugNoPCH', 'Release', 'ReleaseMT', 'Retail', 'RetailMT'
                  }
                }
                excludes
                {
                  exclude
                  {
                    axis
                    {
                      name 'PLATFORM'
                      notValues 'Windows'
                    }
                    axis
                    {
                      name 'CONFIGURATION'
                      values 'DebugMT', 'DebugNoPCH', 'ReleaseMT', 'RetailMT'
                    }
                  }
                }
                stages
                {
                  stage('Build')
                  {
                    steps
                    {
                      echo "Building ${PLATFORM} ${CONFIGURATION}"
                    }
                  }
                  stage('Test')
                  {
                    steps
                    {
                      echo "Testing ${PLATFORM} ${CONFIGURATION}"
                    }
                  }
                }
              }
            }
          }
        }
      }

      When run, this pipeline is successful, but nothing runs.

      [Pipeline] withEnv
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Build)
      [Pipeline] parallel
      No branches to run
      [Pipeline] // parallel
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      Finished: SUCCESS

      If the excludes section is commented out, all targets in the matrix are run.

       

      In the second stream, I simply put the pipeline from the groovy into a Jenkinsfile. It runs correctly both with and without the excludes section.

       

      Our declarative pipeline is rather large and complex, and we have run into the "Method code too large" issue. We were hoping that we could get around it by building with a matrix in the declarative pipeline, but have run into this issue instead.

       

        Attachments

          Activity

          Hide
          p4karl Karl Wirth added a comment - - edited

          Hi Andrew Bayer,

          Thanks for reporting this. Before we start, from a pure Perforce streams point of view I'd recommend against having a library stream in the same stream path as code.

          The code expects that streams are divergent forms of the same basic code. For example a stream may contain: 

          //Project1Stream/Rel1/...
          //Project1Stream/Rel2/...
          //Project1Stream/Main/...
          
          //Project1Stream/Dev/...
          

           which could import

          //LibraryStream/main/...

          Alternatively if you have a deeper stream depth set you could have the two stream trees in the same depot:

          //StreamDepot/Project1/Rel1/...
          //StreamDepot/Project1/Rel2/...
          //StreamDepot/Project1/Main/...
          //StreamDepot/Project1/Dev/...
          
          //StreamDepot/Library/Main/...
          //StreamDepot/Library/Dev/...
          

          If you don't have this structure then there will always be outstanding changes that need to be branched between the library stream and it's parents.

          Now back to the problem. I tried setting this up and keep getting compilation errors:

           

          org.jenkinsci.plugins.workflow.cps.CpsCompilationErrorsException: startup failed:
          /var/lib/jenkins/jobs/JENKINS-60479_Exclusions_MultiBranch/branches/main/builds/2/libs/test-pipeline/vars/lib.groovy: 8: Unknown stage section "matrix". Starting with version 0.5, steps in a stage must be in a ‘steps’ block. @ line 8, column 7.
                   stage('Build')
          

          I am on 2.176.1 so this may be a version problem. Just in case can you please send me the screenshots of the job definition to make sure I'm doing this correctly.

           

           

          Show
          p4karl Karl Wirth added a comment - - edited Hi Andrew Bayer , Thanks for reporting this. Before we start, from a pure Perforce streams point of view I'd recommend against having a library stream in the same stream path as code. The code expects that streams are divergent forms of the same basic code. For example a stream may contain:  //Project1Stream/Rel1/... //Project1Stream/Rel2/... //Project1Stream/Main/... //Project1Stream/Dev/...  which could import //LibraryStream/main/... Alternatively if you have a deeper stream depth set you could have the two stream trees in the same depot: //StreamDepot/Project1/Rel1/... //StreamDepot/Project1/Rel2/... //StreamDepot/Project1/Main/... //StreamDepot/Project1/Dev/... //StreamDepot/Library/Main/... //StreamDepot/Library/Dev/... If you don't have this structure then there will always be outstanding changes that need to be branched between the library stream and it's parents. Now back to the problem. I tried setting this up and keep getting compilation errors:   org.jenkinsci.plugins.workflow.cps.CpsCompilationErrorsException: startup failed: / var /lib/jenkins/jobs/JENKINS-60479_Exclusions_MultiBranch/branches/main/builds/2/libs/test-pipeline/vars/lib.groovy: 8: Unknown stage section "matrix" . Starting with version 0.5, steps in a stage must be in a ‘steps’ block. @ line 8, column 7. stage( 'Build' ) I am on 2.176.1 so this may be a version problem. Just in case can you please send me the screenshots of the job definition to make sure I'm doing this correctly.    
          Hide
          mbrunton27 Matthew Brunton added a comment -

          Thanks for responding. I'll supply screenshots of the setup I'm using to reproduce. I'm also using the Pipeline: Declarative plugin 1.5, which was recently released to add matrix support to declarative pipelines.

          Show
          mbrunton27 Matthew Brunton added a comment - Thanks for responding. I'll supply screenshots of the setup I'm using to reproduce. I'm also using the Pipeline: Declarative plugin 1.5, which was recently released to add matrix support to declarative pipelines.
          Hide
          p4karl Karl Wirth added a comment -

          Hi Matthew Brunton - Thanks. I'll try again and let you know how it goes.

          Show
          p4karl Karl Wirth added a comment - Hi Matthew Brunton - Thanks. I'll try again and let you know how it goes.
          Hide
          p4karl Karl Wirth added a comment -

          Hi Matthew Brunton - The problem was that I needed the latest version of the declarative pipeline plugin.

          I'm easily able to reproduce the behavior and have attached the complete console outputs for with and without excludes.

          I'll check where the message 'No branches to run' comes from and discuss with the developers to see if that cab tell us if the is a p4-plugin or something else problem.

          PerforceTestingConsoleOutput_WithExcludes.txt

          PerforceTestingConsoleOutput_WithOUT_Excludes.txt

          Show
          p4karl Karl Wirth added a comment - Hi Matthew Brunton - The problem was that I needed the latest version of the declarative pipeline plugin. I'm easily able to reproduce the behavior and have attached the complete console outputs for with and without excludes. I'll check where the message 'No branches to run' comes from and discuss with the developers to see if that cab tell us if the is a p4-plugin or something else problem. PerforceTestingConsoleOutput_WithExcludes.txt PerforceTestingConsoleOutput_WithOUT_Excludes.txt
          Hide
          mbrunton27 Matthew Brunton added a comment -

          Thanks for looking into this. I wasn't sure which plugin was causing the issue, it's possible that it's either the p4-plugin or something overlooked when pipeline-model-definition-plugin was updated.

          Show
          mbrunton27 Matthew Brunton added a comment - Thanks for looking into this. I wasn't sure which plugin was causing the issue, it's possible that it's either the p4-plugin or something overlooked when pipeline-model-definition-plugin was updated.
          Hide
          p4karl Karl Wirth added a comment -

          Hi Matthew Brunton - Here's the line of code:

          https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/steps/ParallelStepExecution.java#L42

          I have worked through this with the developers and we don't think this is a problem with our code in p4-plugin.

          There is a slight chance that maybe we need to be inheriting a new  version of one of our dependencies but as it works when in a 'Jenkinsfile' or without the 'excludes' there's more chance it's problem in a library higher up such as the general SCM module.

          If this does turn out to be a bug on our side please let us know.

          Show
          p4karl Karl Wirth added a comment - Hi Matthew Brunton - Here's the line of code: https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/steps/ParallelStepExecution.java#L42 I have worked through this with the developers and we don't think this is a problem with our code in p4-plugin. There is a slight chance that maybe we need to be inheriting a new  version of one of our dependencies but as it works when in a 'Jenkinsfile' or without the 'excludes' there's more chance it's problem in a library higher up such as the general SCM module. If this does turn out to be a bug on our side please let us know.
          Hide
          miquel_tubau Miquel Tubau added a comment -

          Hi, we have the same problem.

          We created a simple pipeline to reproduce the error. When we use the pipeline directly into the pipeline script it works. When it runs inside a shared library it founds no branches to run.

          We are not using the p4-plugin.

          Here is our test pipeline:

          pipeline {
              agent none
          
              stages {
                  stage('Start') {
                      steps {
                          echo "start"
                      }
                  }
                  stage("matrix") {
                      matrix {
                          axes {
                              axis {
                                  name 'PLATFORM'
                                  values 'amazon', 'ios', 'android', 'osx', 'win', 'emscripten', 'linux', 'uwp'
                              }
                              axis {
                                  name 'ENV'
                                  values 'qa', 'live'
                              }
                          }
                          excludes {
                              exclude {
                                  axis {
                                      name 'PLATFORM'
                                      values 'amazon', 'ios', 'android', 'win', 'emscripten', 'linux', 'uwp'
                                  }
                                  axis {
                                      name 'ENV'
                                      values 'live'
                                  }
                              }
                          }
                          
                          stages {
                              stage("build") {
                                  steps {
                                      echo "${PLATFORM} - ${ENV}"
                                  }
                              }
                          }
                      }
                  }
                  stage('Push') {
                      steps {
                          echo "push"
                      }
                  }
              }
          }
          
          Show
          miquel_tubau Miquel Tubau added a comment - Hi, we have the same problem. We created a simple pipeline to reproduce the error. When we use the pipeline directly into the pipeline script it works. When it runs inside a shared library it founds no branches to run. We are not using the p4-plugin. Here is our test pipeline: pipeline { agent none stages { stage( 'Start' ) { steps { echo "start" } } stage( "matrix" ) { matrix { axes { axis { name 'PLATFORM' values 'amazon' , 'ios' , 'android' , 'osx' , 'win' , 'emscripten' , 'linux' , 'uwp' } axis { name 'ENV' values 'qa' , 'live' } } excludes { exclude { axis { name 'PLATFORM' values 'amazon' , 'ios' , 'android' , 'win' , 'emscripten' , 'linux' , 'uwp' } axis { name 'ENV' values 'live' } } } stages { stage( "build" ) { steps { echo "${PLATFORM} - ${ENV}" } } } } } stage( 'Push' ) { steps { echo "push" } } } }

            People

            • Assignee:
              Unassigned
              Reporter:
              mbrunton27 Matthew Brunton
            • Votes:
              3 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: