Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40986

Matrix structure for Declarative Pipeline

    XMLWordPrintable

    Details

    • Epic Link:
    • Sprint:
      Declarative backlog
    • Similar Issues:

      Attachments

        Issue Links

          Activity

          Hide
          jamesdumay James Dumay added a comment -

          Arthur Lutz would something like this solve the matrix capability you are looking for?

          pipeline {
            stages {
              stage(‘browser tests’) {
                matrix {
                  variants {
                    variant (‘firefox’) {
                      agent { node “linux+firefox” }
                	    environment { key: value }
                    }
                    variant ('chrome') {
                      agent { node 'linux+chrome' }
                	    environment { key: value }
                    }
                	}
                  environment { … }
                  stages { … }
                }
              }
          }
          
          Show
          jamesdumay James Dumay added a comment - Arthur Lutz would something like this solve the matrix capability you are looking for? pipeline { stages { stage(‘browser tests’) { matrix { variants { variant (‘firefox’) { agent { node “linux+firefox” } environment { key: value } } variant ( 'chrome' ) { agent { node 'linux+chrome' } environment { key: value } } } environment { … } stages { … } } } }
          Hide
          robhales Robert Hales added a comment -

          I was just pointed at this JIRA today. The syntax above would be extremely wordy for any by the simplest combination. Even something with 4 OSs and 3 variants would be 12 blocks that would have to be written.  This would be much easier to work with if you could provide a map of lists or list of lists and recursively iterate over them to come up with all the combinations. I have done this in pipeline using groovy code and it works well this way. An official syntax might be handy. 

           

           

          https://stackoverflow.com/questions/46601999/jenkins-pipeline-multiconfiguration-project/46675227#46675227

           

          Show
          robhales Robert Hales added a comment - I was just pointed at this JIRA today. The syntax above would be extremely wordy for any by the simplest combination. Even something with 4 OSs and 3 variants would be 12 blocks that would have to be written.  This would be much easier to work with if you could provide a map of lists or list of lists and recursively iterate over them to come up with all the combinations. I have done this in pipeline using groovy code and it works well this way. An official syntax might be handy.      https://stackoverflow.com/questions/46601999/jenkins-pipeline-multiconfiguration-project/46675227#46675227  
          Hide
          abayer Andrew Bayer added a comment -

          Starting to think this one through now that JENKINS-46809 is churning along. The body will basically be a group as will be added by JENKINS-46809, so that part's done. That leaves the axis definitions.

          First, I'm thinking we do ape Matrix jobs - not defining a full "variant" but individual axes and their possible values, leaving the behind the scenes magic to come up with the combinations. My initial thoughts are that we have the following axis "types":

          • name - a base name that we make unique by combining the other axes somehow. Optional, probably. All the others are definitely optional.
          • agent - which would be 1..n possible agent block contents.
          • environmentVariable - better name still to come. =) This would be a single env var and its 1..n possible values. You can specify more than one of these per matrix, but again thinking of how Matrix jobs work, we don't want to just have a big ol' block of multiple variables that you have to copy and tweak a few times to get all the combinations you want. The current thinking on JENKINS-46809 will let you specify an environment block for a stage group, so if you've got common env vars across all the permutations, that's where you'd define them.
          • tools - like agent, 1..n possible tools block contents. Again, similar to Matrix jobs letting you use JDK or Maven version as an axis.

          I am nowhere near thinking of the syntax yet, and this is still months away from actually happening (JENKINS-46809 has to be completed, Blue Ocean and the editor have to be updated accordingly, and then we can implement matrix in Declarative followed by its Blue Ocean/editor changes), but I definitely see how we'll get there now implementation-wise. Woo.

          Show
          abayer Andrew Bayer added a comment - Starting to think this one through now that JENKINS-46809 is churning along. The body will basically be a group as will be added by JENKINS-46809 , so that part's done. That leaves the axis definitions. First, I'm thinking we do ape Matrix jobs - not defining a full "variant" but individual axes and their possible values, leaving the behind the scenes magic to come up with the combinations. My initial thoughts are that we have the following axis "types": name - a base name that we make unique by combining the other axes somehow. Optional, probably. All the others are definitely optional. agent - which would be 1..n possible agent block contents. environmentVariable - better name still to come. =) This would be a single env var and its 1..n possible values. You can specify more than one of these per matrix , but again thinking of how Matrix jobs work, we don't want to just have a big ol' block of multiple variables that you have to copy and tweak a few times to get all the combinations you want. The current thinking on JENKINS-46809 will let you specify an environment block for a stage group , so if you've got common env vars across all the permutations, that's where you'd define them. tools - like agent , 1..n possible tools block contents. Again, similar to Matrix jobs letting you use JDK or Maven version as an axis. I am nowhere near thinking of the syntax yet, and this is still months away from actually happening ( JENKINS-46809 has to be completed, Blue Ocean and the editor have to be updated accordingly, and then we can implement matrix in Declarative followed by its Blue Ocean/editor changes), but I definitely see how we'll get there now implementation-wise. Woo.
          Hide
          neothethird Jan Sprinz added a comment -

          Any news on this? Seems like most of the prerequisites are out of the way and this would be an absolute killer feature to have for declarative pipelines.

          I know it's way to early to think about visualization, but i'm not sure it's a good idea to include this in the normal blueocean pipeline view, since it would get really cluttered really fast. Maybe there could be a popup presenting something similar to the visualization of matrix builds in classic jenkins? I have a job with seven dimensions, i won't even try to imagine what it would look like to have all of those displayed in the normal pipeline view...

          Show
          neothethird Jan Sprinz added a comment - Any news on this? Seems like most of the prerequisites are out of the way and this would be an absolute killer feature to have for declarative pipelines. I know it's way to early to think about visualization, but i'm not sure it's a good idea to include this in the normal blueocean pipeline view, since it would get really cluttered really fast. Maybe there could be a popup presenting something similar to the visualization of matrix builds in classic jenkins? I have a job with seven dimensions, i won't even try to imagine what it would look like to have all of those displayed in the normal pipeline view...
          Hide
          seckler Steffen Seckler added a comment -

          could we up the priority of this?
          seeing that most major other CI tools have this kind of feature, it should be implemented in jenkins rather soon...

          Show
          seckler Steffen Seckler added a comment - could we up the priority of this? seeing that most major other CI tools have this kind of feature, it should be implemented in jenkins rather soon...
          Hide
          neothethird Jan Sprinz added a comment -

          Yes please! It's a must-have, really. The workaround of using loops in a scripted block for this is ugly and error-prone... All i really want for Christmas Easter is declarative Jenkins matrices.

          Show
          neothethird Jan Sprinz added a comment - Yes please! It's a must-have, really. The workaround of using loops in a scripted block for this is ugly and error-prone... All i really want for Christmas Easter is declarative Jenkins matrices.
          Hide
          trejkaz trejkaz added a comment -

          I drafted up what I would hope this sort of thing would look like, and ended up with something like:

          pipeline {
              // ...
              stages {
                  // ...
                  matrixBuild {
                      matrix [label: 'debian',  prettyName: 'Debian'],
                             [label: 'ubuntu',  prettyName: 'Ubuntu'],
                             [label: 'centos',  prettyName: 'CentOS'],
                             [label: 'macos',   prettyName: 'macOS'],
                             [label: 'windows', prettyName: 'Windows']
          
                      template {
                          stage("Test on ${prettyName}") {
                              agent {
                                  label name
                              }
                              steps {
                                  unstash 'compile-artifacts'
                                  unstash 'dot-gradle'
          
                                  gradle tasks: 'check', switches: '--stacktrace'
                              }
                              post {
                                  always {
                                      junit '*/build/test-results/**/*.xml'
                                  }
                              }
                          }
                      }
                  }
                  // ...
              }
          }
          

          The values in the maps for matrix would just become available in the context for template, and if you wanted to put whole closures in as the values, that's fine too.

          Practicality, though, no idea at all. I was going to try to do it as a pipeline library some day if free time ever came up.

          Show
          trejkaz trejkaz added a comment - I drafted up what I would hope this sort of thing would look like, and ended up with something like: pipeline { // ... stages { // ... matrixBuild { matrix [label: 'debian' , prettyName: 'Debian' ], [label: 'ubuntu' , prettyName: 'Ubuntu' ], [label: 'centos' , prettyName: 'CentOS' ], [label: 'macos' , prettyName: 'macOS' ], [label: 'windows' , prettyName: 'Windows' ] template { stage( "Test on ${prettyName}" ) { agent { label name } steps { unstash 'compile-artifacts' unstash 'dot-gradle' gradle tasks: 'check' , switches: '--stacktrace' } post { always { junit '*/build/test-results /**/ *.xml' } } } } } // ... } } The values in the maps for matrix would just become available in the context for template , and if you wanted to put whole closures in as the values, that's fine too. Practicality, though, no idea at all. I was going to try to do it as a pipeline library some day if free time ever came up.

            People

            • Assignee:
              bitwiseman Liam Newman
              Reporter:
              arthurlutz Arthur Lutz
            • Votes:
              23 Vote for this issue
              Watchers:
              29 Start watching this issue

              Dates

              • Created:
                Updated: