Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40986

Matrix structure for Declarative Pipeline

    Details

    • Epic Link:
    • Sprint:
      Declarative backlog
    • Similar Issues:

      Attachments

        Issue Links

          Activity

          Hide
          jamesdumay James Dumay added a comment -

          Arthur Lutz would something like this solve the matrix capability you are looking for?

          pipeline {
            stages {
              stage(‘browser tests’) {
                matrix {
                  variants {
                    variant (‘firefox’) {
                      agent { node “linux+firefox” }
                	    environment { key: value }
                    }
                    variant ('chrome') {
                      agent { node 'linux+chrome' }
                	    environment { key: value }
                    }
                	}
                  environment { … }
                  stages { … }
                }
              }
          }
          
          Show
          jamesdumay James Dumay added a comment - Arthur Lutz would something like this solve the matrix capability you are looking for? pipeline { stages { stage(‘browser tests’) { matrix { variants { variant (‘firefox’) { agent { node “linux+firefox” } environment { key: value } } variant ( 'chrome' ) { agent { node 'linux+chrome' } environment { key: value } } } environment { … } stages { … } } } }
          Hide
          robhales Robert Hales added a comment -

          I was just pointed at this JIRA today. The syntax above would be extremely wordy for any by the simplest combination. Even something with 4 OSs and 3 variants would be 12 blocks that would have to be written.  This would be much easier to work with if you could provide a map of lists or list of lists and recursively iterate over them to come up with all the combinations. I have done this in pipeline using groovy code and it works well this way. An official syntax might be handy. 

           

           

          https://stackoverflow.com/questions/46601999/jenkins-pipeline-multiconfiguration-project/46675227#46675227

           

          Show
          robhales Robert Hales added a comment - I was just pointed at this JIRA today. The syntax above would be extremely wordy for any by the simplest combination. Even something with 4 OSs and 3 variants would be 12 blocks that would have to be written.  This would be much easier to work with if you could provide a map of lists or list of lists and recursively iterate over them to come up with all the combinations. I have done this in pipeline using groovy code and it works well this way. An official syntax might be handy.      https://stackoverflow.com/questions/46601999/jenkins-pipeline-multiconfiguration-project/46675227#46675227  
          Hide
          abayer Andrew Bayer added a comment -

          Starting to think this one through now that JENKINS-46809 is churning along. The body will basically be a group as will be added by JENKINS-46809, so that part's done. That leaves the axis definitions.

          First, I'm thinking we do ape Matrix jobs - not defining a full "variant" but individual axes and their possible values, leaving the behind the scenes magic to come up with the combinations. My initial thoughts are that we have the following axis "types":

          • name - a base name that we make unique by combining the other axes somehow. Optional, probably. All the others are definitely optional.
          • agent - which would be 1..n possible agent block contents.
          • environmentVariable - better name still to come. =) This would be a single env var and its 1..n possible values. You can specify more than one of these per matrix, but again thinking of how Matrix jobs work, we don't want to just have a big ol' block of multiple variables that you have to copy and tweak a few times to get all the combinations you want. The current thinking on JENKINS-46809 will let you specify an environment block for a stage group, so if you've got common env vars across all the permutations, that's where you'd define them.
          • tools - like agent, 1..n possible tools block contents. Again, similar to Matrix jobs letting you use JDK or Maven version as an axis.

          I am nowhere near thinking of the syntax yet, and this is still months away from actually happening (JENKINS-46809 has to be completed, Blue Ocean and the editor have to be updated accordingly, and then we can implement matrix in Declarative followed by its Blue Ocean/editor changes), but I definitely see how we'll get there now implementation-wise. Woo.

          Show
          abayer Andrew Bayer added a comment - Starting to think this one through now that JENKINS-46809 is churning along. The body will basically be a group as will be added by JENKINS-46809 , so that part's done. That leaves the axis definitions. First, I'm thinking we do ape Matrix jobs - not defining a full "variant" but individual axes and their possible values, leaving the behind the scenes magic to come up with the combinations. My initial thoughts are that we have the following axis "types": name - a base name that we make unique by combining the other axes somehow. Optional, probably. All the others are definitely optional. agent - which would be 1..n possible agent block contents. environmentVariable - better name still to come. =) This would be a single env var and its 1..n possible values. You can specify more than one of these per matrix , but again thinking of how Matrix jobs work, we don't want to just have a big ol' block of multiple variables that you have to copy and tweak a few times to get all the combinations you want. The current thinking on JENKINS-46809 will let you specify an environment block for a stage group , so if you've got common env vars across all the permutations, that's where you'd define them. tools - like agent , 1..n possible tools block contents. Again, similar to Matrix jobs letting you use JDK or Maven version as an axis. I am nowhere near thinking of the syntax yet, and this is still months away from actually happening ( JENKINS-46809 has to be completed, Blue Ocean and the editor have to be updated accordingly, and then we can implement matrix in Declarative followed by its Blue Ocean/editor changes), but I definitely see how we'll get there now implementation-wise. Woo.
          Hide
          neothethird Jan Sprinz added a comment -

          Any news on this? Seems like most of the prerequisites are out of the way and this would be an absolute killer feature to have for declarative pipelines.

          I know it's way to early to think about visualization, but i'm not sure it's a good idea to include this in the normal blueocean pipeline view, since it would get really cluttered really fast. Maybe there could be a popup presenting something similar to the visualization of matrix builds in classic jenkins? I have a job with seven dimensions, i won't even try to imagine what it would look like to have all of those displayed in the normal pipeline view...

          Show
          neothethird Jan Sprinz added a comment - Any news on this? Seems like most of the prerequisites are out of the way and this would be an absolute killer feature to have for declarative pipelines. I know it's way to early to think about visualization, but i'm not sure it's a good idea to include this in the normal blueocean pipeline view, since it would get really cluttered really fast. Maybe there could be a popup presenting something similar to the visualization of matrix builds in classic jenkins? I have a job with seven dimensions, i won't even try to imagine what it would look like to have all of those displayed in the normal pipeline view...
          Hide
          seckler Steffen Seckler added a comment -

          could we up the priority of this?
          seeing that most major other CI tools have this kind of feature, it should be implemented in jenkins rather soon...

          Show
          seckler Steffen Seckler added a comment - could we up the priority of this? seeing that most major other CI tools have this kind of feature, it should be implemented in jenkins rather soon...
          Hide
          neothethird Jan Sprinz added a comment -

          Yes please! It's a must-have, really. The workaround of using loops in a scripted block for this is ugly and error-prone... All i really want for Christmas Easter is declarative Jenkins matrices.

          Show
          neothethird Jan Sprinz added a comment - Yes please! It's a must-have, really. The workaround of using loops in a scripted block for this is ugly and error-prone... All i really want for Christmas Easter is declarative Jenkins matrices.
          Hide
          trejkaz trejkaz added a comment -

          I drafted up what I would hope this sort of thing would look like, and ended up with something like:

          pipeline {
              // ...
              stages {
                  // ...
                  matrixBuild {
                      matrix [label: 'debian',  prettyName: 'Debian'],
                             [label: 'ubuntu',  prettyName: 'Ubuntu'],
                             [label: 'centos',  prettyName: 'CentOS'],
                             [label: 'macos',   prettyName: 'macOS'],
                             [label: 'windows', prettyName: 'Windows']
          
                      template {
                          stage("Test on ${prettyName}") {
                              agent {
                                  label name
                              }
                              steps {
                                  unstash 'compile-artifacts'
                                  unstash 'dot-gradle'
          
                                  gradle tasks: 'check', switches: '--stacktrace'
                              }
                              post {
                                  always {
                                      junit '*/build/test-results/**/*.xml'
                                  }
                              }
                          }
                      }
                  }
                  // ...
              }
          }
          

          The values in the maps for matrix would just become available in the context for template, and if you wanted to put whole closures in as the values, that's fine too.

          Practicality, though, no idea at all. I was going to try to do it as a pipeline library some day if free time ever came up.

          Show
          trejkaz trejkaz added a comment - I drafted up what I would hope this sort of thing would look like, and ended up with something like: pipeline { // ... stages { // ... matrixBuild { matrix [label: 'debian' , prettyName: 'Debian' ], [label: 'ubuntu' , prettyName: 'Ubuntu' ], [label: 'centos' , prettyName: 'CentOS' ], [label: 'macos' , prettyName: 'macOS' ], [label: 'windows' , prettyName: 'Windows' ] template { stage( "Test on ${prettyName}" ) { agent { label name } steps { unstash 'compile-artifacts' unstash 'dot-gradle' gradle tasks: 'check' , switches: '--stacktrace' } post { always { junit '*/build/test-results /**/ *.xml' } } } } } // ... } } The values in the maps for matrix would just become available in the context for template , and if you wanted to put whole closures in as the values, that's fine too. Practicality, though, no idea at all. I was going to try to do it as a pipeline library some day if free time ever came up.
          Hide
          maydjin Denis Dudarev added a comment - - edited

          It would be great if not only 2d matrices would be supported(more than 2 axis should be supported). As well only sparsable matrices would be useful.

           

          User case:

           * User has combination of architectures ('x86',  'x86_64', 'armv7', 'armv8')
           * User has combination of platforms ('win32', 'linux', 'osx', 'android')
           * User has combination of compilers ('clang', 'gcc', 'cl', 'icc')
           * User desired that combination '.*'-'linux|osx|android'-'cl' doesn't make sense
           * User desired that combination 'arm.*'-'win32|osx'-'.*' doesn't make sense
           * User desired that combination '.*'-'osx'-'gcc' is not supported yet

           

          It could be regexp syntax, as I offered before, or more imperative like in original "Matrix job". IMHO, only this approach could make sense in 90% user cases.

           

          Also, there should be a way to skip stage/step on given axises combination.

          User case:

          * User has combination of architectures ('x86',  'x86_64', 'armv7', 'armv8')
          * User has combination of platforms ('win32', 'linux', 'osx', 'android')
          * User has combination of compilers ('clang', 'gcc', 'cl', 'icc')
          * User has stage 'build'
          * User has stage 'build MSI package'
          * User desired that 'build' stage makes sense in any combination of axises
          * User desired that 'build MSI package' for '.*'-'linux|osx|android'-'.*' doesn't make sense
          

          Andrew Bayer, please let me know, if there is any design draft of upcoming API.

          Show
          maydjin Denis Dudarev added a comment - - edited It would be great if not only 2d matrices would be supported(more than 2 axis should be supported). As well only sparsable matrices would be useful.   User case: * User has combination of architectures ( 'x86' , 'x86_64' , 'armv7' , 'armv8' ) * User has combination of platforms ( 'win32' , 'linux' , 'osx' , 'android' ) * User has combination of compilers ( 'clang' , 'gcc' , 'cl' , 'icc' ) * User desired that combination '.*' - 'linux|osx|android' - 'cl' doesn't make sense * User desired that combination 'arm.*' - 'win32|osx' - '.*' doesn't make sense * User desired that combination '.*' - 'osx' - 'gcc' is not supported yet   It could be regexp syntax, as I offered before, or more imperative like in original "Matrix job". IMHO, only this approach could make sense in 90% user cases.   Also, there should be a way to skip stage/step on given axises combination. User case: * User has combination of architectures ( 'x86' , 'x86_64' , 'armv7' , 'armv8' ) * User has combination of platforms ( 'win32' , 'linux' , 'osx' , 'android' ) * User has combination of compilers ( 'clang' , 'gcc' , 'cl' , 'icc' ) * User has stage 'build' * User has stage 'build MSI package' * User desired that 'build' stage makes sense in any combination of axises * User desired that 'build MSI package' for '.*' - 'linux|osx|android' - '.*' doesn't make sense Andrew Bayer , please let me know, if there is any design draft of upcoming API.

            People

            • Assignee:
              bitwiseman Liam Newman
              Reporter:
              arthurlutz Arthur Lutz
            • Votes:
              24 Vote for this issue
              Watchers:
              31 Start watching this issue

              Dates

              • Created:
                Updated: