Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41335

Allow variables and functions to be defined within pipeline to be used in any stage

    Details

    • Similar Issues:
    • Epic Link:

      Description

      I have several use cases where I may operate on a list or map on different stages of a pipeline. This variable may not be known ahead of time - it could be defined within a stage. I'd like a syntax that allows me to define variables within a pipeline.

      Today, I have to define a shared variable before entering pipeline {}.

      Jenkinsfile
      #!/bin/groovy
      // Define outside of pipeline to make sure accessible in all script {} blocks
      def my_list
      
      pipeline {
          agent { label 'label' }
          stages {
              stage('stage1') {
                steps {
                 script {
                     my_list = [1,2,3]
                 }
                }
              }
          
              stage('stage2') {
                steps {
                  script {
                      for(int i=0; i<my_list.size();i++) {
                          echo "Doing something with ${my_list[i]}"
                      }
                  }
                }
              }
          }
      }
      

      A simple approach would be to allow "def" to work anywhere within pipeline {}. This avoids any overly new syntax for pipeline.

      Another option would allow the user to provide a special pipeline {} block for defining non-env/params style variables. This could be helpful in allowing the user to use items available in script {} for defining shared state.

      Jenkinsfile with possible example syntax
      #!/bin/groovy
      
      
      pipeline {
          agent { label 'label' }    
      
          define {
             def my_map = [:] //empty map
             def my_list //undefined shared variable
           }
      
          stages {
              stage('stage1') {
                steps {
                  script {
                    //If def can be anywhere in pipeline {}, drop script {}
                    my_list = [1,2,3]
                  }
                }
              }
          
              stage('stage2') {
                steps {
                  script {
                      for(int i=0; i<my_list.size();i++) {
                          echo "Doing something with ${my_list[i]}"
                      }
                  }
                }
              }
          }
      }
      

        Attachments

          Issue Links

            Activity

            Hide
            ftherien Francis Therien added a comment -

            Sam Van Oort I thought I had read somehwhere that the roadmap was for scripted to eventually be deprecated, and that declarative would replace it.

            In any case, I second Liam's point that it would be interesting to see documents explaining this vision. It would help, at the very least,  people to make long term decisions for their CI architecture around Jenkins.

            Show
            ftherien Francis Therien added a comment - Sam Van Oort I thought I had read somehwhere that the roadmap was for scripted to eventually be deprecated, and that declarative would replace it. In any case, I second Liam's point that it would be interesting to see documents explaining this vision. It would help, at the very least,  people to make long term decisions for their CI architecture around Jenkins.
            Hide
            svanoort Sam Van Oort added a comment -

            Francis Therien Caveat: take anything I say here as a personal opinion only. By all means, people should default to Declarative (or Declarative + Shared Libraries as needs grow) because it's easier and shows an opinionated "lit path" to successful CI/CD. And yes, we devote more work to Declarative specifically – I think the split is roughly 30/60/10 for work that is Declarative-only / General Pipeline for both / Scripted-only. That's because Declarative is newer and intended to be the "easy" mechanism that covers 80% of needs, where Scripted lets you roll-your-own and the majority of the features are just General Pipeline Stuff + Groovy.

            But as far as actually deprecating Scripted? You see people propose it from time to time, generally without understanding what it actually means, because Declarative is joined at the hip to Scripted and runs on top of it. For Declarative to replace Scripted, it which would have to absorb all the crazy use cases people can implement in Scripted due to its flexibility. Not to mention the number of organizations who would likely drop Pipeline and Jenkins entirely if they made a major investment in Scripted and Shared Libraries and had to walk away from 'em. I'd encourage anybody who thinks it's a good idea to come talk to me personally; I'll be happy to explain the other reasons why it's both extremely technically/architecturally difficult and a very poor idea (not going to run through it here because it's practically a novella).

            Ergo why I push back on trying to insert Scripted-like features such as this into Declarative – we don't want to find ourselves maintaining a new programming language. Better to make Declarative the best config language it can be, and delegate fancy stuff to a proper programming language (Groovy).

            Hope that all makes sense!

            Show
            svanoort Sam Van Oort added a comment - Francis Therien Caveat: take anything I say here as a personal opinion only. By all means, people should default to Declarative (or Declarative + Shared Libraries as needs grow) because it's easier and shows an opinionated "lit path" to successful CI/CD. And yes, we devote more work to Declarative specifically – I think the split is roughly 30/60/10 for work that is Declarative-only / General Pipeline for both / Scripted-only. That's because Declarative is newer and intended to be the "easy" mechanism that covers 80% of needs, where Scripted lets you roll-your-own and the majority of the features are just General Pipeline Stuff + Groovy. But as far as actually deprecating Scripted? You see people propose it from time to time, generally without understanding what it actually means, because Declarative is joined at the hip to Scripted and runs on top of it. For Declarative to replace Scripted, it which would have to absorb all the crazy use cases people can implement in Scripted due to its flexibility. Not to mention the number of organizations who would likely drop Pipeline and Jenkins entirely if they made a major investment in Scripted and Shared Libraries and had to walk away from 'em. I'd encourage anybody who thinks it's a good idea to come talk to me personally; I'll be happy to explain the other reasons why it's both extremely technically/architecturally difficult and a very poor idea (not going to run through it here because it's practically a novella). Ergo why I push back on trying to insert Scripted-like features such as this into Declarative – we don't want to find ourselves maintaining a new programming language. Better to make Declarative the best config language it can be, and delegate fancy stuff to a proper programming language (Groovy). Hope that all makes sense!
            Hide
            bitwiseman Liam Newman added a comment -

            Andrew Bayer

            I want to be clear, I personally am willing to accept that this is not only out of scope, but also counter to core design.  My point is I'd like to make sure that core design and direction are documented and understandable.

             

            Show
            bitwiseman Liam Newman added a comment - Andrew Bayer I want to be clear, I personally am willing to accept that this is not only out of scope, but also counter to core design.  My point is I'd like to make sure that core design and direction are documented and understandable.  
            Hide
            fmiedniak Florian Miedniak added a comment -

            I very agree with Liam Newman and Francis Therien: Having some background information about the underlying design would it make a lot easier to build Jenkins pipelines the "right" way: Luckily, they are actively developed, thus it is extremely important for users to know about things you should not use because they are getting obsoleted in near future due to changes / advances in design.
            Frankly, today it is sometimes quite challening to justify the maintenance effort for pipelines, if the answer to the question "why did the jenkins update break our pipe?" repeatedly is: "they decided to change it" ...
            IMO, in the recent past Jenkins came to a very good way of communicating and handling of security related issues (pre-announcement mails, changelog, ...)
            Having a design outline and a roadmap with milestones IMO would be a good starting point to achieve a similar level of transparency for the design of a complex, production-grade CI system that Jenkins is.

            Show
            fmiedniak Florian Miedniak added a comment - I very agree with Liam Newman and Francis Therien : Having some background information about the underlying design would it make a lot easier to build Jenkins pipelines the "right" way: Luckily, they are actively developed, thus it is extremely important for users to know about things you should not use because they are getting obsoleted in near future due to changes / advances in design. Frankly, today it is sometimes quite challening to justify the maintenance effort for pipelines, if the answer to the question "why did the jenkins update break our pipe?" repeatedly is: "they decided to change it" ... IMO, in the recent past Jenkins came to a very good way of communicating and handling of security related issues (pre-announcement mails, changelog, ...) Having a design outline and a roadmap with milestones IMO would be a good starting point to achieve a similar level of transparency for the design of a complex, production-grade CI system that Jenkins is.
            Hide
            bitwiseman Liam Newman added a comment -

            Bulk closing resolved issues.

            Show
            bitwiseman Liam Newman added a comment - Bulk closing resolved issues.

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                rpocase Robby Pocase
              • Votes:
                43 Vote for this issue
                Watchers:
                57 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: