Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42753

Generate runtime model directly from AST model

    Details

    • Epic Link:
    • Sprint:
      Declarative - 1.2
    • Similar Issues:

      Description

      My apologies if this isn't a bug, but I've searched high and low and tried a number of things, but I can't seem to find the syntax ex. to solve what's broken.  After upgrading pipeline plugins to the latest releases (as of today), my declarative pipelines broke because it seems there might no longer be ability to use a property in the agent label.  This was working:

      pipeline{
      
          parameters {
              choice(choices: 'server-cd\server-demo', description: DeployEnvironmentHelp, name: 'Environment')
          } 
      
          environment {
              DEPLOYENVIRONMENT = "${params.Environment}"
          }
      
          agent none
      
          stages {
              stage ("Stage 1") {
                  agent {label DEPLOYENVIRONMENT}
                  echo 'Get Deployment Helper Files from SCM repo(s)...'
              }
          } 
      }

      But the pipeline fails saying that the DEPLOYENVIRONMENT is not a proper value (I no longer have the exact message )  

      As of model def. 1.1.1, is it no longer possible to use an environment variable or property in the agent label?   btw - I'm setting DEPLOYENVIRONMENT so I can do other tests and not have to keep using params.Environment, but in other pipelines I set properties so that I can define which agent based on properties that change in the pipeline.

       

       

        Attachments

          Issue Links

            Activity

            Hide
            bitwiseman Liam Newman added a comment -

            Bulk closing resolved issues.

            Show
            bitwiseman Liam Newman added a comment - Bulk closing resolved issues.
            Hide
            abayer Andrew Bayer added a comment -

            Hi Jeff G - it was gonna be released today, but we've decided to get https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/193 done and included, so I would expect the release next week.

            Show
            abayer Andrew Bayer added a comment - Hi Jeff G - it was gonna be released today, but we've decided to get https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/193 done and included, so I would expect the release next week.
            Hide
            jgrant216 Jeff G added a comment -

            Is there a way to get an expected release date for Declarative 1.2?

            Show
            jgrant216 Jeff G added a comment - Is there a way to get an expected release date for Declarative 1.2?
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            pipeline-model-definition/pom.xml
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/Utils.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/AbstractBuildConditionResponder.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Agent.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Environment.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Libraries.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/MethodsToList.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Options.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Parameters.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PostBuild.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PostStage.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PropertiesToMap.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PropertyOptionContainer.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Root.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stage.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/StageConditionals.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stages.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/StepBlockWithOtherArgs.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/StepsBlock.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Tools.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Triggers.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ASTParserUtils.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/BlockStatementMatch.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/Converter.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/RuntimeASTTransformer.groovy
            pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ModelStepLoader.java
            pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/SyntheticStageGraphListener.java
            pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/parser/GroovyShellDecoratorImpl.java
            pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/AllOfConditional.java
            pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/AnyOfConditional.java
            pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/BranchConditional.java
            pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/EnvironmentConditional.java
            pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/ExpressionConditional.java
            pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/NotConditional.java
            pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ClosureModelTranslator.groovy
            pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/LibrariesTranslator.groovy
            pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/MethodsToListTranslator.groovy
            pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy
            pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/OptionsTranslator.groovy
            pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/PropertiesToMapTranslator.groovy
            pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/ExpressionConditionalScript.groovy
            pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/OptionsTest.java
            pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ParametersTest.java
            pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ToolsTest.java
            pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/TriggersTest.java
            pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/steps/CredentialWrapperStepTest.java
            pipeline-model-definition/src/test/resources/credentialsUsedInWhenEnv.groovy
            pipeline-model-definition/src/test/resources/credentialsUsedInWhenExpression.groovy
            pipeline-model-definition/src/test/resources/envVarInOptions.groovy
            pipeline-model-definition/src/test/resources/envVarInParameters.groovy
            pipeline-model-definition/src/test/resources/envVarInTools.groovy
            pipeline-model-definition/src/test/resources/envVarInTriggers.groovy
            pipeline-model-definition/src/test/resources/envVarInWrapper.groovy
            pipeline-model-extensions/pom.xml
            pipeline-model-extensions/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/DeclarativeStageConditionalDescriptor.java
            pom.xml
            http://jenkins-ci.org/commit/pipeline-model-definition-plugin/963d58beea949d4d8eb468986b0908b075205b9c
            Log:
            [FIXED JENKINS-42753] Use AST transformation to generate runtime model

            There've been any number of problems in the past due to the closure
            translation approach for going from `pipeline

            { ... }

            ` to the `Root`
            object we need to actually run. I won't bother going into them here,
            just...yeah. There's lots. Until this change goes in, there are 4
            special CPS-transformed classes for the translation, plus truly
            bizarre special-casing logic for handling environment and when
            expressions via looping circuits of script.evaluate calls. I'm
            serious. It's sad.

            This replaces all that by instead generating Root and its children via
            an AST transformation at parsing/validation time. It gets us away from
            ever having to worry about translating or evaluating within the CPS
            code, using lazily evaluated closures to support things like declaring
            cross-referencing environment variables out of order, or environment
            variables depending on credentials variables, etc...

            Anyway, it works. It's more maintainable. It's cleaner. It's actually
            very slightly faster (since we've eliminated the closure translation
            stuff from CPS execution, and got rid of a now-redundant additional
            parse/validate run to get the execution model). And it's completely
            compatible syntax-wise (hopefully! It's not impossible that I missed
            some weird expressions in environment variable values, but we'll cross
            that bridge if we get to it).

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pipeline-model-definition/pom.xml pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/Utils.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/AbstractBuildConditionResponder.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Agent.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Environment.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Libraries.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/MethodsToList.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Options.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Parameters.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PostBuild.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PostStage.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PropertiesToMap.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PropertyOptionContainer.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Root.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stage.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/StageConditionals.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stages.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/StepBlockWithOtherArgs.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/StepsBlock.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Tools.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Triggers.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ASTParserUtils.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/BlockStatementMatch.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/Converter.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/RuntimeASTTransformer.groovy pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ModelStepLoader.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/SyntheticStageGraphListener.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/parser/GroovyShellDecoratorImpl.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/AllOfConditional.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/AnyOfConditional.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/BranchConditional.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/EnvironmentConditional.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/ExpressionConditional.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/NotConditional.java pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ClosureModelTranslator.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/LibrariesTranslator.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/MethodsToListTranslator.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/OptionsTranslator.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/PropertiesToMapTranslator.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/ExpressionConditionalScript.groovy pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/OptionsTest.java pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ParametersTest.java pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ToolsTest.java pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/TriggersTest.java pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/steps/CredentialWrapperStepTest.java pipeline-model-definition/src/test/resources/credentialsUsedInWhenEnv.groovy pipeline-model-definition/src/test/resources/credentialsUsedInWhenExpression.groovy pipeline-model-definition/src/test/resources/envVarInOptions.groovy pipeline-model-definition/src/test/resources/envVarInParameters.groovy pipeline-model-definition/src/test/resources/envVarInTools.groovy pipeline-model-definition/src/test/resources/envVarInTriggers.groovy pipeline-model-definition/src/test/resources/envVarInWrapper.groovy pipeline-model-extensions/pom.xml pipeline-model-extensions/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/DeclarativeStageConditionalDescriptor.java pom.xml http://jenkins-ci.org/commit/pipeline-model-definition-plugin/963d58beea949d4d8eb468986b0908b075205b9c Log: [FIXED JENKINS-42753] Use AST transformation to generate runtime model There've been any number of problems in the past due to the closure translation approach for going from `pipeline { ... } ` to the `Root` object we need to actually run. I won't bother going into them here, just...yeah. There's lots. Until this change goes in, there are 4 special CPS-transformed classes for the translation, plus truly bizarre special-casing logic for handling environment and when expressions via looping circuits of script.evaluate calls. I'm serious. It's sad. This replaces all that by instead generating Root and its children via an AST transformation at parsing/validation time. It gets us away from ever having to worry about translating or evaluating within the CPS code, using lazily evaluated closures to support things like declaring cross-referencing environment variables out of order, or environment variables depending on credentials variables, etc... Anyway, it works. It's more maintainable. It's cleaner. It's actually very slightly faster (since we've eliminated the closure translation stuff from CPS execution, and got rid of a now-redundant additional parse/validate run to get the execution model). And it's completely compatible syntax-wise (hopefully! It's not impossible that I missed some weird expressions in environment variable values, but we'll cross that bridge if we get to it).
            Hide
            jgrant216 Jeff G added a comment -

            Adding to the list of users who are blocked from upgrading this pipeline plugin since March.

            Show
            jgrant216 Jeff G added a comment - Adding to the list of users who are blocked from upgrading this pipeline plugin since March.

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                jedavis Jason Davis
              • Votes:
                5 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: