Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-28447

CpsScmFlowDefinition does not resolve variables

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: workflow-cps-plugin
    • Labels:
    • Environment:
      Jenins 1.613, Workflow plugin 1.6
      Also: Jenkins 1.651.1, workflow-job 2.3
    • Similar Issues:

      Description

      When selecting the "Groovy CPS DSL from SCM" option for a worflow job, the SCM plugins do not appear to resolve build parameters or environment variables. I am using the git plugin and when I use it from other jobs I can specify a build parameter, like "BuildBranch", and use that when specifying what branch should be built:

      Branches to build: */${BuildBranch}

      This does not work when I use the Groovy CPS DSL from SCM.

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            Not quite sure I understand how to reproduce here, but bear in mind that CpsScmFlowDefinition is merely a convenience. For more control you can use CpsFlowDefinition (inline scipt) and just specify something like

            node {
              checkout(…) // anything supported by the SCM
              load 'flow.groovy' // run everything else
            }
            
            Show
            jglick Jesse Glick added a comment - Not quite sure I understand how to reproduce here, but bear in mind that CpsScmFlowDefinition is merely a convenience. For more control you can use CpsFlowDefinition (inline scipt) and just specify something like node { checkout(…) // anything supported by the SCM load 'flow.groovy' // run everything else }
            Hide
            nsnewland Nicholas Newland added a comment - - edited

            That's what I did. I simply used the CpsFlowDefinition. Maybe my expectations where wrong, but this is what I was expecting to be able to do.

            Freestyle project:
            1. Define new job
            2. Check "This build is parameterized"
            3. Create string parameter with name "BuildBranch" and default value "master"
            4. Under Source code management select "Git"
            5. in "Branches to build" enter "* / $ BuildBranch"
            6. when the build runs with the default value it checks out the master branch of my git repo and runs the build
            7. If a developer runs the build and enters "feature/awesome" for the BuildBranch parameter, the feature/awesome branch is checked out from git and the build runs.

            When I use the CpsScmFlowDefinition, I can do all of the above and select Git as the SCM and fill in "* / $ BuildBranch in the "Branches to build" text box, but git always complains that there is no branch named "* / $ BuildBranch". It doesn't seem to be resolving the build parameter.

            Show
            nsnewland Nicholas Newland added a comment - - edited That's what I did. I simply used the CpsFlowDefinition. Maybe my expectations where wrong, but this is what I was expecting to be able to do. Freestyle project: 1. Define new job 2. Check "This build is parameterized" 3. Create string parameter with name "BuildBranch" and default value "master" 4. Under Source code management select "Git" 5. in "Branches to build" enter "* / $ BuildBranch" 6. when the build runs with the default value it checks out the master branch of my git repo and runs the build 7. If a developer runs the build and enters "feature/awesome" for the BuildBranch parameter, the feature/awesome branch is checked out from git and the build runs. When I use the CpsScmFlowDefinition, I can do all of the above and select Git as the SCM and fill in "* / $ BuildBranch in the "Branches to build" text box, but git always complains that there is no branch named "* / $ BuildBranch". It doesn't seem to be resolving the build parameter.
            Hide
            mijay Mitya Kononchuk added a comment - - edited

            The workaround @Jesse Glick proposes has one disadvantage: you have to allocate an extra node for the build, while using CpsScmFlowDefinition you don't.

            Show
            mijay Mitya Kononchuk added a comment - - edited The workaround @Jesse Glick proposes has one disadvantage: you have to allocate an extra node for the build, while using CpsScmFlowDefinition you don't.
            Hide
            jglick Jesse Glick added a comment -

            Mitya Kononchuk not if flow.groovy runs inside the same node it was given.

            Show
            jglick Jesse Glick added a comment - Mitya Kononchuk not if flow.groovy runs inside the same node it was given.
            Hide
            nirvine_bnsv Nick Irvine added a comment - - edited

            This is affecting me as well. The Help documentation for the field advertises it as a feature:

            ${ENV_VARIABLE}
            It is also possible to use environment variables. In this case the variables are evaluated and the result is used as described above.
            E.g. ${TREEISH}, refs/tags/${TAGNAME},...

            So one of these is a bug.

            Show
            nirvine_bnsv Nick Irvine added a comment - - edited This is affecting me as well. The Help documentation for the field advertises it as a feature: ${ENV_VARIABLE} It is also possible to use environment variables. In this case the variables are evaluated and the result is used as described above. E.g. ${TREEISH}, refs/tags/${TAGNAME},... So one of these is a bug.
            Hide
            jglick Jesse Glick added a comment -

            Of course the inline help was written for the plugin when only freestyle projects existed.

            Show
            jglick Jesse Glick added a comment - Of course the inline help was written for the plugin when only freestyle projects existed.
            Hide
            nirvine_bnsv Nick Irvine added a comment -

            Jesse Glick: I'm just saying, since this bug is several months old, it would appease me at least if the docs were updated to say something like "(Not available in Workflow Step; see JENKINS-28447)". Seems like a reasonable stop-gap solution, coupled with your suggestion above.

            In fact, I'm willing to contribute a patch if you think that's worthwhile. And provided it's as simple a fix as I imagine – I'm not familiar with the codebase whatsoever.

            Show
            nirvine_bnsv Nick Irvine added a comment - Jesse Glick : I'm just saying, since this bug is several months old, it would appease me at least if the docs were updated to say something like "(Not available in Workflow Step; see JENKINS-28447 )". Seems like a reasonable stop-gap solution, coupled with your suggestion above. In fact, I'm willing to contribute a patch if you think that's worthwhile. And provided it's as simple a fix as I imagine – I'm not familiar with the codebase whatsoever.
            Hide
            jglick Jesse Glick added a comment -

            Well this would apply to any SCM plugin that has any field which gets variables substituted, I expect. Have not investigated in detail.

            Show
            jglick Jesse Glick added a comment - Well this would apply to any SCM plugin that has any field which gets variables substituted, I expect. Have not investigated in detail.
            Hide
            abayer Andrew Bayer added a comment -

            Bumping this to critical - integration with things like the Gerrit trigger plugin with scripts from SCM will be problematic without this.

            Show
            abayer Andrew Bayer added a comment - Bumping this to critical - integration with things like the Gerrit trigger plugin with scripts from SCM will be problematic without this.
            Hide
            jglick Jesse Glick added a comment -

            SCM implementations seem to call Run.getEnvironment(TaskListener) so it is possible an implementation of JENKINS-30910 would automatically fix this, but certainly deserves its own integration tests.

            Show
            jglick Jesse Glick added a comment - SCM implementations seem to call Run.getEnvironment(TaskListener) so it is possible an implementation of JENKINS-30910 would automatically fix this, but certainly deserves its own integration tests.
            Hide
            kad Alexander Kanevskiy added a comment -

            bump. without being able to use variables in pipeline from scm is practically making pipeline type of job useless, as can't be integrated e.g. with Github pull requests processing.

            Show
            kad Alexander Kanevskiy added a comment - bump. without being able to use variables in pipeline from scm is practically making pipeline type of job useless, as can't be integrated e.g. with Github pull requests processing.
            Hide
            docwhat Christian Höltje added a comment -

            So this breaks pipeline jobs using Gerrit. The Gerrit variables are not available in the "Pipeline script from SCM" section. The are not replaced, but left as $FOOBAR; example: git -c core.askpass=true fetch --tags --progress ssh://jenkins@gerrit.example.com:29418/noodles/jenkinsfile.git $GERRIT_REFSPEC

            It breaks other cases too. For example, in the same job above, I've set a build parameter $GERRIT_REFSPEC and it doesn't work either.

            Show
            docwhat Christian Höltje added a comment - So this breaks pipeline jobs using Gerrit. The Gerrit variables are not available in the "Pipeline script from SCM" section. The are not replaced, but left as $FOOBAR ; example: git -c core.askpass=true fetch --tags --progress ssh://jenkins@gerrit.example.com:29418/noodles/jenkinsfile.git $GERRIT_REFSPEC It breaks other cases too. For example, in the same job above, I've set a build parameter $GERRIT_REFSPEC and it doesn't work either.
            Hide
            kmbulebu2 Kevin Bulebush added a comment -

            The suggested workaround will not work if your script uses 'checkout scm'.

            {{java.lang.IllegalStateException: inappropriate context
            at org.jenkinsci.plugins.workflow.multibranch.SCMVar.getValue(SCMVar.java:75)}}

            'checkout scm' is only allowed for CpsScmFlowDefinition type jobs, not the CpsFlowDefinition type job used by the workaround.

            In my case, I fully worked out a Pipeline script to build the master branch, then duped the job and modified it to trigger on patchset-created events in Gerrit. I suppose I'll go back to the original script and modify it to build its own scm object. I'll have to keep in mind that it could be invoked by either a ref-updated or patchset-created event, changing the parameters to the git scm class. It's a bit of a rabbit hole.

            Show
            kmbulebu2 Kevin Bulebush added a comment - The suggested workaround will not work if your script uses 'checkout scm'. {{java.lang.IllegalStateException: inappropriate context at org.jenkinsci.plugins.workflow.multibranch.SCMVar.getValue(SCMVar.java:75)}} 'checkout scm' is only allowed for CpsScmFlowDefinition type jobs, not the CpsFlowDefinition type job used by the workaround. In my case, I fully worked out a Pipeline script to build the master branch, then duped the job and modified it to trigger on patchset-created events in Gerrit. I suppose I'll go back to the original script and modify it to build its own scm object. I'll have to keep in mind that it could be invoked by either a ref-updated or patchset-created event, changing the parameters to the git scm class. It's a bit of a rabbit hole.
            Hide
            gareth_western Gareth Western added a comment - - edited

            Is this a duplicate of JENKINS-34876 and JENKINS-34540?

            Show
            gareth_western Gareth Western added a comment - - edited Is this a duplicate of JENKINS-34876 and JENKINS-34540 ?
            Hide
            jglick Jesse Glick added a comment -

            Have some evidence that JENKINS-30910 would fix this implicitly.

            Show
            jglick Jesse Glick added a comment - Have some evidence that JENKINS-30910 would fix this implicitly.
            Hide
            jglick Jesse Glick added a comment -

            I think I have finally reproduced this in a test, the build ending with something like

            …
             > git rev-parse origin/${VERSION}^{commit} # timeout=10
             > git rev-parse ${VERSION}^{commit} # timeout=10
            ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job.
            Finished: FAILURE
            
            Show
            jglick Jesse Glick added a comment - I think I have finally reproduced this in a test, the build ending with something like … > git rev-parse origin/${VERSION}^{commit} # timeout=10 > git rev-parse ${VERSION}^{commit} # timeout=10 ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job. Finished: FAILURE
            Hide
            jwillemsen Johnny Willemsen added a comment -

            Would be useful for our use case when the Script Path could use the job name, that way we can have a generic setup where we just have to copy jobs and rename them, the script path will use the correct file for that job from scm

            Show
            jwillemsen Johnny Willemsen added a comment - Would be useful for our use case when the Script Path could use the job name, that way we can have a generic setup where we just have to copy jobs and rename them, the script path will use the correct file for that job from scm
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/test/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinitionTest.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/86c7ea988f371a8b50d2a2ed41ad2fcb4cd6de25
            Log:
            [FIXED JENKINS-28447] CpsScmFlowDefinition does not resolve variables.
            Actual fix is implied by JENKINS-30910; this is just the integration test.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/test/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinitionTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/86c7ea988f371a8b50d2a2ed41ad2fcb4cd6de25 Log: [FIXED JENKINS-28447] CpsScmFlowDefinition does not resolve variables. Actual fix is implied by JENKINS-30910 ; this is just the integration test.
            Hide
            jglick Jesse Glick added a comment -

            Johnny Willemsen this issue is about variable expansion in the SCM configuration. Your case is something else, best filed as a separate RFE. (Or really you may just want to be using the Job DSL plugin.)

            Show
            jglick Jesse Glick added a comment - Johnny Willemsen this issue is about variable expansion in the SCM configuration. Your case is something else, best filed as a separate RFE. (Or really you may just want to be using the Job DSL plugin.)
            Hide
            atikhono Anna Tikhonova added a comment -

            I don't understand why it's marked as resolved. I can still reproduce it with the latest workflow-cps (2.30) and workflow-job (2.10) plugins. Reproducible as it is in the description.

            • Create pipeline project
            • In configure: check 'This project is parametrized' and add a String Parameter "branch" with some default value. Select 'Pipeline script from SCM' in definition. Fill in repository URL, Branch Specifier as ${branch} and Script Path (Jenkinsfile)
            • At URL, create a git repository with Jenkinsfile. Put 'node('master') { checkout scm }' in the Jenkinsfile.
            • Run the job. It will fail with: 
              hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/${branch}:refs/remotes/origin/${branch} --prune" returned status code 128:
              stdout: 
              stderr: fatal: Couldn't find remote ref refs/heads/${branch}
              
              	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1833)
              	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1552)
              	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:66)
              	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:343)
              	at jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:306)
              	at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:196)
              	at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:172)
              	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:99)
              	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:59)
              	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:232)
              	at hudson.model.ResourceController.execute(ResourceController.java:98)
              	at hudson.model.Executor.run(Executor.java:405)
              
            Show
            atikhono Anna Tikhonova added a comment - I don't understand why it's marked as resolved. I can still reproduce it with the latest workflow-cps (2.30) and workflow-job (2.10) plugins. Reproducible as it is in the description. Create pipeline project In configure: check 'This project is parametrized' and add a String Parameter "branch" with some default value. Select 'Pipeline script from SCM' in definition. Fill in repository URL, Branch Specifier as ${branch} and Script Path (Jenkinsfile) At URL, create a git repository with Jenkinsfile. Put 'node('master') { checkout scm }' in the Jenkinsfile. Run the job. It will fail with:  hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/${branch}:refs/remotes/origin/${branch} --prune" returned status code 128: stdout: stderr: fatal: Couldn't find remote ref refs/heads/${branch} at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1833) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1552) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:66) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:343) at jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:306) at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:196) at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:172) at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:99) at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:59) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:232) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:405)
            Hide
            atikhono Anna Tikhonova added a comment -

            Also, I'm facing the issue when configuring git reference repository in the Pipeline SCM section.

            • All the same configuration as in my comment above, but replace ${branch} with an existing branch, e.g master
            • In 'Additional Behaviours' select 'Advanced Clone Behaviours'
            • Set 'Path of the reference repo to use during clone' to some value depending on an environment variable, e.g. ${HOME}/jenkins/reference/big-git-repo.git. The job will report: 
              Cloning repository ...URL...
               > git init ... # timeout=10
              ERROR: Reference path does not exist: ${HOME}/jenkins/reference/big-git-repo.git

            Would be very handy to have this feature. In my case, I want to use the 'checkout scm' shortcut in my pipeline script that runs several builds in parallel on different platforms. So I want to specify where a reference repository exists on a specific node. For instance, on a Windows node I want to put it in D:\jenkins\reference, on a Mac – to /Users/jenkins/reference and, finally, on linux – /home/jenkins/reference. I could set an environment variable ${REFERENCE} to the above values in node configuration and than use it in 'Path of the reference repo to use during clone' as ${REFERENCE}.

            Show
            atikhono Anna Tikhonova added a comment - Also, I'm facing the issue when configuring git reference repository in the Pipeline SCM section. All the same configuration as in my comment above, but replace ${branch} with an existing branch, e.g master In 'Additional Behaviours' select 'Advanced Clone Behaviours' Set 'Path of the reference repo to use during clone' to some value depending on an environment variable, e.g. ${HOME}/jenkins/reference/big-git-repo.git. The job will report:  Cloning repository ...URL... > git init ... # timeout=10 ERROR: Reference path does not exist: ${HOME}/jenkins/reference/big-git-repo.git Would be very handy to have this feature. In my case, I want to use the 'checkout scm' shortcut in my pipeline script that runs several builds in parallel on different platforms. So I want to specify where a reference repository exists on a specific node. For instance, on a Windows node I want to put it in D:\jenkins\reference, on a Mac – to /Users/jenkins/reference and, finally, on linux – /home/jenkins/reference. I could set an environment variable ${REFERENCE} to the above values in node configuration and than use it in 'Path of the reference repo to use during clone' as ${REFERENCE}.
            Hide
            jglick Jesse Glick added a comment -

            Anna Tikhonova you must disable lightweight checkout.

            Show
            jglick Jesse Glick added a comment - Anna Tikhonova you must disable lightweight checkout.
            Hide
            jglick Jesse Glick added a comment -
            Show
            jglick Jesse Glick added a comment - Johnny Willemsen see  JENKINS-42836 .
            Hide
            atikhono Anna Tikhonova added a comment -

            Jesse Glick Thanks, with lightweight checkout off ${branch} is resolved. Still, git reference repository path is not. Should I file another issue for that?

            Show
            atikhono Anna Tikhonova added a comment - Jesse Glick Thanks, with lightweight checkout off ${branch} is resolved. Still, git reference repository path is not. Should I file another issue for that?
            Hide
            jglick Jesse Glick added a comment -

            Anna Tikhonova that sounds like an RFE for git-plugin not specific to Pipeline.

            Show
            jglick Jesse Glick added a comment - Anna Tikhonova that sounds like an RFE for git-plugin not specific to Pipeline.
            Hide
            asfaltboy Pavel Savshenko added a comment -

            Jesse Glick I came across this issue as well, and unchecking "lightweight checkout" also helped. However I cannot figure out why exactly, can you give a bit more context into why the env/parameter interpolation is disabled when LC is on ?

            Show
            asfaltboy Pavel Savshenko added a comment - Jesse Glick I came across this issue as well, and unchecking "lightweight checkout" also helped. However I cannot figure out why exactly, can you give a bit more context into why the env/parameter interpolation is disabled when LC is on ?
            Hide
            jglick Jesse Glick added a comment -

            Lightweight checkout is going through a completely unrelated code path, and currently SCMFileSystem implementations do not honor variables (or even have an API allowing them to honor variables IIRC). Would have to be a separate feature.

            Show
            jglick Jesse Glick added a comment - Lightweight checkout is going through a completely unrelated code path, and currently SCMFileSystem implementations do not honor variables (or even have an API allowing them to honor variables IIRC). Would have to be a separate feature.
            Hide
            mmallard Matthew Mallard added a comment -

            Has this "separate feature" been added yet to the "lightweight checkout" functionality? In my case, I have a pipeline job that is parameterized. One of the parameters is the branch name which I intend to use later on in the job when selecting the definition "Pipeline script from SCM". When selecting that you have the option to select your Git repository and Branches to build, but it doesn't recognize the parameter when "Lightweight checkout" is selected.

            Show
            mmallard Matthew Mallard added a comment - Has this "separate feature" been added yet to the "lightweight checkout" functionality? In my case, I have a pipeline job that is parameterized. One of the parameters is the branch name which I intend to use later on in the job when selecting the definition "Pipeline script from SCM". When selecting that you have the option to select your Git repository and Branches to build, but it doesn't recognize the parameter when "Lightweight checkout" is selected.

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                nsnewland Nicholas Newland
              • Votes:
                50 Vote for this issue
                Watchers:
                65 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: