Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-57417

configFileProvider / withCredentials not working in declarative pipeline

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Labels:
      None
    • Environment:
      Jenkins 2.174
      Config File Provider Plugin 3.6
      Credentials Bind Plugin 1.18
    • Similar Issues:

      Description

      From what I read which googling the issue, I am under the impression that:

      1) The credentials bind plugin does not assign environment variables, but rather groovy variables; and even if it did, environment variables do not propagate "up" to the parent context.

      2) The "glue" that is available using scripted pipelines does not work in declarative pipelines; specifically the ability to define local variables and assign environment variables within a stage, and access them via the underlying token macro plugin that backs the configFileProvider.

      • Use of groovy variables, ${PIP_USERNAME}, resulted in an "Unrecoginzed macro" error.
      • Use of ${env.PIP_USERNAME} did not substitute the variable at all.
      • Use of ${ENV, var = "PIP_USERNAME"} substitutes the variable with null string.

       

      Custom Config File - 05f48227-0980-4313-ab24-f007d78090cf
      [global]
      index = https://${ENV, var="PIP_USERNAME"}:${ENV, var="PIP_PASSWORD"}@nexus/repository/pypi-all/pypi
      index-url = https://${ENV, var="PIP_USERNAME"}:${ENV, var="PIP_PASSWORD"}@nexus/repository/pypi-all/simple
      cert = /etc/pki/ca-trust/source/anchors/ca.crt
      
      [list]
      format=columns
      

       

      Pipeline
      pipeline {
          agent any
          
          environment {
              PIP_FILE = '05f48227-0980-4313-ab24-f007d78090cf'
          }
          stages {
              stage('Build') {
                  steps {
                      withCredentials([usernamePassword(credentialsId: '0353637f-ef0b-46e5-b95a-6322b1e073d7', passwordVariable: 'PIP_PASSWORD', usernameVariable: 'PIP_USERNAME')]) {
                          configFileProvider([configFile(fileId: "${env.PIP_FILE}", replaceTokens: true, variable: 'PIP_TARGET')]) {
                              sh "cat ${PIP_TARGET}"
                          }
                      }
                  }
              }
          }
      }
      
      Output
      ...
      Pipeline] {
      [Pipeline] withEnv
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Build)
      [Pipeline] withCredentials
      Masking only exact matches of $PIP_USERNAME or $PIP_PASSWORD
      [Pipeline] {
      [Pipeline] wrap
      provisioning config files...
      copy managed file [Python pip.conf for Debugging] to file:/home/jenkins/workspace/OPIN/jenkins-issue@tmp/config8203247759059609056tmp
      [Pipeline] {
      [Pipeline] sh
      + cat /home/jenkins/workspace/OPIN/jenkins-issue@tmp/config8203247759059609056tmp
      [global]
      index = https://:@nexus/repository/pypi-all/pypi
      index-url = https://:@nexus/repository/pypi-all/simple
      cert = /etc/pki/ca-trust/source/anchors/ca.crt
      
      [list]
      [Pipeline] }
      Deleting 1 temporary files
      ...
      

      Is this a bug, or am I missing something?

        Attachments

          Issue Links

            Activity

            Hide
            imod Dominik Bartholdi added a comment - - edited

            Why do you handle the ids different for withCredentials and configFileProvider? fileId is not substituted by the configFileProvider, therefore you have to pass the id as it is. But after all you can use groovy if you like, this should work:

            var pipFile = '05f48227-0980-4313-ab24-f007d78090cf'
            pipeline {
                agent any
                
                stages {
                    stage('Build') {
                        steps {
                            withCredentials([usernamePassword(credentialsId: '0353637f-ef0b-46e5-b95a-6322b1e073d7', passwordVariable: 'PIP_PASSWORD', usernameVariable: 'PIP_USERNAME')]) {
                                configFileProvider([configFile(fileId: pipFile, replaceTokens: true, variable: 'PIP_TARGET')]) {
                                    sh "cat ${PIP_TARGET}"
                                }
                            }
                        }
                    }
                }
            }
            Show
            imod Dominik Bartholdi added a comment - - edited Why do you handle the ids different for withCredentials and configFileProvider ? fileId is not substituted by the configFileProvider, therefore you have to pass the id as it is. But after all you can use groovy if you like, this should work: var pipFile = '05f48227-0980-4313-ab24-f007d78090cf' pipeline { agent any stages { stage( 'Build' ) { steps { withCredentials([usernamePassword(credentialsId: '0353637f-ef0b-46e5-b95a-6322b1e073d7' , passwordVariable: 'PIP_PASSWORD' , usernameVariable: 'PIP_USERNAME' )]) { configFileProvider([configFile(fileId: pipFile, replaceTokens: true , variable: 'PIP_TARGET' )]) { sh "cat ${PIP_TARGET}" } } } } } }
            Hide
            crashvb Richard Davis added a comment -

            Thank you for the reply. This code was isolated from a larger pipeline to reproduce the specific issue being reported.

            I ran the same job again with the fileId inline (no local variable), and got the same outcome:

            configFileProvider([configFile(fileId: '05f48227-0980-4313-ab24-f007d78090cf', replaceTokens: true, variable: 'PIP_TARGET')]) { ... }
            

            I don't believe that the issue is related to resolution of fileId, as it the plugin is able to replicate the file (with parameters omitted):

            ...
            copy managed file [Python pip.conf for Debugging] to file:/home/jenkins/workspace/OPIN/jenkins-issue@tmp/config8203247759059609056tmp
            ...
            

            Since the last post, I have also add a

            sh "echo ${PIP_USERNAME} ${PIP_PASSWORD}"

            within the withCredentials block, and saw that they were successfully masked:

            Confirmation of ENV assignment
            ...
            [Pipeline] withCredentials
            Masking only exact matches of $PIP_USERNAME or $PIP_PASSWORD
            [Pipeline] {
            [Pipeline] sh
            + echo **** ****
            **** ****
            [Pipeline] wrap
            ...
            

            I remember seeing an old PR against the config file provider, where it wasn't initially replacing macros within pipelines, only scripts; but i'm not familiar enough with the code base to remember in which file it was located. From looking at the source of the token macro, it appears to accept [environment as a parameter](https://github.com/jenkinsci/token-macro-plugin/blob/master/src/main/java/org/jenkinsci/plugins/tokenmacro/impl/EnvironmentVariableMacro.java#L45). Could this be an issue with how it's being invoked?

            Show
            crashvb Richard Davis added a comment - Thank you for the reply. This code was isolated from a larger pipeline to reproduce the specific issue being reported. I ran the same job again with the fileId inline (no local variable), and got the same outcome: configFileProvider([configFile(fileId: '05f48227-0980-4313-ab24-f007d78090cf' , replaceTokens: true , variable: 'PIP_TARGET' )]) { ... } I don't believe that the issue is related to resolution of fileId, as it the plugin is able to replicate the file (with parameters omitted): ... copy managed file [Python pip.conf for Debugging] to file:/home/jenkins/workspace/OPIN/jenkins-issue@tmp/config8203247759059609056tmp ... Since the last post, I have also add a sh "echo ${PIP_USERNAME} ${PIP_PASSWORD}" within the withCredentials block, and saw that they were successfully masked: Confirmation of ENV assignment ... [Pipeline] withCredentials Masking only exact matches of $PIP_USERNAME or $PIP_PASSWORD [Pipeline] { [Pipeline] sh + echo **** **** **** **** [Pipeline] wrap ... I remember seeing an old PR against the config file provider, where it wasn't initially replacing macros within pipelines, only scripts; but i'm not familiar enough with the code base to remember in which file it was located. From looking at the source of the token macro, it appears to accept [environment as a parameter] ( https://github.com/jenkinsci/token-macro-plugin/blob/master/src/main/java/org/jenkinsci/plugins/tokenmacro/impl/EnvironmentVariableMacro.java#L45 ). Could this be an issue with how it's being invoked?
            Hide
            crashvb Richard Davis added a comment - - edited

            I've dug a little deeper into the source repository. it looks like the Util.replaceMacro method is being invoked correctly with the provided environment; however, it appears that the environment is not being propagated from higher context, and is instead being explicitly provided as null.

            I don't know enough about the way jenkins pipelines are implemented to assess if this omission is intentional, due to an alternative data channel and / or design limitation, or if it is a bug ...

            Is use of the configFileProvider within declarative pipelines supported, or should this be re-characterized as an feature enhancement?

            (I've also removed the 'credentials-binding-plugin' and 'token-macro-plugin' components, as they have been verified in parallel and don't appear to be the root cause.)

            Show
            crashvb Richard Davis added a comment - - edited I've dug a little deeper into the source repository. it looks like the Util.replaceMacro method is being invoked correctly with the provided environment ; however, it appears that the environment is not being propagated from higher context, and is instead being explicitly provided as null . I don't know enough about the way jenkins pipelines are implemented to assess if this omission is intentional, due to an alternative data channel and / or design limitation, or if it is a bug ... Is use of the configFileProvider within declarative pipelines supported, or should this be re-characterized as an feature enhancement? (I've also removed the 'credentials-binding-plugin' and 'token-macro-plugin' components, as they have been verified in parallel and don't appear to be the root cause.)
            Hide
            crashvb Richard Davis added a comment -

            Continuation ...

            Would it be possible to amend the ManagedFileUtil.provisionConfigFiles method to accept env, and populate it in ConfigFileBuildWrapper.setUp using context.getEnv()?

            Show
            crashvb Richard Davis added a comment - Continuation ... Would it be possible to amend the ManagedFileUtil.provisionConfigFiles method to accept  env , and populate it in ConfigFileBuildWrapper.setUp using context.getEnv() ?
            Hide
            crashvb Richard Davis added a comment -

            Submitted PR #65

            Show
            crashvb Richard Davis added a comment - Submitted PR #65
            Hide
            imod Dominik Bartholdi added a comment -

            released with 3.6.1

            Show
            imod Dominik Bartholdi added a comment - released with 3.6.1

              People

              • Assignee:
                domi Dominik Bartholdi
                Reporter:
                crashvb Richard Davis
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: