Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-16233

EnvInject plugin using a cached value for ${WORKSPACE}

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Starting yesterday, the EnvInject plugin seems to suddenly start using a cached value for ${WORKSPACE}. We have the following value in "Properties Content" under "Inject environment variables to the build process":

      PATH=${WORKSPACE}/.env/bin:$PATH

      When working properly, in the console output this appears as:

      [EnvInject] - Injecting as environment variables the properties content
      PATH=${WORKSPACE}/.env/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/bin

      Over the past couple of days this has suddenly started appearing as:

      [EnvInject] - Injecting as environment variables the properties content
      PATH=/Users/Shared/Jenkins/Home/jobs/bidpom.stage/workspace/.env/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/bin

      Which is using a workspace path from another job. This causes this job, and all other jobs, to fail. If I restart Jenkins the problem goes away, for awhile, but at some point that exact same path starts getting used by the EnvInject plugin again. This cycle has happened 4 times over the past couple of days, where jobs will run fine for awhile and then suddenly start failing because the path injected changes from:

      ${WORKSPACE}/.env/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/bin

      to:

      /Users/Shared/Jenkins/Home/jobs/bidpom.stage/workspace/.env/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/bin

      When this first happened we downgraded the plugin from 1.78 to 1.77 but this hasn't addressed the problem.

      Any help anyone can provide would be greatly appreciated.

        Attachments

          Activity

          bobsilverberg Bob Silverberg created issue -
          Hide
          bobsilverberg Bob Silverberg added a comment -

          This problem went away for about a week after rebooting a few times but just cropped up again. Any help that anyone can provide would be greatly appreciated.

          Show
          bobsilverberg Bob Silverberg added a comment - This problem went away for about a week after rebooting a few times but just cropped up again. Any help that anyone can provide would be greatly appreciated.
          Hide
          davehunt Dave Hunt added a comment -

          I suspect this is related to JENKINS-15658

          Show
          davehunt Dave Hunt added a comment - I suspect this is related to JENKINS-15658
          Hide
          gbois Gregory Boissinot added a comment -

          JENKINS-15658 was introduced in version 1.73. Therefore, if it is the cause, you have to downgrade to 1.72.
          For this issue, it seems not to be a workspace from another job but a shared workspace.
          Does this value is specified in another place in the job or in the global configuration?

          Show
          gbois Gregory Boissinot added a comment - JENKINS-15658 was introduced in version 1.73. Therefore, if it is the cause, you have to downgrade to 1.72. For this issue, it seems not to be a workspace from another job but a shared workspace. Does this value is specified in another place in the job or in the global configuration?
          gbois Gregory Boissinot made changes -
          Field Original Value New Value
          Status Open [ 1 ] In Progress [ 3 ]
          Hide
          gbois Gregory Boissinot added a comment -

          Hi,
          could you check your job configuration?
          Thanks

          Show
          gbois Gregory Boissinot added a comment - Hi, could you check your job configuration? Thanks
          Hide
          bobsilverberg Bob Silverberg added a comment -

          The value is not specified anywhere else in the job nor in the global configuration. It seems to be caching the workspace value that was used for a previous job.

          Show
          bobsilverberg Bob Silverberg added a comment - The value is not specified anywhere else in the job nor in the global configuration. It seems to be caching the workspace value that was used for a previous job.
          Hide
          davehunt Dave Hunt added a comment -

          This is happening right now. We need to restart Jenkins to fix this, but it will no doubt happen again. I've done a little investigation and found that echoing ${WORKSPACE} in a build shell shows the correct value, but EnvInject has injected the wrong value:

          [CHECKED] Inject environment variables to the build process
          Properties Content:
          TEST=${WORKSPACE}
          PATH=${WORKSPACE}/.env/bin:$PATH

          Execute shell:
          printenv
          echo ${WORKSPACE}
          echo ${PATH}

          Console output:
          ...
          [EnvInject] - Executing scripts and injecting environment variables after the SCM step.
          [EnvInject] - Injecting as environment variables the properties content
          TEST=/Users/Shared/Jenkins/Home/jobs/bidpom.stage.saucelabs/workspace
          PATH=/Users/Shared/Jenkins/Home/jobs/bidpom.stage.saucelabs/workspace/.env/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
          [EnvInject] - Variables injected successfully.
          ...
          + printenv
          ...
          TEST=/Users/Shared/Jenkins/Home/jobs/bidpom.stage.saucelabs/workspace
          WORKSPACE=/Users/Shared/Jenkins/Home/jobs/bouncer.prod/workspace
          PATH=/Users/Shared/Jenkins/Home/jobs/bidpom.stage.saucelabs/workspace/.env/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
          ...
          + echo /Users/Shared/Jenkins/Home/jobs/bouncer.prod/workspace
          /Users/Shared/Jenkins/Home/jobs/bouncer.prod/workspace
          + echo /Users/Shared/Jenkins/Home/jobs/bidpom.stage.saucelabs/workspace/.env/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
          /Users/Shared/Jenkins/Home/jobs/bidpom.stage.saucelabs/workspace/.env/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin

          So you can see that WORKSPACE is accurate according to Jenkins environment variables, but EnvInject is expanding it to an incorrect value.

          Please let me know if there's any additional information that would be useful for the next time this fails.

          Show
          davehunt Dave Hunt added a comment - This is happening right now. We need to restart Jenkins to fix this, but it will no doubt happen again. I've done a little investigation and found that echoing ${WORKSPACE} in a build shell shows the correct value, but EnvInject has injected the wrong value: [CHECKED] Inject environment variables to the build process Properties Content: TEST=${WORKSPACE} PATH=${WORKSPACE}/.env/bin:$PATH Execute shell: printenv echo ${WORKSPACE} echo ${PATH} Console output: ... [EnvInject] - Executing scripts and injecting environment variables after the SCM step. [EnvInject] - Injecting as environment variables the properties content TEST=/Users/Shared/Jenkins/Home/jobs/bidpom.stage.saucelabs/workspace PATH=/Users/Shared/Jenkins/Home/jobs/bidpom.stage.saucelabs/workspace/.env/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin [EnvInject] - Variables injected successfully. ... + printenv ... TEST=/Users/Shared/Jenkins/Home/jobs/bidpom.stage.saucelabs/workspace WORKSPACE=/Users/Shared/Jenkins/Home/jobs/bouncer.prod/workspace PATH=/Users/Shared/Jenkins/Home/jobs/bidpom.stage.saucelabs/workspace/.env/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin ... + echo /Users/Shared/Jenkins/Home/jobs/bouncer.prod/workspace /Users/Shared/Jenkins/Home/jobs/bouncer.prod/workspace + echo /Users/Shared/Jenkins/Home/jobs/bidpom.stage.saucelabs/workspace/.env/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin /Users/Shared/Jenkins/Home/jobs/bidpom.stage.saucelabs/workspace/.env/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin So you can see that WORKSPACE is accurate according to Jenkins environment variables, but EnvInject is expanding it to an incorrect value. Please let me know if there's any additional information that would be useful for the next time this fails.
          Hide
          gbois Gregory Boissinot added a comment -

          Does '/Users/Shared/Jenkins/Home/jobs/bouncer.prod/workspace' match a specific slave where the build run?

          Show
          gbois Gregory Boissinot added a comment - Does '/Users/Shared/Jenkins/Home/jobs/bouncer.prod/workspace' match a specific slave where the build run?
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Gregory Boissinot
          Path:
          src/main/java/org/jenkinsci/plugins/envinject/EnvInjectBuildWrapper.java
          src/main/java/org/jenkinsci/plugins/envinject/EnvInjectBuilder.java
          src/main/java/org/jenkinsci/plugins/envinject/EnvInjectListener.java
          src/test/java/org/jenkinsci/plugins/envinject/GlobalPropertiesTest.java
          http://jenkins-ci.org/commit/envinject-plugin/23a7086f231e0647bfc24daceacaa1f078f5aa7f
          Log:
          Fix JENKINS-16233

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Gregory Boissinot Path: src/main/java/org/jenkinsci/plugins/envinject/EnvInjectBuildWrapper.java src/main/java/org/jenkinsci/plugins/envinject/EnvInjectBuilder.java src/main/java/org/jenkinsci/plugins/envinject/EnvInjectListener.java src/test/java/org/jenkinsci/plugins/envinject/GlobalPropertiesTest.java http://jenkins-ci.org/commit/envinject-plugin/23a7086f231e0647bfc24daceacaa1f078f5aa7f Log: Fix JENKINS-16233
          Hide
          gbois Gregory Boissinot added a comment -

          EnvInject 1.80 should fix this issue.
          Please let me know if it works in your environment.
          Thanks

          Show
          gbois Gregory Boissinot added a comment - EnvInject 1.80 should fix this issue. Please let me know if it works in your environment. Thanks
          gbois Gregory Boissinot made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          davehunt Dave Hunt added a comment -

          Were you able to replicate the issue? We have upgraded to 1.80 and still see the issue. I've taken a look at the changes and not sure that 'we set a global workspace at job level' is accurate for what we are doing. We are setting an environment variable to the value including ${WORKSPACE} and instead of getting the job workspace it is the value of a previous job's workspace...

          Show
          davehunt Dave Hunt added a comment - Were you able to replicate the issue? We have upgraded to 1.80 and still see the issue. I've taken a look at the changes and not sure that 'we set a global workspace at job level' is accurate for what we are doing. We are setting an environment variable to the value including ${WORKSPACE} and instead of getting the job workspace it is the value of a previous job's workspace...
          davehunt Dave Hunt made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          gbois Gregory Boissinot added a comment -

          I'm sorry I though the previous fix could fix the issue.
          I'm not reproducing the issue. Which type is your job (freestyle, maven, matrix, ...)?

          Please feel free to isolate the problem and attach a zip with your Jenkins env?
          Without isolating the problem, I don't know how to fix it.

          Show
          gbois Gregory Boissinot added a comment - I'm sorry I though the previous fix could fix the issue. I'm not reproducing the issue. Which type is your job (freestyle, maven, matrix, ...)? Please feel free to isolate the problem and attach a zip with your Jenkins env? Without isolating the problem, I don't know how to fix it.
          Hide
          magjac Magnus Jacobsson added a comment -
          Show
          magjac Magnus Jacobsson added a comment - I see the same problem, see https://issues.jenkins-ci.org/browse/JENKINS-16448 .
          Hide
          gbois Gregory Boissinot added a comment -

          Does it for always for a matrix job?

          Show
          gbois Gregory Boissinot added a comment - Does it for always for a matrix job?
          Hide
          magjac Magnus Jacobsson added a comment -

          No, but when it does, it's always the same wrong value, i.e. the value of a since long deleted project. In my case the value is always "main-multiconf" which was the value of and old matrix job that was deleted several months ago. It may actually have been the first matrix job we ever created.

          Show
          magjac Magnus Jacobsson added a comment - No, but when it does, it's always the same wrong value, i.e. the value of a since long deleted project. In my case the value is always "main-multiconf" which was the value of and old matrix job that was deleted several months ago. It may actually have been the first matrix job we ever created.
          Hide
          magjac Magnus Jacobsson added a comment -

          I meant that the value is always "/var/lib/jenkins/jobs/main-multiconf/workspace".

          Show
          magjac Magnus Jacobsson added a comment - I meant that the value is always "/var/lib/jenkins/jobs/main-multiconf/workspace".
          Hide
          davehunt Dave Hunt added a comment -

          We have rolled back to version 1.61 and so far have not seen this issue since.

          Show
          davehunt Dave Hunt added a comment - We have rolled back to version 1.61 and so far have not seen this issue since.
          Hide
          gbois Gregory Boissinot added a comment -

          Is it possible for you to reproduce the problem within the last version in an isolated environment?
          Thanks in advance.

          Show
          gbois Gregory Boissinot added a comment - Is it possible for you to reproduce the problem within the last version in an isolated environment? Thanks in advance.
          Hide
          magjac Magnus Jacobsson added a comment -

          Job config that shows the problem. Log from run below. Note that the workspace is /var/lib/jenkins/workspace/magjac-workspace-test, but the WORKSPACE variable is /var/lib/jenkins/workspace/saab-dev-branch_test_repo, which is another jobs workspace.

          Started by user magjac
          [EnvInject] - Loading node environment variables.
          Building in workspace /var/lib/jenkins/workspace/magjac-workspace-test
          [magjac-workspace-test] $ /bin/sh -xe /tmp/hudson102135884180944618.sh
          + env
          LC_PAPER=sv_SE.UTF-8
          BUILD_URL=http://worabu.netinsight.se:8080/job/magjac-workspace-test/1/
          HUDSON_SERVER_COOKIE=89a7b7e616d9e722
          SHELL=/bin/bash
          TERM=unknown
          XDG_SESSION_COOKIE=da3dd519a721d0e1b7b22e424d2ae4df-1360064288.175876-277727223
          BUILD_TAG=jenkins-magjac-workspace-test-1
          WORKSPACE=/var/lib/jenkins/workspace/saab-dev-branch_test_repo
          JOB_URL=http://worabu.netinsight.se:8080/job/magjac-workspace-test/
          USER=jenkins
          LD_LIBRARY_PATH=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64
          JENKINS_HOME=/var/lib/jenkins
          PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
          MAIL=/var/mail/jenkins
          BUILD_CAUSE_MANUALTRIGGER=true
          _=/usr/bin/env
          LC_COLLATE=sv_SE.UTF-8
          PWD=/var/lib/jenkins/workspace/magjac-workspace-test
          HUDSON_URL=http://worabu.netinsight.se:8080/
          LANG=C
          JOB_NAME=magjac-workspace-test
          JENKINS_URL=http://worabu.netinsight.se:8080/
          BUILD_CAUSE=MANUALTRIGGER
          BUILD_ID=2013-02-19_08-30-44
          SPEECHD_PORT=6682
          HOME=/var/lib/jenkins
          SHLVL=2
          JENKINS_SERVER_COOKIE=89a7b7e616d9e722
          EXECUTOR_NUMBER=1
          NODE_LABELS=master
          LOGNAME=jenkins
          HUDSON_HOME=/var/lib/jenkins
          LC_CTYPE=sv_SE.UTF-8
          NODE_NAME=master
          BUILD_NUMBER=1
          HUDSON_COOKIE=d112eba8-2c55-4ff8-82d1-254f86f0b5e4
          DISPLAY=:0.0
          LC_TIME=en_DK.UTF-8
          XAUTHORITY=/tmp/libgksu-Mbf5UN/.Xauthority
          Collecting metadata...
          Metadata collection done.
          Finished: SUCCESS

          Show
          magjac Magnus Jacobsson added a comment - Job config that shows the problem. Log from run below. Note that the workspace is /var/lib/jenkins/workspace/magjac-workspace-test, but the WORKSPACE variable is /var/lib/jenkins/workspace/saab-dev-branch_test_repo, which is another jobs workspace. Started by user magjac [EnvInject] - Loading node environment variables. Building in workspace /var/lib/jenkins/workspace/magjac-workspace-test [magjac-workspace-test] $ /bin/sh -xe /tmp/hudson102135884180944618.sh + env LC_PAPER=sv_SE.UTF-8 BUILD_URL= http://worabu.netinsight.se:8080/job/magjac-workspace-test/1/ HUDSON_SERVER_COOKIE=89a7b7e616d9e722 SHELL=/bin/bash TERM=unknown XDG_SESSION_COOKIE=da3dd519a721d0e1b7b22e424d2ae4df-1360064288.175876-277727223 BUILD_TAG=jenkins-magjac-workspace-test-1 WORKSPACE=/var/lib/jenkins/workspace/saab-dev-branch_test_repo JOB_URL= http://worabu.netinsight.se:8080/job/magjac-workspace-test/ USER=jenkins LD_LIBRARY_PATH=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64 JENKINS_HOME=/var/lib/jenkins PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games MAIL=/var/mail/jenkins BUILD_CAUSE_MANUALTRIGGER=true _=/usr/bin/env LC_COLLATE=sv_SE.UTF-8 PWD=/var/lib/jenkins/workspace/magjac-workspace-test HUDSON_URL= http://worabu.netinsight.se:8080/ LANG=C JOB_NAME=magjac-workspace-test JENKINS_URL= http://worabu.netinsight.se:8080/ BUILD_CAUSE=MANUALTRIGGER BUILD_ID=2013-02-19_08-30-44 SPEECHD_PORT=6682 HOME=/var/lib/jenkins SHLVL=2 JENKINS_SERVER_COOKIE=89a7b7e616d9e722 EXECUTOR_NUMBER=1 NODE_LABELS=master LOGNAME=jenkins HUDSON_HOME=/var/lib/jenkins LC_CTYPE=sv_SE.UTF-8 NODE_NAME=master BUILD_NUMBER=1 HUDSON_COOKIE=d112eba8-2c55-4ff8-82d1-254f86f0b5e4 DISPLAY=:0.0 LC_TIME=en_DK.UTF-8 XAUTHORITY=/tmp/libgksu-Mbf5UN/.Xauthority Collecting metadata... Metadata collection done. Finished: SUCCESS
          magjac Magnus Jacobsson made changes -
          Attachment magjac-workspace-test_config.xml [ 23201 ]
          Hide
          byao Bob Yao added a comment -

          I'm getting this issue as well. I've restarted several times, but problem didn't go away. Let me know how I can help give you more info.

          [EnvInject] - Loading node environment variables.
          Building on master in workspace /home/jenkins/jobs/aura-snapshot-build/workspace
          Checkout:workspace / /home/jenkins/jobs/aura-snapshot-build/workspace - hudson.remoting.LocalChannel@42e934e
          Using strategy: Default
          Last Built Revision: Revision 1092e00a1953421e0e65447d3aa0089f933fdc92 (origin/master-pending-update-copyright-headers)
          Wiping out workspace first.
          Cloning the remote Git repository
          Cloning repository [OMIT]
          git --version
          git version 1.7.0.4
          Fetching upstream changes from origin
          Cleaning workspace
          Resetting working tree
          Seen branch in repository origin/master-pending-update-copyright-headers
          Commencing build of Revision 1092e00a1953421e0e65447d3aa0089f933fdc92 (origin/master-pending-update-copyright-headers)
          Checking out Revision 1092e00a1953421e0e65447d3aa0089f933fdc92 (origin/master-pending-update-copyright-headers)
          Cleaning workspace
          Resetting working tree
          No emails were triggered.
          [workspace] $ /bin/bash -xe /tmp/hudson3897671398590473161.sh
          + mkdir -p /home/jenkins/jobs/~masterArtifactMonitor/workspace/tmp

          mkdir -p ${WORKSPACE}/tmp
          WORKSPACE=/home/jenkins/jobs/~masterArtifactMonitor/workspace

          actual workspace should be /home/jenkins/jobs/aura-snapshot-build/workspace

          Show
          byao Bob Yao added a comment - I'm getting this issue as well. I've restarted several times, but problem didn't go away. Let me know how I can help give you more info. [EnvInject] - Loading node environment variables. Building on master in workspace /home/jenkins/jobs/aura-snapshot-build/workspace Checkout:workspace / /home/jenkins/jobs/aura-snapshot-build/workspace - hudson.remoting.LocalChannel@42e934e Using strategy: Default Last Built Revision: Revision 1092e00a1953421e0e65447d3aa0089f933fdc92 (origin/master-pending-update-copyright-headers) Wiping out workspace first. Cloning the remote Git repository Cloning repository [OMIT] git --version git version 1.7.0.4 Fetching upstream changes from origin Cleaning workspace Resetting working tree Seen branch in repository origin/master-pending-update-copyright-headers Commencing build of Revision 1092e00a1953421e0e65447d3aa0089f933fdc92 (origin/master-pending-update-copyright-headers) Checking out Revision 1092e00a1953421e0e65447d3aa0089f933fdc92 (origin/master-pending-update-copyright-headers) Cleaning workspace Resetting working tree No emails were triggered. [workspace] $ /bin/bash -xe /tmp/hudson3897671398590473161.sh + mkdir -p /home/jenkins/jobs/~masterArtifactMonitor/workspace/tmp mkdir -p ${WORKSPACE}/tmp WORKSPACE=/home/jenkins/jobs/~masterArtifactMonitor/workspace actual workspace should be /home/jenkins/jobs/aura-snapshot-build/workspace
          Hide
          gbois Gregory Boissinot added a comment -

          For now, the challenge is to reproduce this issue in an isolated environment.
          For example, an environment with only a Jenkins instance and the envinject plugin.

          Show
          gbois Gregory Boissinot added a comment - For now, the challenge is to reproduce this issue in an isolated environment. For example, an environment with only a Jenkins instance and the envinject plugin.
          Hide
          magjac Magnus Jacobsson added a comment -

          Is this the same problem as JENKINS-13348?

          Show
          magjac Magnus Jacobsson added a comment - Is this the same problem as JENKINS-13348 ?
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 147046 ] JNJira + In-Review [ 186102 ]

            People

            • Assignee:
              gbois Gregory Boissinot
              Reporter:
              bobsilverberg Bob Silverberg
            • Votes:
              6 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated: