Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26133

Shell script taking/returning output/status

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Currently sh has no meaningful return value, and throws an exception if the exit status is not zero. Would be nice to have an option to have it return the exit code (zero or not) as an integer value:

      def r = sh script: 'someCommand', returnStatus: true
      

      Current workaround:

      sh 'someCommand; echo $? > status'
      def r = readFile('status').trim()
      

      Or to have it return its standard output (akin to shell backticks):

      def lines = sh(script: 'dumpStuff.sh', returnStdout: true).split("\r?\n")
      

      Workaround:

      sh 'dumpStuff.sh > result'
      def lines = readFile('result').split("\r?\n")
      

      Or to have it take something on standard input:

      sh script: 'loadStuff.sh', stdin: someText
      

      Workaround:

      writeFile file: 'input', text: someText
      sh 'loadStuff.sh < input'
      

      Probably requires some API changes in durable-task.

        Attachments

          Issue Links

            Activity

            jglick Jesse Glick created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Link This issue is duplicated by JENKINS-26812 [ JENKINS-26812 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-28302 [ JENKINS-28302 ]
            Hide
            jglick Jesse Glick added a comment -

            Capture of exit status as a return value can be handled directly in DurableTaskStep.Execution, by simply returning Integer rather than returning void or throwing AbortException.

            Capture of standard output is a little trickier because we would like to continue to direct standard error to the build log, and durable-task controllers currently mix them in the same stream. The API likely needs to be changed so that when launching the process you can request separate streams. (jenkins-out.txt vs. jenkins-err.txt I suppose, and two log position fields.) DurableTaskStep.Execution could then maintain a ByteArrayOutputStream-like buffer (but Serializable) collecting the output to be returned. Alternately, the controller could just return the complete standard output when the process exits, alongside the status code, which would make for a simpler API change, under the assumption that future clients do not want to stream output.

            Likewise, sending standard input would require an API addition in durable-task. Streaming the input would be very awkward for a durable task (providing an InputStream is not possible) so you would probably have to provide the byte[] content up front.

            Both API changes could potentially be avoided by just making DurableTaskStep.Execution modify the script to use redirects as shown in the workarounds here, using a random temporary file. There are some tricky parts here, too, though: the file ought to be outside the workspace to avoid clashes with scripts which expect to “own” the whole directory tree (cf. discussion in JENKINS-27152); and to handle multiline scripts, or even scripts with shebangs (#!/usr/bin/python), you really need to save the user-provided script into another temporary file, then do a little dance to make it into a complete executable that could be called from the redirecting wrapper script (and here cf. JENKINS-29877).

            Useful also for bat. According to answers here, Bourne shell syntax works for batch scripts too.

            Show
            jglick Jesse Glick added a comment - Capture of exit status as a return value can be handled directly in DurableTaskStep.Execution , by simply returning Integer rather than returning void or throwing AbortException . Capture of standard output is a little trickier because we would like to continue to direct standard error to the build log, and durable-task controllers currently mix them in the same stream. The API likely needs to be changed so that when launching the process you can request separate streams. ( jenkins-out.txt vs. jenkins-err.txt I suppose, and two log position fields.) DurableTaskStep.Execution could then maintain a ByteArrayOutputStream -like buffer (but Serializable ) collecting the output to be returned. Alternately, the controller could just return the complete standard output when the process exits, alongside the status code, which would make for a simpler API change, under the assumption that future clients do not want to stream output. Likewise, sending standard input would require an API addition in durable-task . Streaming the input would be very awkward for a durable task (providing an InputStream is not possible) so you would probably have to provide the byte[] content up front. Both API changes could potentially be avoided by just making DurableTaskStep.Execution modify the script to use redirects as shown in the workarounds here, using a random temporary file. There are some tricky parts here, too, though: the file ought to be outside the workspace to avoid clashes with scripts which expect to “own” the whole directory tree (cf. discussion in JENKINS-27152 ); and to handle multiline scripts, or even scripts with shebangs ( #!/usr/bin/python ), you really need to save the user-provided script into another temporary file, then do a little dance to make it into a complete executable that could be called from the redirecting wrapper script (and here cf. JENKINS-29877 ). Useful also for bat . According to answers here , Bourne shell syntax works for batch scripts too.
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-29877 [ JENKINS-29877 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-27152 [ JENKINS-27152 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-33173 [ JENKINS-33173 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-34331 [ JENKINS-34331 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-34331 [ JENKINS-34331 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-34331 [ JENKINS-34331 ]
            hrmpw Patrick Wolf made changes -
            Labels api api followup
            jglick Jesse Glick made changes -
            Epic Link JENKINS-35394 [ 171187 ]
            Hide
            jglick Jesse Glick added a comment -

            I suspect that passing standard input is a rare enough requirement that it is best skipped; the existing workaround suffices, though I would amend it a bit to avoid putting junk in the workspace:

            def input = "${pwd tmp: true}/input"
            writeFile file: input, text: someText
            sh "loadStuff.sh < '${input}'"
            
            Show
            jglick Jesse Glick added a comment - I suspect that passing standard input is a rare enough requirement that it is best skipped; the existing workaround suffices, though I would amend it a bit to avoid putting junk in the workspace: def input = "${pwd tmp: true }/input" writeFile file: input, text: someText sh "loadStuff.sh < '${input}' "
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 11 (Web Link)" [ 14622 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 160123 ] JNJira + In-Review [ 185560 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            janus Janus Troelsen added a comment -

            Why isn't the returnStdout parameter documented at https://jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#sh-shell-script ?

            Show
            janus Janus Troelsen added a comment - Why isn't the returnStdout parameter documented at https://jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#sh-shell-script ?
            Hide
            wstrange Warren Strange added a comment -

            Is this bug fixed?

            I can not find any example of how to capture the exit status from a shell command

            Show
            wstrange Warren Strange added a comment - Is this bug fixed? I can not find any example of how to capture the exit status from a shell command
            Show
            awsteele Aidan Steele added a comment - Janus Troelsen and Warren Strange it appears that it's now in the documentation: https://jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#code-sh-code-shell-script
            abayer Andrew Bayer made changes -
            Component/s pipeline-general [ 21692 ]
            abayer Andrew Bayer made changes -
            Component/s workflow-plugin [ 18820 ]
            jcarsique Julien Carsique made changes -
            Component/s workflow-durable-task-step-plugin [ 21715 ]
            Hide
            jcarsique Julien Carsique added a comment -

            Fixed in pipeline (workflow-durable-task-step-plugin) 2.4

            Show
            jcarsique Julien Carsique added a comment - Fixed in pipeline (workflow-durable-task-step-plugin) 2.4
            Hide
            ravennous Raven Alef added a comment -

            This still seems to occur in https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Shared+Groovy+Libraries+Plugin. Am I mistaken on the fixed version?

            Show
            ravennous Raven Alef added a comment - This still seems to occur in https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Shared+Groovy+Libraries+Plugin . Am I mistaken on the fixed version?
            Hide
            jglick Jesse Glick added a comment -

            Raven Alef you are looking at an unrelated plugin.

            Show
            jglick Jesse Glick added a comment - Raven Alef you are looking at an unrelated plugin.
            Hide
            kbaltrinic Kenneth Baltrinic added a comment - - edited

            I am curious as to why this has been implemented in the way it was.   Would it not have made more sense to use a flag like throwOnFail defaulting to true to retain the original behavior, but if false, return a object (or maybe just a map of [exitCode: 0, stdOut: '...', stdErr: '...'])?  This seems like it would be a lot more useful.  As it is, if I need both output and exit code I am pretty much stuck with trying to parse the output to determine if things failed or succeeded.

            Show
            kbaltrinic Kenneth Baltrinic added a comment - - edited I am curious as to why this has been implemented in the way it was.   Would it not have made more sense to use a flag like throwOnFail defaulting to true to retain the original behavior, but if false, return a object (or maybe just a map of [exitCode: 0, stdOut: '...', stdErr: '...'] )?  This seems like it would be a lot more useful.  As it is, if I need both output and exit code I am pretty much stuck with trying to parse the output to determine if things failed or succeeded.
            Hide
            jglick Jesse Glick added a comment -

            Kenneth Baltrinic there is no need to parse anything. Even before this RFE was implemented, you could get the same effect pretty easily by either redirecting stdout to a temp file, or echoing $? to a temp file, or both.

            Show
            jglick Jesse Glick added a comment - Kenneth Baltrinic there is no need to parse anything. Even before this RFE was implemented, you could get the same effect pretty easily by either redirecting stdout to a temp file, or echoing $? to a temp file, or both.
            Hide
            rachel Rachel M. added a comment -

            It would be interesting to have an additional field in JIRA to know quickly how an issue has been solved, because I've just seen an answer at StackOverflow, from last year, recommending the workaround with a temporary file.

            Solution

            Documentation:

            https://jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#code-sh-code-shell-script

            Example:
            def VERSION = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim()

            Show
            rachel Rachel M. added a comment - It would be interesting to have an additional field in JIRA to know quickly how an issue has been solved, because I've just seen an answer at StackOverflow, from last year, recommending the workaround with a temporary file. Solution Documentation : https://jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#code-sh-code-shell-script Example: def VERSION = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim()
            Hide
            jglick Jesse Glick added a comment -

            Rachel M. you can click on the History tab to see that this was fixed on in July 2016.

            Whatever the StackOverflow answer is, supply a better answer and get yourself a reputation boost!

            Show
            jglick Jesse Glick added a comment - Rachel M. you can click on the History  tab to see that this was fixed on in July 2016. Whatever the StackOverflow answer is, supply a better answer and get yourself a reputation boost!
            Hide
            ssbarnea Sorin Sbarnea added a comment -

            Reading documentation didn't help much, reading this bug history didn't help either.

            I have a simple use case which seems (so far) to be impossible to cover using current implementation:

            I want to display the stdout if the exit code of sh() is an error, avoiding spamming the console with a verbose task that matters only if there are errors.

            If I enable the option to return stdout, I lose the ability to get the return code.

            The default behaviour does raise an exception if there are errors, so I lose the ability to print the stdout that would have being returned by the sh(). I imagine that the raised exception could have a stdout field but I really doubt that all possible raised exceptions would have this field.

            Was anyone able to find a solution to cover this use-case?

            Show
            ssbarnea Sorin Sbarnea added a comment - Reading documentation didn't help much, reading this bug history didn't help either. I have a simple use case which seems (so far) to be impossible to cover using current implementation: I want to display the stdout if the exit code of sh() is an error, avoiding spamming the console with a verbose task that matters only if there are errors. If I enable the option to return stdout, I lose the ability to get the return code. The default behaviour does raise an exception if there are errors, so I lose the ability to print the stdout that would have being returned by the sh(). I imagine that the raised exception could have a stdout field but I really doubt that all possible raised exceptions would have this field. Was anyone able to find a solution to cover this use-case?
            Hide
            jglick Jesse Glick added a comment -

            See my comment of Apr 13.

            Show
            jglick Jesse Glick added a comment - See my comment of Apr 13.
            Hide
            tridnguyen Tri Nguyen added a comment -

             What version of Jenkins did this go in?

            Show
            tridnguyen Tri Nguyen added a comment -  What version of Jenkins did this go in?
            Hide
            batmat Baptiste Mathus added a comment -

            Tri Nguyen not Jenkins, but a plugin. See https://github.com/jenkinsci/workflow-durable-task-step-plugin/commit/94ad105034de65a614d6e7d3d0504b4d1a6a761e

            So you need the Pipeline: Nodes and Processes plugin in version >= 2.4

            Show
            batmat Baptiste Mathus added a comment - Tri Nguyen not Jenkins, but a plugin. See https://github.com/jenkinsci/workflow-durable-task-step-plugin/commit/94ad105034de65a614d6e7d3d0504b4d1a6a761e So you need the Pipeline: Nodes and Processes plugin in version >= 2.4
            Hide
            quas Jakub Pawlinski added a comment -

            Its returning result or output? what if I want both? why not always return something like ScriptResultObject that has both status and the output in it?

            then you would only need a flag if you want to propagate status to jenkins, or you could have customised propagation by parsing output.

            Show
            quas Jakub Pawlinski added a comment - Its returning result or output? what if I want both? why not always return something like ScriptResultObject that has both status and the output in it? then you would only need a flag if you want to propagate status to jenkins, or you could have customised propagation by parsing output.
            Hide
            mrose Michael Rose added a comment -

            +1 for Jakub proposed improvement.

            Show
            mrose Michael Rose added a comment - +1 for Jakub proposed improvement.
            Hide
            jglick Jesse Glick added a comment -

            Jakub Pawlinski Michael Rose etc. I have no plans to implement such an RFE (see the repeated explanations of the alternative you can use today), but if you want to file it anyway, it should be a separate linked issue (and please use votes, not comments, to register your personal priority to reduce noise to watchers).

            Show
            jglick Jesse Glick added a comment - Jakub Pawlinski Michael Rose etc. I have no plans to implement such an RFE (see the repeated explanations of the alternative you can use today), but if you want to file it anyway, it should be a separate linked issue (and please use votes, not comments, to register your personal priority to reduce noise to watchers).
            Hide
            benji2006 ben ji added a comment -

            I am using a docker push command in a pipeline:  sh "docker push 10.10.174.28/dev/vote:0.$BUILD_NUMBER"

            I need to provide a password using standard input to the command - (there doesn't seem to be any other way to pass in the data in an automated environment)

            I don't understand how to apply the workaround to my case:
            writeFile file: 'input', text: someText
            sh 'loadStuff.sh < input'
            Any help would be greatly appreciated.

             

             

            Show
            benji2006 ben ji added a comment - I am using a docker push command in a pipeline:  sh "docker push 10.10.174.28/dev/vote:0.$BUILD_NUMBER" I need to provide a password using standard input to the command - (there doesn't seem to be any other way to pass in the data in an automated environment) I don't understand how to apply the workaround to my case: writeFile file: 'input', text: someText sh 'loadStuff.sh < input' Any help would be greatly appreciated.    
            Hide
            shott85 David Schott added a comment -

            Hi ben ji, there are other ways to provide a password besides stdin.

            Check out this example Jenkinsfile (apologies for weird formatting) and note the usage of Credentials & 'environment' in the "Build & Push Docker Image" stage.

            Show
            shott85 David Schott added a comment - Hi ben ji , there are other ways to provide a password besides stdin. Check out this example Jenkinsfile (apologies for weird formatting) and note the usage of Credentials & 'environment' in the "Build & Push Docker Image" stage.
            Hide
            benji2006 ben ji added a comment - - edited

            Hi David Schott,

            this isn't a regular docker log in - in this particular scenario using Notary, it looks like the only way to pass in the required content is via stdin - there are no env variables that work - see the second update to this bug report:

             

            https://forums.docker.com/t/cannot-get-trust-delegation-to-work-notary-v0-3-0-solved/14370

            SECOND UPDATE:

            Looks as if I can supply the passphrase on standard input – seems to be working now.

             

             

            I can get the command to work outside Jenkins using 

            echo "delegationpass" | docker push .....

            and need to emulate that in my pipeline script

             

            Show
            benji2006 ben ji added a comment - - edited Hi David Schott , this isn't a regular docker log in - in this particular scenario using Notary, it looks like the only way to pass in the required content is via stdin - there are no env variables that work - see the second update to this bug report:   https://forums.docker.com/t/cannot-get-trust-delegation-to-work-notary-v0-3-0-solved/14370 SECOND UPDATE: Looks as if I can supply the passphrase on standard input – seems to be working now.     I can get the command to work outside Jenkins using  echo "delegationpass" | docker push ..... and need to emulate that in my pipeline script  
            Hide
            jglick Jesse Glick added a comment -

            ben ji please do not use JIRA as a help forum. There is a users’ list and other places to ask for help using Jenkins.

            Show
            jglick Jesse Glick added a comment - ben ji please do not use JIRA as a help forum. There is a users’ list and other places to ask for help using Jenkins.
            drewish andrew morton made changes -
            Link This issue is related to JENKINS-44930 [ JENKINS-44930 ]
            Hide
            drewish andrew morton added a comment -

            I went ahead and created https://issues.jenkins-ci.org/browse/JENKINS-44930 to focus on returning the exit status and standard output at the same time. So I hope all the folks who commented asking for this feature will go up vote that

            Show
            drewish andrew morton added a comment - I went ahead and created https://issues.jenkins-ci.org/browse/JENKINS-44930  to focus on returning the exit status and standard output at the same time. So I hope all the folks who commented asking for this feature will go up vote that

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                jglick Jesse Glick
              • Votes:
                50 Vote for this issue
                Watchers:
                64 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: