Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48271

Strings with exclamation mark passed to sh step get single quoted

    Details

    • Similar Issues:

      Description

      Using this as a minimal example:

       

      script {
          def testString1 = "echo a="
          def testString2 = "!123"
          def testString3 = "b="
          def testString4 = "2937"
      
          print "testString1+2 = "+testString1+testString2
          print "testString3+4 = "+testString3+testString4
          print "total: "+testString1+testString2+testString3+testString4
      
          sh "echo ${testString1}${testString2}"
          sh "eval ${testString1}${testString2}"
          
          sh "echo ${testString1}${testString2} ${testString3}${testString4}"
          sh "eval ${testString1}${testString2} ${testString3}${testString4}"
      }
      

       

      The result while running the pipeline is:

       

      
      [Pipeline] script
      [Pipeline] {
      [Pipeline] echo
      testString1+2 = echo a=!123
      [Pipeline] echo
      testString3+4 = b=2937
      [Pipeline] echo
      total: echo a=!123b=2937
      [Pipeline] sh
      [pipeName] Running shell script
      + echo echo 'a=!123'
      echo a=!123
      [Pipeline] sh
      [pipeName] Running shell script
      + eval echo 'a=!123'
      ++ echo 'a=!123'
      a=!123
      [Pipeline] sh
      [pipeName] Running shell script
      + echo echo 'a=!123' b=2937
      echo a=!123 b=2937
      [Pipeline] sh
      [pipeName] Running shell script
      + eval echo 'a=!123' b=2937
      ++ echo 'a=!123' b=2937
      a=!123 b=2937
      [Pipeline] }
      [Pipeline] // script
      

       

      Notice that last echo treats strings differently depending on whether or not the exclamation mark is contained within a contiguous string.

      This creates issues when, for example, creating maven options dynamically at runtime. Trying to generate 

      -Dopt1=test -Dopt2=test!1

       leads to a string

      -Dopt1=test '-Dopt2=test!1'

      which then doesn't work.

      This treatment of strings lacking consistency, and despite all my efforts to try to trim the string (or create the string within a shell script, etc), ultimately this issue is a bottleneck.

      Let me know if I am missing something, I will happily provide more information if needed.

       

       

       

        Attachments

          Activity

          Hide
          abayer Andrew Bayer added a comment -

          And

          def testString5 = "non_existing_thingie -Dpw=Password1!"
          sh "eval ${testString5}"
          

          resulted in

          + eval non_existing_thingie '-Dpw=Password1!'
          ++ non_existing_thingie '-Dpw=Password1!'
          
          Show
          abayer Andrew Bayer added a comment - And def testString5 = "non_existing_thingie -Dpw=Password1!" sh "eval ${testString5}" resulted in + eval non_existing_thingie '-Dpw=Password1!' ++ non_existing_thingie '-Dpw=Password1!'
          Hide
          abayer Andrew Bayer added a comment -

          Ok, I could be wrong:
          This:

          node {
              writeFile(file:'non_existing_thingie',text:'''
          #!/bin/bash 
          echo $@
          ''')
              sh "chmod +x non_existing_thingie"
              def testString1 = "./non_existing_thingie clean test "
              def testString2 = "-Denv=beta"
              def testString3 = "-Dpw=Password1!"
              def testString4 = "-Dpw2=test2"
              assert testString3 == "-Dpw=Password1" + '!'
              def fullTestString = "${testString1} ${testString2} ${testString3} ${testString4}"
              echo "fts: ${fullTestString}"
              sh "echo ${fullTestString}"
              sh "eval ${fullTestString}"
              def testString5 = "./non_existing_thingie -Dpw=Password1!"
              sh "eval ${testString5}"
          }
          

          resulted in

          [Pipeline] node
          Running on centos in /home/jenkins/workspace/bug-reproduction/jenkins-48271
          [Pipeline] {
          [Pipeline] writeFile
          [Pipeline] sh
          [jenkins-48271] Running shell script
          + chmod +x non_existing_thingie
          [Pipeline] echo
          fts: ./non_existing_thingie clean test  -Denv=beta -Dpw=Password1! -Dpw2=test2
          [Pipeline] sh
          [jenkins-48271] Running shell script
          + echo ./non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2
          ./non_existing_thingie clean test -Denv=beta -Dpw=Password1! -Dpw2=test2
          [Pipeline] sh
          [jenkins-48271] Running shell script
          + eval ./non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2
          ++ ./non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2
          clean test -Denv=beta -Dpw=Password1! -Dpw2=test2
          [Pipeline] sh
          [jenkins-48271] Running shell script
          + eval ./non_existing_thingie '-Dpw=Password1!'
          ++ ./non_existing_thingie '-Dpw=Password1!'
          -Dpw=Password1!
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] End of Pipeline
          Finished: SUCCESS
          
          Show
          abayer Andrew Bayer added a comment - Ok, I could be wrong: This: node { writeFile(file: 'non_existing_thingie' ,text:''' #!/bin/bash echo $@ ''') sh "chmod +x non_existing_thingie" def testString1 = "./non_existing_thingie clean test " def testString2 = "-Denv=beta" def testString3 = "-Dpw=Password1!" def testString4 = "-Dpw2=test2" assert testString3 == "-Dpw=Password1" + '!' def fullTestString = "${testString1} ${testString2} ${testString3} ${testString4}" echo "fts: ${fullTestString}" sh "echo ${fullTestString}" sh "eval ${fullTestString}" def testString5 = "./non_existing_thingie -Dpw=Password1!" sh "eval ${testString5}" } resulted in [Pipeline] node Running on centos in /home/jenkins/workspace/bug-reproduction/jenkins-48271 [Pipeline] { [Pipeline] writeFile [Pipeline] sh [jenkins-48271] Running shell script + chmod +x non_existing_thingie [Pipeline] echo fts: ./non_existing_thingie clean test -Denv=beta -Dpw=Password1! -Dpw2=test2 [Pipeline] sh [jenkins-48271] Running shell script + echo ./non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2 ./non_existing_thingie clean test -Denv=beta -Dpw=Password1! -Dpw2=test2 [Pipeline] sh [jenkins-48271] Running shell script + eval ./non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2 ++ ./non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2 clean test -Denv=beta -Dpw=Password1! -Dpw2=test2 [Pipeline] sh [jenkins-48271] Running shell script + eval ./non_existing_thingie '-Dpw=Password1!' ++ ./non_existing_thingie '-Dpw=Password1!' -Dpw=Password1! [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS
          Hide
          kon Kalle Niemitalo added a comment -

          I don't see why one would use "eval" here in the first place, since ${testString1}${testString2} is expanding Groovy variables rather than shell variables.

          Mehdy Chaillou, what is the error you get from maven? Does it fail slowly enough that you have time to run "ps" during the build and check the command-line arguments that way?

          Show
          kon Kalle Niemitalo added a comment - I don't see why one would use "eval" here in the first place, since ${testString1}${testString2} is expanding Groovy variables rather than shell variables. Mehdy Chaillou , what is the error you get from maven? Does it fail slowly enough that you have time to run "ps" during the build and check the command-line arguments that way?
          Hide
          cmehdy Mehdy Chaillou added a comment - - edited

          I'm creating runtime options on the fly in a few different places, and put it all together at the shell step (with or without the withMaven() plugin).

          With mvnTest being a groovy function (mvnTest.groovy in /vars with a call() defined, to return a string like:

          -Dpassword1=test -Dpassword2=test!2

          The following steps:  

          script {
            mvnCommand = mvnTest(suiteFile)
          }
          
          withMaven(maven: "M3", //){
           options: [
           junitPublisher(disabled:true),
           artifactsPublisher(disabled: true),
           findbugsPublisher(disabled: true),
           openTasksPublisher(disabled: true)
           ]) {
           sh "mvn ${mvnCommand}"
          }

          end up calling maven this way:

          + mvn clean test -Dpasssword1=test '-Dpassword2=test!2' -Dotherstuff=blah
           [INFO] Scanning for projects...

          Something which doesn't show, were I to just try to 'print mvnCommand', or to sh "echo ${mvnCommand}".

           

          Interestingly, and following your suggestion, doing a `ps` while the step is being executed shows that the parameters are passed without single or double quotes.

          I've had to remove some sensitive data so I hope you'll forgive me if it's not the most readable, but you get the point.

           

           

          Show
          cmehdy Mehdy Chaillou added a comment - - edited I'm creating runtime options on the fly in a few different places, and put it all together at the shell step (with or without the withMaven() plugin). With mvnTest being a groovy function (mvnTest.groovy in /vars with a call() defined, to return a string like: -Dpassword1=test -Dpassword2=test!2 The following steps:   script {   mvnCommand = mvnTest(suiteFile) } withMaven(maven: "M3" , //){ options: [ junitPublisher(disabled: true ), artifactsPublisher(disabled: true ), findbugsPublisher(disabled: true ), openTasksPublisher(disabled: true ) ]) { sh "mvn ${mvnCommand}" } end up calling maven this way: + mvn clean test -Dpasssword1=test '-Dpassword2=test!2' -Dotherstuff=blah [INFO] Scanning for projects... Something which doesn't show, were I to just try to 'print mvnCommand', or to sh "echo ${mvnCommand}".   Interestingly, and following your suggestion, doing a `ps` while the step is being executed shows that the parameters are passed without single or double quotes. I've had to remove some sensitive data so I hope you'll forgive me if it's not the most readable, but you get the point.    
          Hide
          abayer Andrew Bayer added a comment -

          So it sounds like this isn't actually an issue, if the Maven process got the argument correctly. I'm going to close this as not a defect.

          Show
          abayer Andrew Bayer added a comment - So it sounds like this isn't actually an issue, if the Maven process got the argument correctly. I'm going to close this as not a defect.

            People

            • Assignee:
              Unassigned
              Reporter:
              cmehdy Mehdy Chaillou
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: