Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45758

Pipeline sh, bat & sleep steps hang and executors don't release.

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Incomplete
    • Component/s: pipeline
    • Environment:
      Jenkins v2.69, running on Ubuntu 16.04LTS, with agents running Windows 10 and Ubuntu. Pipeline v2.5, Durable Task v1.14
    • Similar Issues:

      Description

      Jenkins intermittently enters a state where:

      1. Any started pipeline jobs hang indefinitely on sh, bat, and sleep steps regardless of if they are executed on master or an agent, Unix or Windows systems
      2. Pipelines without sh/bat/sleep steps complete, but do not release their executors when completed
      3. Using "/stop" on the pipeline job is completely unresponsive, but "/term" and "/kill" works
      4. Jenkins is otherwise responsive
      5. *Update* Executing shell/batch commands with groovy using String.execute() does NOT hang, but at the end of the pipeline, the executor is still not released.

      The following Pipeline script with "sleep"

      node('NodeName') {
        echo 'Hello World'
        sleep(1)
      }
      

      results in hanging at the sleep step... 

      [Pipeline] node
      Running on NodeName in C:\Jenkins_work\workspace\test_sleep_hang
      [Pipeline] {
      [Pipeline] echo
      Hello World
      [Pipeline] sleep
      Sleeping for 1 sec
      

      This pipeline script completes, 

      node('NodeName') {
        echo 'Hello World'
      }

      but does not release the executor on "NodeName" and then other jobs are blocked from running on that node as executors are exhausted.

      *Update* Multiple ThreadDumps attached from described Jenkins state.

        Attachments

        1. ThreadDump.txt
          736 kB
        2. ThreadDump2.txt
          645 kB
        3. ThreadDump-Agent.txt
          0.2 kB

          Activity

          anshuarya Anshu Arya created issue -
          anshuarya Anshu Arya made changes -
          Field Original Value New Value
          Description Jenkins intermittently enters a state where:
           # Any started pipeline jobs *hang indefinitely on sh, bat, and sleep* steps regardless of if they are executed on master or an agent, Unix or Windows systems
           # Pipelines without sh/bat/sleep steps complete, but *do not release their executors* when completed
           # Using "/stop" on the pipeline job is completely unresponsive, but "/term" and "/kill" works
           # Jenkins is otherwise responsive

           

          The following Pipeline script with "sleep"

           
          {code:java}
          node('NodeName') {
            echo 'Hello World'
            sleep(1)
          }
          {code}
          results in hanging at the sleep step...

           

           
          {code:java}
          [Pipeline] node
          Running on NodeName in C:\Jenkins_work\workspace\test_sleep_hang
          [Pipeline] {
          [Pipeline] echo
          Hello World
          [Pipeline] sleep
          Sleeping for 1 sec
          {code}
           

          This pipeline script completes, 

           
          {code:java}
          node('NodeName') {
            echo 'Hello World'
          }{code}
          but does not release the executor on "NodeName" and then other jobs are blocked from running on that node as executors are exhausted.

           

           

          ThreadDump attached from described Jenkins state.
          Jenkins intermittently enters a state where:
           # Any started pipeline jobs *hang indefinitely on sh, bat, and sleep* steps regardless of if they are executed on master or an agent, Unix or Windows systems
           # Pipelines without sh/bat/sleep steps complete, but *do not release their executors* when completed
           # Using "/stop" on the pipeline job is completely unresponsive, but "/term" and "/kill" works
           # Jenkins is otherwise responsive
           # **Update** Executing shell/batch commands with groovy using String.execute() does NOT hang, but at the end of the pipeline, the executor is still not released.

           

          The following Pipeline script with "sleep"

           
          {code:java}
          node('NodeName') {
            echo 'Hello World'
            sleep(1)
          }
          {code}
          results in hanging at the sleep step...

           

           
          {code:java}
          [Pipeline] node
          Running on NodeName in C:\Jenkins_work\workspace\test_sleep_hang
          [Pipeline] {
          [Pipeline] echo
          Hello World
          [Pipeline] sleep
          Sleeping for 1 sec
          {code}
           

          This pipeline script completes, 

           
          {code:java}
          node('NodeName') {
            echo 'Hello World'
          }{code}
          but does not release the executor on "NodeName" and then other jobs are blocked from running on that node as executors are exhausted.

           

           

          ThreadDump attached from described Jenkins state.
          anshuarya Anshu Arya made changes -
          Attachment ThreadDump2.txt [ 39019 ]
          anshuarya Anshu Arya made changes -
          Description Jenkins intermittently enters a state where:
           # Any started pipeline jobs *hang indefinitely on sh, bat, and sleep* steps regardless of if they are executed on master or an agent, Unix or Windows systems
           # Pipelines without sh/bat/sleep steps complete, but *do not release their executors* when completed
           # Using "/stop" on the pipeline job is completely unresponsive, but "/term" and "/kill" works
           # Jenkins is otherwise responsive
           # **Update** Executing shell/batch commands with groovy using String.execute() does NOT hang, but at the end of the pipeline, the executor is still not released.

           

          The following Pipeline script with "sleep"

           
          {code:java}
          node('NodeName') {
            echo 'Hello World'
            sleep(1)
          }
          {code}
          results in hanging at the sleep step...

           

           
          {code:java}
          [Pipeline] node
          Running on NodeName in C:\Jenkins_work\workspace\test_sleep_hang
          [Pipeline] {
          [Pipeline] echo
          Hello World
          [Pipeline] sleep
          Sleeping for 1 sec
          {code}
           

          This pipeline script completes, 

           
          {code:java}
          node('NodeName') {
            echo 'Hello World'
          }{code}
          but does not release the executor on "NodeName" and then other jobs are blocked from running on that node as executors are exhausted.

           

           

          ThreadDump attached from described Jenkins state.
          Jenkins intermittently enters a state where:
           # Any started pipeline jobs *hang indefinitely on sh, bat, and sleep* steps regardless of if they are executed on master or an agent, Unix or Windows systems
           # Pipelines without sh/bat/sleep steps complete, but *do not release their executors* when completed
           # Using "/stop" on the pipeline job is completely unresponsive, but "/term" and "/kill" works
           # Jenkins is otherwise responsive
           # **Update** Executing shell/batch commands with groovy using String.execute() does NOT hang, but at the end of the pipeline, the executor is still not released.

           

          The following Pipeline script with "sleep"

           
          {code:java}
          node('NodeName') {
            echo 'Hello World'
            sleep(1)
          }
          {code}
          results in hanging at the sleep step...

           

           
          {code:java}
          [Pipeline] node
          Running on NodeName in C:\Jenkins_work\workspace\test_sleep_hang
          [Pipeline] {
          [Pipeline] echo
          Hello World
          [Pipeline] sleep
          Sleeping for 1 sec
          {code}
           

          This pipeline script completes, 

           
          {code:java}
          node('NodeName') {
            echo 'Hello World'
          }{code}
          but does not release the executor on "NodeName" and then other jobs are blocked from running on that node as executors are exhausted.

          **Update** Multiple ThreadDumps attached from described Jenkins state.
          anshuarya Anshu Arya made changes -
          Attachment ThreadDump-Agent.txt [ 39021 ]
          anshuarya Anshu Arya made changes -
          Description Jenkins intermittently enters a state where:
           # Any started pipeline jobs *hang indefinitely on sh, bat, and sleep* steps regardless of if they are executed on master or an agent, Unix or Windows systems
           # Pipelines without sh/bat/sleep steps complete, but *do not release their executors* when completed
           # Using "/stop" on the pipeline job is completely unresponsive, but "/term" and "/kill" works
           # Jenkins is otherwise responsive
           # **Update** Executing shell/batch commands with groovy using String.execute() does NOT hang, but at the end of the pipeline, the executor is still not released.

           

          The following Pipeline script with "sleep"

           
          {code:java}
          node('NodeName') {
            echo 'Hello World'
            sleep(1)
          }
          {code}
          results in hanging at the sleep step...

           

           
          {code:java}
          [Pipeline] node
          Running on NodeName in C:\Jenkins_work\workspace\test_sleep_hang
          [Pipeline] {
          [Pipeline] echo
          Hello World
          [Pipeline] sleep
          Sleeping for 1 sec
          {code}
           

          This pipeline script completes, 

           
          {code:java}
          node('NodeName') {
            echo 'Hello World'
          }{code}
          but does not release the executor on "NodeName" and then other jobs are blocked from running on that node as executors are exhausted.

          **Update** Multiple ThreadDumps attached from described Jenkins state.
          Jenkins intermittently enters a state where:
           # Any started pipeline jobs *hang indefinitely on sh, bat, and sleep* steps regardless of if they are executed on master or an agent, Unix or Windows systems
           # Pipelines without sh/bat/sleep steps complete, but *do not release their executors* when completed
           # Using "/stop" on the pipeline job is completely unresponsive, but "/term" and "/kill" works
           # Jenkins is otherwise responsive
           # **Update** Executing shell/batch commands with groovy using String.execute() does NOT hang, but at the end of the pipeline, the executor is still not released.

           

          The following Pipeline script with "sleep"

           
          {code:java}
          node('NodeName') {
            echo 'Hello World'
            sleep(1)
          }
          {code}
          results in hanging at the sleep step... 
          {code:java}
          [Pipeline] node
          Running on NodeName in C:\Jenkins_work\workspace\test_sleep_hang
          [Pipeline] {
          [Pipeline] echo
          Hello World
          [Pipeline] sleep
          Sleeping for 1 sec
          {code}
          This pipeline script completes, 
          {code:java}
          node('NodeName') {
            echo 'Hello World'
          }{code}
          but does not release the executor on "NodeName" and then other jobs are blocked from running on that node as executors are exhausted.

          **Update** Multiple ThreadDumps attached from described Jenkins state.
          anshuarya Anshu Arya made changes -
          Description Jenkins intermittently enters a state where:
           # Any started pipeline jobs *hang indefinitely on sh, bat, and sleep* steps regardless of if they are executed on master or an agent, Unix or Windows systems
           # Pipelines without sh/bat/sleep steps complete, but *do not release their executors* when completed
           # Using "/stop" on the pipeline job is completely unresponsive, but "/term" and "/kill" works
           # Jenkins is otherwise responsive
           # **Update** Executing shell/batch commands with groovy using String.execute() does NOT hang, but at the end of the pipeline, the executor is still not released.

           

          The following Pipeline script with "sleep"

           
          {code:java}
          node('NodeName') {
            echo 'Hello World'
            sleep(1)
          }
          {code}
          results in hanging at the sleep step... 
          {code:java}
          [Pipeline] node
          Running on NodeName in C:\Jenkins_work\workspace\test_sleep_hang
          [Pipeline] {
          [Pipeline] echo
          Hello World
          [Pipeline] sleep
          Sleeping for 1 sec
          {code}
          This pipeline script completes, 
          {code:java}
          node('NodeName') {
            echo 'Hello World'
          }{code}
          but does not release the executor on "NodeName" and then other jobs are blocked from running on that node as executors are exhausted.

          **Update** Multiple ThreadDumps attached from described Jenkins state.
          Jenkins intermittently enters a state where:
           # Any started pipeline jobs *hang indefinitely on sh, bat, and sleep* steps regardless of if they are executed on master or an agent, Unix or Windows systems
           # Pipelines without sh/bat/sleep steps complete, but *do not release their executors* when completed
           # Using "/stop" on the pipeline job is completely unresponsive, but "/term" and "/kill" works
           # Jenkins is otherwise responsive
           # **Update** Executing shell/batch commands with groovy using String.execute() does NOT hang, but at the end of the pipeline, the executor is still not released.

          The following Pipeline script with "sleep"
          {code:java}
          node('NodeName') {
            echo 'Hello World'
            sleep(1)
          }
          {code}
          results in hanging at the sleep step... 
          {code:java}
          [Pipeline] node
          Running on NodeName in C:\Jenkins_work\workspace\test_sleep_hang
          [Pipeline] {
          [Pipeline] echo
          Hello World
          [Pipeline] sleep
          Sleeping for 1 sec
          {code}
          This pipeline script completes, 
          {code:java}
          node('NodeName') {
            echo 'Hello World'
          }{code}
          but does not release the executor on "NodeName" and then other jobs are blocked from running on that node as executors are exhausted.

          **Update** Multiple ThreadDumps attached from described Jenkins state.
          jglick Jesse Glick made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Incomplete [ 4 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              anshuarya Anshu Arya
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: