Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53315

Timeout step should support a closure to execute prior to killing body

    Details

    • Similar Issues:

      Description

      Currently, the timeout step simply kills whatever processes were launched during execution of its body, and then throws an exception.  This makes it difficult to perform any automated debugging on these processes, since they are killed by the time the user finds out that they are hung (or slow).   It would be nice to be able to get some information about the state of affairs before things are killed, and maybe even perform safe shutdown steps prior to kill.

      Currently: 

      try {
        timeout(time: 1, unit: 'HOURS') {
           sh "java IntermittentlySlowProcess"
        }
      } catch (t) {
          //It's too late to, for example, send a "kill -3" to the slow/hung java process
      }
      

      What I'd propose

      (and I'm willing to try to make a PR if this seems reasonable):

      timeout(time: 1, unit: 'HOURS', beforeKill: {
         sh "killall -3 java" //for example
      }) {
         sh "java IntermittentlySlowProcess"
      }
      

      The new  "beforeKill" closure can be used for clean shutdown of complex tasks, analysis of problems, etc.

      One workaround may be to wrap whatever you are running and trap signals, but that's ugly and error-prone (and will likely cause zombies)

      Thoughts welcome.

       

       

        Attachments

          Activity

          akom Alexander Komarov created issue -
          akom Alexander Komarov made changes -
          Field Original Value New Value
          Description Currently, the timeout step simply kills whatever processes were launched during execution of its body, and then throws an exception.  This makes it *difficult to perform any automated debugging* on these processes, since they are killed by the time the user finds out that they are hung (or slow). 

          You can argue that this should be done outside of Jenkins, but, too often, Jenkins is the only place this is visible due to frequency of execution or differences in the environment.
          h3. Currently:

           
          {code:java}
          try {
            timeout(time: 1, unit: 'HOURS') {
               sh "java IntermittentlySlowProcess"
            }
          } catch (t) {
              //It's too late to, for example, send a "kill -3" to the slow/hung java process
          }
          {code}
           
          h3. What I'd propose

          (and I'm willing to try to make a PR if this seems reasonable):

           
          {code:java}
          timeout(time: 1, unit: 'HOURS', beforeKill: {
             sh "killall -3 java" //for example
          }) {
             sh "java IntermittentlySlowProcess"
          }
          {code}
           

           

          "beforeKill" can be used for clean shutdown of complex tasks, analysis of problems, etc.

          Thoughts welcome.
          Currently, the timeout step simply kills whatever processes were launched during execution of its body, and then throws an exception.  This makes it *difficult to perform any automated debugging* on these processes, since they are killed by the time the user finds out that they are hung (or slow). 

          You can argue that this should be done outside of Jenkins, but, too often, Jenkins is the only place this is visible due to frequency of execution or differences in the environment.
          h3. Currently: 
          {code:java}
          try {
            timeout(time: 1, unit: 'HOURS') {
               sh "java IntermittentlySlowProcess"
            }
          } catch (t) {
              //It's too late to, for example, send a "kill -3" to the slow/hung java process
          }
          {code}
           
          h3. What I'd propose

          (and I'm willing to try to make a PR if this seems reasonable):
          {code:java}
          timeout(time: 1, unit: 'HOURS', beforeKill: {
             sh "killall -3 java" //for example
          }) {
             sh "java IntermittentlySlowProcess"
          }
          {code}
          The new  "beforeKill" closure can be used for clean shutdown of complex tasks, analysis of problems, etc.

          Thoughts welcome.
          akom Alexander Komarov made changes -
          Description Currently, the timeout step simply kills whatever processes were launched during execution of its body, and then throws an exception.  This makes it *difficult to perform any automated debugging* on these processes, since they are killed by the time the user finds out that they are hung (or slow). 

          You can argue that this should be done outside of Jenkins, but, too often, Jenkins is the only place this is visible due to frequency of execution or differences in the environment.
          h3. Currently: 
          {code:java}
          try {
            timeout(time: 1, unit: 'HOURS') {
               sh "java IntermittentlySlowProcess"
            }
          } catch (t) {
              //It's too late to, for example, send a "kill -3" to the slow/hung java process
          }
          {code}
           
          h3. What I'd propose

          (and I'm willing to try to make a PR if this seems reasonable):
          {code:java}
          timeout(time: 1, unit: 'HOURS', beforeKill: {
             sh "killall -3 java" //for example
          }) {
             sh "java IntermittentlySlowProcess"
          }
          {code}
          The new  "beforeKill" closure can be used for clean shutdown of complex tasks, analysis of problems, etc.

          Thoughts welcome.
          Currently, the timeout step simply kills whatever processes were launched during execution of its body, and then throws an exception.  This makes it *difficult to perform any automated debugging* on these processes, since they are killed by the time the user finds out that they are hung (or slow). 

          You can argue that this should be done outside of Jenkins, but, too often, Jenkins is the only place this is visible due to frequency of execution or differences in the environment.
          h3. Currently: 
          {code:java}
          try {
            timeout(time: 1, unit: 'HOURS') {
               sh "java IntermittentlySlowProcess"
            }
          } catch (t) {
              //It's too late to, for example, send a "kill -3" to the slow/hung java process
          }
          {code}
          h3. What I'd propose

          (and I'm willing to try to make a PR if this seems reasonable):
          {code:java}
          timeout(time: 1, unit: 'HOURS', beforeKill: {
             sh "killall -3 java" //for example
          }) {
             sh "java IntermittentlySlowProcess"
          }
          {code}
          The new  "beforeKill" closure can be used for clean shutdown of complex tasks, analysis of problems, etc.

          Thoughts welcome.
          akom Alexander Komarov made changes -
          Description Currently, the timeout step simply kills whatever processes were launched during execution of its body, and then throws an exception.  This makes it *difficult to perform any automated debugging* on these processes, since they are killed by the time the user finds out that they are hung (or slow). 

          You can argue that this should be done outside of Jenkins, but, too often, Jenkins is the only place this is visible due to frequency of execution or differences in the environment.
          h3. Currently: 
          {code:java}
          try {
            timeout(time: 1, unit: 'HOURS') {
               sh "java IntermittentlySlowProcess"
            }
          } catch (t) {
              //It's too late to, for example, send a "kill -3" to the slow/hung java process
          }
          {code}
          h3. What I'd propose

          (and I'm willing to try to make a PR if this seems reasonable):
          {code:java}
          timeout(time: 1, unit: 'HOURS', beforeKill: {
             sh "killall -3 java" //for example
          }) {
             sh "java IntermittentlySlowProcess"
          }
          {code}
          The new  "beforeKill" closure can be used for clean shutdown of complex tasks, analysis of problems, etc.

          Thoughts welcome.
          Currently, the timeout step simply kills whatever processes were launched during execution of its body, and then throws an exception.  This makes it *difficult to perform any automated debugging* on these processes, since they are killed by the time the user finds out that they are hung (or slow).   It would be nice to be able to get some information about the state of affairs before things are killed, and maybe even perform safe shutdown steps prior to kill.
          h3. Currently: 
          {code:java}
          try {
            timeout(time: 1, unit: 'HOURS') {
               sh "java IntermittentlySlowProcess"
            }
          } catch (t) {
              //It's too late to, for example, send a "kill -3" to the slow/hung java process
          }
          {code}
          h3. What I'd propose

          (and I'm willing to try to make a PR if this seems reasonable):
          {code:java}
          timeout(time: 1, unit: 'HOURS', beforeKill: {
             sh "killall -3 java" //for example
          }) {
             sh "java IntermittentlySlowProcess"
          }
          {code}
          The new  "beforeKill" closure can be used for clean shutdown of complex tasks, analysis of problems, etc.

          One workaround may be to wrap whatever you are running and trap signals, but that's ugly and error-prone (and will likely cause zombies)

          Thoughts welcome.

           

           
          akom Alexander Komarov made changes -
          Status Open [ 1 ] Closed [ 6 ]
          Resolution Won't Do [ 10001 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              akom Alexander Komarov
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: