Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46089

ProcessTreeKiller broken in pipeline jobs

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      After one of the recent updates to pipeline plug-ins the process tree killer always kills all processes even if the BUILD_ID (as described in https://wiki.jenkins.io/display/JENKINS/ProcessTreeKiller) is changed. This is a serious regression because some of our build jobs stopped working.

      The following pipeline script can be used to reproduce the problem:

      node('master') {
        sh '''BUILD_ID="dont-kill-me bla/blub" sleep 60 &'''
      }
      
      

      The sleep process is killed immediately when the job finishes. The problem only occurs in Pipeline jobs, freestyle jobs still work as expected, i.e. the sleep process continues running for 60 seconds.

        Attachments

          Issue Links

            Activity

            Hide
            svanoort Sam Van Oort added a comment -

            Hi Thorsten,
            This is Pipeline working as designed – what you're describing would permit the Pipeline to leak processes potentially due to how it's launching them with the shell step. IIUC what you're saying, the fact that it doesn't kill the process in FreeStyle projects would be a bug with Freestyle.

            Show
            svanoort Sam Van Oort added a comment - Hi Thorsten, This is Pipeline working as designed – what you're describing would permit the Pipeline to leak processes potentially due to how it's launching them with the shell step. IIUC what you're saying, the fact that it doesn't kill the process in FreeStyle projects would be a bug with Freestyle.
            Hide
            gfriedrich Georg Friedrich added a comment -

            Hi Sam Van Oort,

            this is either a bug for FreeStyle projects and therefore also a bug in the documentation, because the documentation page of the ProcessTreeKiller @ https://wiki.jenkins.io/display/JENKINS/ProcessTreeKiller clearly states what you should do when you want to keep a Daemon process alive. Or this is working as intended for FreeStyle projects but is broken for Pipeline builds.

            Besides there are very good reasons for keeping a process running. E.g. when you are using Gradle for your build environment, Gradle starts a Daemon that make subsequent builds faster. As Jenkins kills all Daemons at the end of each build, it clearly breaks a feature of Gradle. So there have to be a possibility to keep created processes alive.

            Oleg Nenashev or Jesse Glick maybe one of you can comment on this.

            Thanks in advance!

            Show
            gfriedrich Georg Friedrich added a comment - Hi Sam Van Oort , this is either a bug for FreeStyle projects and therefore also a bug in the documentation, because the documentation page of the ProcessTreeKiller @ https://wiki.jenkins.io/display/JENKINS/ProcessTreeKiller clearly states what you should do when you want to keep a Daemon process alive. Or this is working as intended for FreeStyle projects but is broken for Pipeline builds. Besides there are very good reasons for keeping a process running. E.g. when you are using Gradle for your build environment, Gradle starts a Daemon that make subsequent builds faster. As Jenkins kills all Daemons at the end of each build, it clearly breaks a feature of Gradle. So there have to be a possibility to keep created processes alive. Oleg Nenashev or Jesse Glick maybe one of you can comment on this. Thanks in advance!
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Generally I agree with your diagnosis Georg Friedrich. The feature is documented in a way that it makes impression that it should work in Pipeline. Just because there was no Pipeline in April 2011 when the page was updated last time...

            "dont-kill-me" always seemed to be a hack to me. If there are processes which should not be terminated by builds, they should be ideally to be started in another way (e.g. by service, etc.). In the case of Gradlethe daemon could be started automatically outside the Jenkins job (e.g. by VM initialization logic or by a Gradle tool installer).

            • If Durable Task logic does not support ProcessTreeKiller extension points, this is a bug.
            • If it just does not support "dont-kill-me", I think it would be enough to document it.
            Show
            oleg_nenashev Oleg Nenashev added a comment - Generally I agree with your diagnosis Georg Friedrich . The feature is documented in a way that it makes impression that it should work in Pipeline. Just because there was no Pipeline in April 2011 when the page was updated last time... "dont-kill-me" always seemed to be a hack to me. If there are processes which should not be terminated by builds, they should be ideally to be started in another way (e.g. by service, etc.). In the case of Gradlethe daemon could be started automatically outside the Jenkins job (e.g. by VM initialization logic or by a Gradle tool installer). If Durable Task logic does not support ProcessTreeKiller extension points, this is a bug. If it just does not support "dont-kill-me", I think it would be enough to document it.
            Hide
            sithmein Thorsten Meinl added a comment -

            It would be a real shame if this very useful functionality wasn't available in Pipeline jobs. We use it to deploy the full application stack after a build which can then be used for testing and playing around. Since we have many branches it's not feasible to set up environments outside of Jenkins and somehow deploy the build results.

            Show
            sithmein Thorsten Meinl added a comment - It would be a real shame if this very useful functionality wasn't available in Pipeline jobs. We use it to deploy the full application stack after a build which can then be used for testing and playing around. Since we have many branches it's not feasible to set up environments outside of Jenkins and somehow deploy the build results.
            Hide
            gfriedrich Georg Friedrich added a comment -

            I think so too, Oleg Nenashev. Overriding the BUILD_ID environment variable looks very hacky.

            Unfortunately there is no possibility for Gradle to run the daemons initially. And of course the daemon feature of Gradle will only make sense for "always-on" build nodes, so using it in combination with docker agents is useless.

            That said: Would it be possible (from an architectural point of view) to disable the ProcessTreeKiller for specific builds instead of using this BUILD_ID hack? I think there are enough reasons here to give users the possibility to do so.

            Show
            gfriedrich Georg Friedrich added a comment - I think so too, Oleg Nenashev . Overriding the BUILD_ID environment variable looks very hacky. Unfortunately there is no possibility for Gradle to run the daemons initially. And of course the daemon feature of Gradle will only make sense for "always-on" build nodes, so using it in combination with docker agents is useless. That said: Would it be possible (from an architectural point of view) to disable the ProcessTreeKiller for specific builds instead of using this BUILD_ID hack? I think there are enough reasons here to give users the possibility to do so.
            Hide
            gfriedrich Georg Friedrich added a comment -

            Ah, I just found that there already IS the ability to stop Jenkins from killing processes by using the "veto kill process"-feature. You should have mentioned this Oleg Nenashev

            As far as I can see this is/was bugged for slave nodes but just got fixed 2 days ago. Unfortunately it missed the 2.119 release but I guess it will be available with 2.120 then.

            That's really great news, so I will wait patiently for the 2.120 and will create an extension for not killing the Gradle daemons.

            Show
            gfriedrich Georg Friedrich added a comment - Ah, I just found that there already IS the ability to stop Jenkins from killing processes by using the "veto kill process"-feature. You should have mentioned this Oleg Nenashev As far as I can see this is/was bugged for slave nodes but just got fixed 2 days ago. Unfortunately it missed the 2.119 release but I guess it will be available with 2.120 then. That's really great news, so I will wait patiently for the 2.120 and will create an extension for not killing the Gradle daemons.
            Hide
            jglick Jesse Glick added a comment -

            Try

            sh '''
            JENKINS_SERVER_COOKIE=ignore ./gradlew whatever
            '''
            

            Maybe it works, maybe not, just a thought.

            Show
            jglick Jesse Glick added a comment - Try sh ''' JENKINS_SERVER_COOKIE=ignore ./gradlew whatever ''' Maybe it works, maybe not, just a thought.

              People

              • Assignee:
                Unassigned
                Reporter:
                sithmein Thorsten Meinl
              • Votes:
                1 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated: