Details

    • Similar Issues:
    • Released As:
      durable-task 1.31

      Description

      durable task step currently uses nohup to launch a durable process. But if Jenkins is started from an interactive terminal and the user presses Ctrl+C, the forked process is still gone. So far we've blushed it off saying this is not how Jenkins is typically run. That may be true, but it is also a perfectly reasonable way to run Jenkins, for example for the first time evaluation.

      The reason these processes get killed with Ctrl+C is because shell sends SIGINT to all the processes in the process group (source). In looking at nohup.c, nohup only ignores SIGHUP. You can also run a command like nohup sleep 30 from the command line. hit Ctrl+C, and observe that the sleep 30 process gets killed.

      The root problem is that nohup is a poor way to isolate a child process. Specifically, it doesn't put the process into a new process group, so it's vulnerable to any signal sent to the entire process group (of which Ctrl+C is one.) setsid is a better way of doing this. This puts the process into a new session (hence also new process group.) So no group-wide signal will get to the child process.

      See Wikipedia process group page for interaction of signals, process groups, and sessions.

        Attachments

          Issue Links

            Activity

            Hide
            rodrigc Craig Rodrigues added a comment -

            Kohsuke Kawaguchi I'm still having tons of problems with durable task plugin due to this problem.  I think setsid is the way to go on this.

            The idea to write a setsid.py wrapper in Python is intriguing.  It would definitely work.

            My concern is that now Python becomes a core dependency for running Jenkins Pipeline.

            I have no problem with that because I love Python and use Python heavily in my projects.

            However, from a devops perspective this might be a problem because people now need to add Python to their dependencies for running Jenkins.  This becomes more interesting as more Jenkins deployments are running in a dockerized type of world.

            Would it be possible to write a small setsid wrapper in JNA?

            Show
            rodrigc Craig Rodrigues added a comment - Kohsuke Kawaguchi I'm still having tons of problems with durable task plugin due to this problem.  I think setsid is the way to go on this. The idea to write a setsid.py wrapper in Python is intriguing.  It would definitely work. My concern is that now Python becomes a core dependency for running Jenkins Pipeline. I have no problem with that because I love Python and use Python heavily in my projects. However, from a devops perspective this might be a problem because people now need to add Python to their dependencies for running Jenkins.  This becomes more interesting as more Jenkins deployments are running in a dockerized type of world. Would it be possible to write a small setsid wrapper in JNA?
            Hide
            rodrigc Craig Rodrigues added a comment -

            jnr-posix has a wrapper for setsid() so this might be doable in Java:

             

            https://github.com/jnr/jnr-posix/blob/master/src/main/java/jnr/posix/POSIX.java#L107

             

            Show
            rodrigc Craig Rodrigues added a comment - jnr-posix has a wrapper for setsid() so this might be doable in Java:   https://github.com/jnr/jnr-posix/blob/master/src/main/java/jnr/posix/POSIX.java#L107  
            Hide
            jglick Jesse Glick added a comment -

            The JVM is likely to be too slow. A (statically-compiled) Go wrapper is probably most practical.

            Show
            jglick Jesse Glick added a comment - The JVM is likely to be too slow. A (statically-compiled) Go wrapper is probably most practical.
            Hide
            jglick Jesse Glick added a comment -

            The advantages of Golang for this purpose are clear: small size; fast startup; static linking to minimize the chance of divergent runtime behaviors; access to platform-dependent system APIs like this one; and of course the availability of cross-compilation so we could have a Dockerized build process creating binaries for all supported platforms (whatever subset of go tool dist list) at once.

            There will however still be some exotic platforms not supported by Golang, such as z/OS, and probably some cases where sh is being used in a semi-POSIX environment like Cygwin that might choke on lower-level calls. So I think the current shell wrapper still needs to be available as a fallback (not sure how you decide when to use it) and given some smoke test coverage, though it could be stripped of all its advanced features like PID tracking.

            Show
            jglick Jesse Glick added a comment - The advantages of Golang for this purpose are clear: small size; fast startup; static linking to minimize the chance of divergent runtime behaviors; access to platform-dependent system APIs like this one ; and of course the availability of cross-compilation so we could have a Dockerized build process creating binaries for all supported platforms (whatever subset of go tool dist list ) at once. There will however still be some exotic platforms not supported by Golang, such as z/OS, and probably some cases where sh is being used in a semi-POSIX environment like Cygwin that might choke on lower-level calls. So I think the current shell wrapper still needs to be available as a fallback (not sure how you decide when to use it) and given some smoke test coverage, though it could be stripped of all its advanced features like PID tracking.
            Hide
            carroll Carroll Chiou added a comment -

            Shell wrapper script has now been replaced by pre-compiled golang binaries (for Unix and Darwin). The binary allows the script to have its own session id and removes some polling overhead. This should increase the survivability of long-running scripts when jenkins terminates unexpectedly. This does not change the survivability of the script when the underlying environment decides to terminate processes due to things like low memory.
            Note: Outside of *NIX systems, the behavior is unchanged.

            The binary itself is ~2.5MB per binary. There are 4 pre-compiled binaries (32 and 64bit versions for unix and darwin) while the memory footprint is ~800KB heavier than the shell wrapper.
            Here is a high-level breakdown of the memory footprint:
            Original version:
            two wrapper shell processes that are spawned, each process between 610-640KB
            a single 548KB sleep process used to poll the log output file
            Go binary version:
            a single golang binary ~2560KB

            Show
            carroll Carroll Chiou added a comment - Shell wrapper script has now been replaced by pre-compiled golang binaries (for Unix and Darwin). The binary allows the script to have its own session id and removes some polling overhead. This should increase the survivability of long-running scripts when jenkins terminates unexpectedly. This does not change the survivability of the script when the underlying environment decides to terminate processes due to things like low memory. Note: Outside of *NIX systems, the behavior is unchanged. The binary itself is ~2.5MB per binary. There are 4 pre-compiled binaries (32 and 64bit versions for unix and darwin) while the memory footprint is ~800KB heavier than the shell wrapper. Here is a high-level breakdown of the memory footprint: Original version: two wrapper shell processes that are spawned, each process between 610-640KB a single 548KB sleep process used to poll the log output file Go binary version: a single golang binary ~2560KB

              People

              • Assignee:
                carroll Carroll Chiou
                Reporter:
                kohsuke Kohsuke Kawaguchi
              • Votes:
                1 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: