Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49119

Support suppression of SSH server inactivity timeouts

    Details

    • Type: Improvement
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Labels:
      None
    • Environment:
      Jenkins 2.89.3
      Publish Over SSH 1.18
      CentOS Linux 7.3
    • Similar Issues:

      Description

      Our team runs a lot of Jenkins jobs using Publish over SSH. Many of them use long-running SQL queries and all timeouts in Publish over SSH have been disabled. However, we are running into server SSH inactivity timeouts, set as ClientAliveInterval 1500 in /etc/ssh/sshd_config for security reasons.

      One workaround we tried is to configure server-alives on a client side of the Jenkins server in ~/.ssh/config of the Jenkins user:

      Host *
      ServerAliveInterval 120
      ServerAliveCountMax 30
      ConnectTimeout 30
      

      This approach did not work. What worked is a wait/print loop in every remote script that needs to be executed. It works like this:

      # Run the main script
      Technical/Template_Job/template.sh &
      pid=$!
      
      # Wait on a background job completion. Query status every 10 minutes.
      declare -i elapsed=0
      # `ps -p ${pid}` works on macOS and CentOS. On both OSes `ps ${pid}` works as well.
      while ps -p ${pid} >/dev/null; do
        sleep 1
        if (( ++elapsed % 600 == 0 )); then
          echo "Waiting for the completion of the main script. $((elapsed / 60))m and counting ..."
        fi
      done
      
      # Return the exit code of the terminated background process. This works in Bash 4.4 despite what Bash docs say:
      # "If neither jobspec nor pid specifies an active child process of the shell, the return status is 127."
      wait ${pid}
      

      This works, but it's a hassle to be adding wait/print loops into each script we execute. It'd be very nice if Publish over SSH supports periodic server-alive messages or some other option to suppress server inactivity timeouts.

        Attachments

          Activity

          Hide
          slide_o_mix Alex Earl added a comment -

          So, would this be something like a keep alive?

          Show
          slide_o_mix Alex Earl added a comment - So, would this be something like a keep alive?
          Hide
          dk Dmitriy Korobskiy added a comment -

          I guess. SSH clients send server-alive messages and can suppress automatic inactivity termination if configured so. My SSH clients are configured correspondingly and it does work with our servers, but our Jenkins jobs are out of luck.

          Show
          dk Dmitriy Korobskiy added a comment - I guess. SSH clients send server-alive messages and can suppress automatic inactivity termination if configured so. My SSH clients are configured correspondingly and it does work with our servers, but our Jenkins jobs are out of luck.

            People

            • Assignee:
              Unassigned
              Reporter:
              dk Dmitriy Korobskiy
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: