Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58301

Kubernetes Plugin Repeated Socket Ping Timeout Exceptions

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: kubernetes-plugin
    • Labels:
    • Environment:
      Jenkins version: 2.182
      Kubernetes Plugin: 1.16.2
      Kubernetes: v1.13.7-eks-c57ff8
    • Similar Issues:

      Description

      I often get issues like this:
      java.net.SocketTimeoutException: sent ping but didn't receive pong within 1000ms (after 330 successful ping/pongs)

      One single issue breaks the entire task and makes it hard to even cancel the task. Should this not be retried rather than break execution? Our Jenkins will run longer running tasks as well. Any single task breaking stop in the middle is a real issue, and I don't see why one network issue after 330 successful ones (in this case) is such a big issue.

      x

        Attachments

          Issue Links

            Activity

            Hide
            deiwin Deiwin Sarjas added a comment -

            We've also been seeing quite a bit of this recently on EKS with Jenkins 2.164.2, Kubernetes plugin 1.15.2.

            Show
            deiwin Deiwin Sarjas added a comment - We've also been seeing quite a bit of this recently on EKS with Jenkins 2.164.2, Kubernetes plugin 1.15.2.
            Hide
            chengjingtao cheng jingtao added a comment -

            +1

            Show
            chengjingtao cheng jingtao added a comment - +1
            Hide
            tyrone_grech Tyrone Grech added a comment -

            We are also encountering this issue fairly often in our CI system running:

            • On premises Kubernetes cluster on version 1.14.1
            • Jenkins version 2.186
            • Kubernetes Plugin version 1.17.2
            Show
            tyrone_grech Tyrone Grech added a comment - We are also encountering this issue fairly often in our CI system running: On premises Kubernetes cluster on version 1.14.1 Jenkins version 2.186 Kubernetes Plugin version 1.17.2
            Hide
            deiwin Deiwin Sarjas added a comment -

            We configured -Dkubernetes.websocket.ping.interval=30000 for Jenkins based on this comment on another issue. I'll report back if it helps or not.

            Show
            deiwin Deiwin Sarjas added a comment - We configured -Dkubernetes.websocket.ping.interval=30000 for Jenkins based on this comment on another issue . I'll report back if it helps or not.
            Hide
            juhtie01 Juha Tiensyrjä added a comment -

            That option helped for us. But the reason why the pings started to fail was actually the JVM garbage collector, which caused the master to hang for more than 1 second. We switched from the default to G1GC to reduce time the master is blocked, and this helped with other timeouts too.

            Show
            juhtie01 Juha Tiensyrjä added a comment - That option helped for us. But the reason why the pings started to fail was actually the JVM garbage collector, which caused the master to hang for more than 1 second. We switched from the default to G1GC to reduce time the master is blocked, and this helped with other timeouts too.

              People

              • Assignee:
                Unassigned
                Reporter:
                autarchprinceps autarch princeps
              • Votes:
                5 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated: