Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53668

hudson.remoting.ChannelClosedException

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: docker-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.140
      Docker Plugin 1.1.5
      Docker version 18.06.1-ce, build e68fc7a
      Ubuntu 18.04.1 LTS
    • Similar Issues:

      Description

      We started using Docker Plugin 1.1.5 with Jenkins 2.140 and we are seeing this "channel closed down" message an almost every job.
      [WORK] Cannot contact docker-000cu0ya5igsh: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on docker-000cu0ya5igsh failed. The channel is closing down or has closed down
      It always feels like this is while a larger amount of work is being done in the container (for example running "mvn test" where we have thousands of tests).

       

        Attachments

          Issue Links

            Activity

            Hide
            pjdarton pjdarton added a comment -

            I've experienced this kind of issue where I work.  In our case, it was caused by the host running out of memory, triggering the oom-killer, and the oom-killer then decided that the Jenkins "java -jar slave.jar" process (the one responsible for keeping the slave connected to the master) was the least important process and killed it.

            The result was that, when things got busy, slaves died at random, despite doing nothing wrong themselves.

            This was particularly caused by our use of certain software packages that decide how much memory they're going to allocate to themselves based on the amount of memory available ... and that look at the whole host's memory instead of the container's fair share of that memory.  It doesn't take many processes to each allocate themselves half of the host's entire RAM before things get tight and the oom-killer gets invoked.

             

            Try turning off memory overcommit in your docker host, limiting the amount of memory available to each container, and limiting the number of containers you run concurrently.

            Show
            pjdarton pjdarton added a comment - I've experienced this kind of issue where I work.  In our case, it was caused by the host running out of memory, triggering the oom-killer, and the oom-killer then decided that the Jenkins "java -jar slave.jar" process (the one responsible for keeping the slave connected to the master) was the least important process and killed it. The result was that, when things got busy, slaves died at random, despite doing nothing wrong themselves. This was particularly caused by our use of certain software packages that decide how much memory they're going to allocate to themselves based on the amount of memory available ... and that look at the whole host's memory instead of the container's fair share of that memory.  It doesn't take many processes to each allocate themselves half of the host's entire RAM before things get tight and the oom-killer gets invoked.   Try turning off memory overcommit in your docker host, limiting the amount of memory available to each container, and limiting the number of containers you run concurrently.

              People

              • Assignee:
                Unassigned
                Reporter:
                mgreco2k Michael Greco
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: