Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52166

Jenkins unresponsive when using native https

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: core, winstone-jetty
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      We are trying to run jenkins using https as described here : https://wiki.jenkins.io/pages/viewpage.action?pageId=135468777

       At first it appeared to work properly, but now the Jenkins web server is unresponsive. Https requests timeout, both when accessing the web pages through a browser and using the http API from other tools.

      We tried taking out the https related java parameters and it appears to work with http only.

      However, when using both http and https on different ports, Jenkins is unresponsive on both ports.

       

      Jenkins does not appear to be running out of memory. It is running dozens of threads but they don't seem to be consuming much CPU time.

      Looking at the thread dump does not seem to indicate a deadlock, at least some thread are still running.

      Furthermore, we run builds every night and they appear to be running properly. However, since we use http requests to fetch the logs, we have little more then the build result to go on.

      Outside of Jenkins, our build is not very stable but has comparable stability to what Jenkins seems to be reporting.

       

      Update 1:

      Jenkins version is 2.121.1

       

      Update 2:

      Looking at various logs. It appears that:

       - Native Https was working on May 29, which is when we started using https instead of http.

       - Jenkins was Upgraded from version 2.107.3 to version 2.121.1 on June 07.

       - We observed that Jenkins was unresponsive on June 08.

       

      Update 3:

      After reverting Jenkins to version  2.107.3,

      I can confirm that Native Https support was broken in version 2.121.1.

        Attachments

          Activity

          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Ugh, I didn't know that we have documentation for it. I've always thought that we support HTTPS only via external services.
          Thierry Delisle which Jenkins version do you use?

          Show
          oleg_nenashev Oleg Nenashev added a comment - Ugh, I didn't know that we have documentation for it. I've always thought that we support HTTPS only via external services. Thierry Delisle which Jenkins version do you use?
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          The wiki page (https://wiki.jenkins.io/pages/viewpage.action?pageId=135468777) has been created in 2017 by Jeff Thornsen and then edited by joey piccola.

          Olivier Lamy Daniel Beck WDYT? Should actually support it? Or should we add a kind of disclaimer to the Wiki page ("available but not supported" or so).

          Show
          oleg_nenashev Oleg Nenashev added a comment - The wiki page ( https://wiki.jenkins.io/pages/viewpage.action?pageId=135468777 ) has been created in 2017 by Jeff Thornsen and then edited by joey piccola . Olivier Lamy Daniel Beck WDYT? Should actually support it? Or should we add a kind of disclaimer to the Wiki page ("available but not supported" or so).
          Hide
          jthornsen Jeff Thornsen added a comment -

          I wrote those instructions back when I was required to deploy Jenkins over HTTPS on Windows for one of my customers.  This was on a network without internet access, and they forced me to use Windows, so I figured out a way to make the Jenkins embedded webserver listen over HTTPS on port 443 natively and documented it.  Normally I would just deploy Jenkins on Linux and throw haproxy or nginx or apache httpd in front to handle the SSL.

          AFAIK that server had no issues with the web service going down.  It is likely running Jenkins LTS 2.60.3 or maybe something slightly earlier than that.

          I do notice in the the Jenkins LTS Changelog that Jenkins LTS 2.73 introduced a new version of Winstone and again in 2.121.1 so it's possible the different Winstone/Jetty versions are causing the issue?

          Show
          jthornsen Jeff Thornsen added a comment - I wrote those instructions back when I was required to deploy Jenkins over HTTPS on Windows for one of my customers.  This was on a network without internet access, and they forced me to use Windows, so I figured out a way to make the Jenkins embedded webserver listen over HTTPS on port 443 natively and documented it.  Normally I would just deploy Jenkins on Linux and throw haproxy or nginx or apache httpd in front to handle the SSL. AFAIK that server had no issues with the web service going down.  It is likely running Jenkins LTS 2.60.3 or maybe something slightly earlier than that. I do notice in the the Jenkins LTS Changelog that Jenkins LTS 2.73 introduced a new version of Winstone and again in 2.121.1 so it's possible the different Winstone/Jetty versions are causing the issue?
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Thanks Jeff Thornsen!

          Yes, it may be related to Jetty upgrades. That's why I CCed Olivier Lamy. We do not have test automation for such configuration, so it's hard to say when particularly it became unstable (if it is a wide issue unrelated to the reporter's configuration).

          Show
          oleg_nenashev Oleg Nenashev added a comment - Thanks Jeff Thornsen ! Yes, it may be related to Jetty upgrades. That's why I CCed Olivier Lamy . We do not have test automation for such configuration, so it's hard to say when particularly it became unstable (if it is a wide issue unrelated to the reporter's configuration).
          Hide
          tdelisle Thierry Delisle added a comment -

          I would like to add that using native HTTPS is much more convenient in our case. 

          We run Jenkins on a university network, therefore it seems to me that adding a proxy in front of Jenkins won't prevent insecure connections. 

          Show
          tdelisle Thierry Delisle added a comment - I would like to add that using native HTTPS is much more convenient in our case.  We run Jenkins on a university network, therefore it seems to me that adding a proxy in front of Jenkins won't prevent insecure connections. 
          Hide
          tdelisle Thierry Delisle added a comment -

          To add to Jeff Thornsen's hypothesis, it appears that the problem appear immediately after upgrading to 2.121.1, and was not present before.

          We will try to revert to the previous version and see if the problem goes away.

          Show
          tdelisle Thierry Delisle added a comment - To add to Jeff Thornsen 's hypothesis, it appears that the problem appear immediately after upgrading to 2.121.1, and was not present before. We will try to revert to the previous version and see if the problem goes away.
          Hide
          danielbeck Daniel Beck added a comment - - edited

          therefore it seems to me that adding a proxy in front of Jenkins won't prevent insecure connections

          https://github.com/jenkinsci/winstone/#command-line-options has httpListenAddress, make that 127.0.0.1.

          Show
          danielbeck Daniel Beck added a comment - - edited therefore it seems to me that adding a proxy in front of Jenkins won't prevent insecure connections https://github.com/jenkinsci/winstone/#command-line-options has httpListenAddress, make that 127.0.0.1.

            People

            • Assignee:
              Unassigned
              Reporter:
              tdelisle Thierry Delisle
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: