Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47279

Full-duplex HTTP(S) transport with plain CLI protocol does not work with Apache reverse proxy in Jenkins >= 2.54

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: cli, core
    • Labels:
    • Environment:
      Jenkins >= 2.54 (Tested on 2.78 and 2.82)
      Apache 2.2.22 and 2.4.22
      Ubuntu 12.04
    • Similar Issues:

      Description

      I've been testing various reverse proxies with Jenkins, and am unable to use the CLI over an Apache reverse proxy using mod_proxy with the new full-duplex HTTP transport and the plain CLI protocol that was introduced in Jenkins 2.54. Both Nginx and HAProxy work using the full duplex HTTP transport correctly. Accessing Jenkins through my browser over the Apache proxy works correctly.

      I used Wireshark to see what was happening, and the issue appears to be that Apache tries to buffer the body of the upload connection before proxying it to Jenkins, so Jenkins never sees the upload connection. As far as I can tell, the current code for FullDuplexHttpService and FullDuplexHttpStream require that both connections are unbuffered. The relevant setting in Apache has a minimum value of 512, and neither SetEnv proxy-sendchunks 1 nor SetEnv proxy-sendchunked 1 have a noticeable effect.

      I am testing using https://github.com/jglick/jenkins-demo-reverse-proxy, and attempting to run the following CLI command:

      java -jar jenkins-cli.jar -s "http://localhost:80/jenkins|http://localhost/jenkins" -logger FINE help
      

      This is the error that is thrown by Jenkins 2.82:

      WARNING: null
      java.io.IOException: HTTP full-duplex channel timeout: 829c689a-05cb-4e9a-a559-eeed9c099460
      	at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:108)
      	at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:175)
      

      Assuming that Jenkins' full-duplex HTTP connection is spec compliant, this is arguably a bug in Apache.

        Attachments

          Issue Links

            Activity

            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            As a workaround, you can use the CLI over SSH, or use Nginx or HAProxy for your reverse proxy server.

            Show
            dnusbaum Devin Nusbaum added a comment - - edited As a workaround, you can use the CLI over SSH , or use Nginx or HAProxy for your reverse proxy server.
            Hide
            jglick Jesse Glick added a comment -

            Search JIRA for other issues mentioning HTTP full-duplex channel timeout and link them if appropriate.

            Show
            jglick Jesse Glick added a comment - Search JIRA for other issues mentioning HTTP full-duplex channel timeout and link them if appropriate.
            Hide
            jglick Jesse Glick added a comment -

            Assuming that Jenkins' full-duplex HTTP connection is spec compliant

            I doubt it is.

            Show
            jglick Jesse Glick added a comment - Assuming that Jenkins' full-duplex HTTP connection is spec compliant I doubt it is.
            Hide
            bitwiseman Liam Newman added a comment -

            This has been added to the docs.  Reverse this PR when this issue is fixed. 

            Show
            bitwiseman Liam Newman added a comment - This has been added to the docs.  Reverse this PR when this issue is fixed. 
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Liam Newman
            Path:
            content/doc/book/managing/cli.adoc
            http://jenkins-ci.org/commit/jenkins.io/b00d1424637c32b430423479e6ea5d71d5b28e3a
            Log:
            Merge pull request #1164 from dwnusbaum/JENKINS-47279

            JENKINS-47279 Add warning for HTTP mode CLI with Apache reverse proxies

            Compare: https://github.com/jenkins-infra/jenkins.io/compare/0c002fbba7c8...b00d1424637c

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Liam Newman Path: content/doc/book/managing/cli.adoc http://jenkins-ci.org/commit/jenkins.io/b00d1424637c32b430423479e6ea5d71d5b28e3a Log: Merge pull request #1164 from dwnusbaum/ JENKINS-47279 JENKINS-47279 Add warning for HTTP mode CLI with Apache reverse proxies Compare: https://github.com/jenkins-infra/jenkins.io/compare/0c002fbba7c8...b00d1424637c
            Hide
            jglick Jesse Glick added a comment -

            If we want to maintain the CLI system in the long term (as opposed to guiding users toward REST), then we could offer the “plain” protocol over WebSocket transport rather than the full-duplex HTTP hack.

            Show
            jglick Jesse Glick added a comment - If we want to maintain the CLI system in the long term (as opposed to guiding users toward REST), then we could offer the “plain” protocol over WebSocket transport rather than the full-duplex HTTP hack.
            Hide
            ringerc Craig Ringer added a comment - - edited

            I've seen reports this also affects nginx with proxy_request_buffering and/or proxy_buffering.

            This makes me wonder if changes to mod_proxy's ProxyIOBufferSize and/or ProxyReceiveBufferSize may address this, perhaps in combination with ensuring jenkins-cli sends a certain minimum size request/response.

            Show
            ringerc Craig Ringer added a comment - - edited I've seen reports this also affects nginx with proxy_request_buffering and/or proxy_buffering . This makes me wonder if changes to mod_proxy 's ProxyIOBufferSize and/or ProxyReceiveBufferSize may address this, perhaps in combination with ensuring jenkins-cli sends a certain minimum size request/response.
            Hide
            dnusbaum Devin Nusbaum added a comment -

            Craig Ringer Yes, in Nginx you must use proxy_request_buffering off; and sometimes proxy_buffering off;, but Apache does not have any such feature as far as I can tell. Changes to mod_proxy could probably fix the issue, but I think you would end up having to provide a way to stop buffering requests/responses completely, because IIRC the plain CLI protocol sends only a single byte before waiting for a response in some cases.

            Show
            dnusbaum Devin Nusbaum added a comment - Craig Ringer Yes, in Nginx you must use  proxy_request_buffering off; and sometimes proxy_buffering off; , but Apache does not have any such feature as far as I can tell. Changes to mod_proxy could probably fix the issue, but I think you would end up having to provide a way to stop buffering requests/responses completely, because IIRC the plain CLI protocol sends only a single byte before waiting for a response in some cases.

              People

              • Assignee:
                Unassigned
                Reporter:
                dnusbaum Devin Nusbaum
              • Votes:
                7 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated: