Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48775

Proxy authentication error 407 even if 'check proxy' works

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core
    • Environment:
    • Similar Issues:

      Description

      Proxy configuration doesn't work even if 'check proxy' works! No workaround found.

      Situation: Fresh installed Windows Server 2016 with latest, fresh installed  LTS Jenkins version 2.89.2. Server is behind a proxy. Check of Jenkins proxy configuration says 'Ok' but checking plugins crashes jenkins (see attached screen shot).

      I have a second machine (Windows 7 enterprise) with the same Jenkins version, behind the same proxy, with (really!) the same proxy configuration: Check Jenkins proxy settings says: "Error 407", BUT: Checking plugins works! (WTF!?!) One differrence is the java runtime verision: on this machine = 1.8.0_66-b18.

        Attachments

          Issue Links

            Activity

            go Gerald Ortner created issue -
            Hide
            orrc Christopher Orr added a comment -

            From the release notes:

            Default the built-in Jenkins Update Center URL to https://updates.jenkins.io instead of obsolete HTTP endpoint. This requires a JRE compatible with Let's Encrypt, e.g. Oracle JRE 8u101.

            i.e. as you mention, the Java version makes a difference.

            Show
            orrc Christopher Orr added a comment - From the  release notes : Default the built-in Jenkins Update Center URL to https://updates.jenkins.io instead of obsolete HTTP endpoint. This requires a JRE compatible with Let's Encrypt , e.g. Oracle JRE 8u101. i.e. as you mention, the Java version makes a difference.
            orrc Christopher Orr made changes -
            Field Original Value New Value
            Status Open [ 1 ] Closed [ 6 ]
            Resolution Not A Defect [ 7 ]
            Hide
            go Gerald Ortner added a comment -

            Hi Christopher,

            at first thank you for your fast answer!
            Unfortunately, the situation on my both Jenkins servers is inverted: That one with JRE 1.8.0_66-b18 is working and the one with 1.8.0_144-b01 isn't working. Even the 'proxy check' is the other way around.

            bg Gerald

             

            Show
            go Gerald Ortner added a comment - Hi Christopher, at first thank you for your fast answer! Unfortunately, the situation on my both Jenkins servers is inverted: That one with JRE 1.8.0_66-b18 is working and the one with 1.8.0_144-b01 isn't working. Even the 'proxy check' is the other way around. bg Gerald  
            Hide
            go Gerald Ortner added a comment -

            due to the situation that the Jenkins Server with the newer JRE makes the troubles => still an issue.

             

            Show
            go Gerald Ortner added a comment - due to the situation that the Jenkins Server with the newer JRE makes the troubles => still an issue.  
            go Gerald Ortner made changes -
            Resolution Not A Defect [ 7 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            go Gerald Ortner made changes -
            Description Proxy configuration doesn't work even if 'check proxy' works! No workaround found.

            Situation: Fresh installed Windows Server 2016 with latest, fresh installed  LTS Jenkins version 2.89.2. Server is behind a proxy. Check of Jenkins proxy configuration says 'Ok' but checking plugins crashes jenkins (see attached screen shot).

            I have a second machine (Windows 7 enterprise) with the same Jenkins version, behind the same proxy, with (really!) the same proxy configuration: Check Jenkins proxy settings says: "Error 407", BUT: Checking plugins works! (WTF!?!) One differrence is the jave runtime verision: on this machine = 1.8.0_66-b18.
            Proxy configuration doesn't work even if 'check proxy' works! No workaround found.

            Situation: Fresh installed Windows Server 2016 with latest, fresh installed  LTS Jenkins version 2.89.2. Server is behind a proxy. Check of Jenkins proxy configuration says 'Ok' but checking plugins crashes jenkins (see attached screen shot).

            I have a second machine (Windows 7 enterprise) with the same Jenkins version, behind the same proxy, with (really!) the same proxy configuration: Check Jenkins proxy settings says: "Error 407", BUT: Checking plugins works! (WTF!?!) One differrence is the java runtime verision: on this machine = 1.8.0_66-b18.
            rtyler R. Tyler Croy made changes -
            Rank Ranked higher
            rtyler R. Tyler Croy made changes -
            Project Infrastructure [ 10301 ] Jenkins [ 10172 ]
            Key INFRA-1455 JENKINS-48775
            Workflow classic default workflow [ 224912 ] JNJira + In-Review [ 224949 ]
            Component/s core [ 15593 ]
            Component/s core [ 21134 ]
            Component/s update-center [ 22120 ]
            Hide
            go Gerald Ortner added a comment -

            Hi rtyler,

            I have new gained insights of this problem:

            After consulting out IT guys, it was clear that the Jenkins server DON'T send any authentication to the proxy (maybe except when pressing 'check proxy').
            So there were made two configurations on the company proxy server:
            1. added 'https://updates.jenkins.io' to the 'DoNotAuthenticate whitelist'
            2. turned off the 'SSL Interception' for this URL
            Now, with this workaround, checking and downloading plugin works! But that can't be the final solution!

            Thank you for your effort!

            BG Gerald

             

            Show
            go Gerald Ortner added a comment - Hi rtyler, I have new gained insights of this problem: After consulting out IT guys, it was clear that the Jenkins server  DON'T send any authentication to the proxy (maybe except when pressing 'check proxy'). So there were made two configurations on the company proxy server: 1. added ' https://updates.jenkins.io'  to the 'DoNotAuthenticate whitelist' 2. turned off the 'SSL Interception' for this URL Now, with this workaround, checking and downloading plugin works! But that can't be the final solution! Thank you for your effort! BG Gerald  
            Hide
            onodera Alexey Marin added a comment -

            I can confirm that I have the same problem with Jenkins 2.89.3 running on Oracle JRE 8u161 for x64 Linux: proxy configuration check succeeds, checking for plugins on Jenkins updates produces a stack trace about HTTP error 407.

            Show
            onodera Alexey Marin added a comment - I can confirm that I have the same problem with Jenkins 2.89.3 running on Oracle JRE 8u161 for x64 Linux: proxy configuration check succeeds, checking for plugins on Jenkins updates produces a stack trace about HTTP error 407.
            Hide
            ausername User Name added a comment - - edited

            Jenkins 2.101 here.
            Proxy Settings OK
            Update Center / Advanced  Proxy / Test OK
            wget from within the docker container (when setting the proxy there): OK
            wget from the host machine OK
            Trying to get the plugin update infos - fails.

            Proxy doesn't need an authentification.

            Show
            ausername User Name added a comment - - edited Jenkins 2.101 here. Proxy Settings OK Update Center / Advanced  Proxy / Test OK wget from within the docker container (when setting the proxy there): OK wget from the host machine OK Trying to get the plugin update infos - fails. Proxy doesn't need an authentification.
            Hide
            go Gerald Ortner added a comment -

            Now I saw the difference of fetching the updates: The machine with the older JRE version checks '[*http://updates.jenkins-ci.org/update-center.json*'] and the newer one (fresh installled Jenklins 2.89.2) calls '*https://updates.jenkins.io/update-center.json'*!

            So this exlains the different behavier but not what's going on in the newer Jenkins software!

            bg Gerald

            Show
            go Gerald Ortner added a comment - Now I saw the difference of fetching the updates: The machine with the older JRE version checks ' [*http://updates.jenkins-ci.org/update-center.json*']  and the newer one (fresh installled Jenklins 2.89.2) calls '* https://updates.jenkins.io/update-center.json'* ! So this exlains the different behavier but not what's going on in the newer Jenkins software! bg Gerald
            Hide
            ausername User Name added a comment -

            Jenkins Docker Container with V2.101 and Java 1.8.0.151 still an issue

            Show
            ausername User Name added a comment - Jenkins Docker Container with V2.101 and Java 1.8.0.151 still an issue
            Hide
            stephenconnolly Stephen Connolly added a comment -

            WORKAROUND:

            This is crazy, but will work...

            Run the following in the script console:

            ProxyConfiguration.open(new java.net.URL("http://jenkins.io")).content

            Then the update center should work... unclear why this works at this point in time!

            Show
            stephenconnolly Stephen Connolly added a comment - WORKAROUND: This is crazy, but will work... Run the following in the script console: ProxyConfiguration.open( new java.net.URL( "http: //jenkins.io" )).content Then the update center should work... unclear why this works at this point in time!
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Interestingly...

            ProxyConfiguration.open(new java.net.URL("https://jenkins.io")).content

            Will give the 

            java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"

            error... looks like a JDK bug whereby the HttpsURLConnectionImpl wherein it will fail to setup a connection to the proxy but will reuse an existing one

             

            Show
            stephenconnolly Stephen Connolly added a comment - Interestingly... ProxyConfiguration.open( new java.net.URL( "https: //jenkins.io" )).content Will give the  java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required" error... looks like a JDK bug whereby the HttpsURLConnectionImpl wherein it will fail to setup a connection to the proxy but will reuse an existing one  
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Ok, so Http URLs will populate the sun.net.www.protocol.http.AuthenticationInfo.cache.hashtable but Https URLs will not populate, only consult

            Show
            stephenconnolly Stephen Connolly added a comment - Ok, so Http URLs will populate the sun.net.www.protocol.http.AuthenticationInfo.cache.hashtable but Https URLs will not populate, only consult
            stephenconnolly Stephen Connolly made changes -
            Link This issue is related to JENKINS-42089 [ JENKINS-42089 ]
            Hide
            stephenconnolly Stephen Connolly added a comment -

            FYI the workaround may only last for a short-ish period of time as the cache seems to get purged under circumstances I have been unable to determine

            Show
            stephenconnolly Stephen Connolly added a comment - FYI the workaround may only last for a short-ish period of time as the cache seems to get purged under circumstances I have been unable to determine
            danielbeck Daniel Beck made changes -
            Labels 407 authentication proxy Proxy lts-candidate
            danielbeck Daniel Beck made changes -
            Assignee R. Tyler Croy [ rtyler ] Stephen Connolly [ stephenconnolly ]
            danielbeck Daniel Beck made changes -
            Status Reopened [ 4 ] Open [ 1 ]
            danielbeck Daniel Beck made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            danielbeck Daniel Beck made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            danielbeck Daniel Beck made changes -
            Remote Link This issue links to "PR 3283 (Web Link)" [ 20021 ]
            stephenconnolly Stephen Connolly made changes -
            Remote Link This issue links to "PR#3283 (Web Link)" [ 20022 ]
            stephenconnolly Stephen Connolly made changes -
            Remote Link This issue links to "PR#3283 (Web Link)" [ 20022 ]
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal CJP-8338 (Web Link)" [ 20156 ]
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal ANALYTICS-1279 (Web Link)" [ 20163 ]
            Hide
            israelromero Israel Romero Fijo added a comment -

            Same issue with 2.107.2 version 

            Show
            israelromero Israel Romero Fijo added a comment - Same issue with 2.107.2 version 
            Hide
            harrygg Harry G. added a comment -

            For us it is fixed, probably either by removal of a duplicate update center we had (same update center twice, disappeared after update and restart) or with the update to 2.107.3 (possibly related in the changelog: "Add new update center root CA certificate. (INFRA-1502)" ?)

            Show
            harrygg Harry G. added a comment - For us it is fixed, probably either by removal of a duplicate update center we had (same update center twice, disappeared after update and restart) or with the update to 2.107.3 (possibly related in the changelog: "Add new update center root CA certificate. ( INFRA-1502 )" ?)
            Hide
            jyanesancert Jorge Yanes added a comment - - edited

            Still happens to me with 2.121.3, behind a proxy that requires NTLM or basic auth. I am using basic auth, but the password expired. I updated my proxy settings and although the proxy test check succeeds, the plugin manager was getting a 407 when checking for updates. I issued  

            ProxyConfiguration.open(new java.net.URL("http://jenkins.io")).content
            

            and that refreshed the caches and it's now working; I guess it will work until the password expires again. EDIT: never mind, restarting jenkins causes the 407 again, even if the credentials are right.

            Please don't dismiss this bug yet. How about adding a simple "ping" to an http resource to force the cache refresh?

            Show
            jyanesancert Jorge Yanes added a comment - - edited Still happens to me with 2.121.3, behind a proxy that requires NTLM or basic auth. I am using basic auth, but the password expired. I updated my proxy settings and although the proxy test check succeeds, the plugin manager was getting a 407 when checking for updates. I issued   ProxyConfiguration.open( new java.net.URL( "http: //jenkins.io" )).content and that refreshed the caches and it's now working; I guess it will work until the password expires again. EDIT: never mind, restarting jenkins causes the 407 again, even if the credentials are right. Please don't dismiss this bug yet. How about adding a simple "ping" to an http resource to force the cache refresh?
            Hide
            fuku1026 Fukusuke Takahashi added a comment -

            Same issue with 2.138.2 version

            Show
            fuku1026 Fukusuke Takahashi added a comment - Same issue with 2.138.2 version
            Hide
            israelromero Israel Romero Fijo added a comment - - edited

            I respond to myself here to help anyone with same issue.

            The problem wasn't Jenkins. Problem comes from Java JDK8 and above. Since this version, Oracle has disabled basic auth tunnelling for Basic authentication. If you want to use Basic mode again (security will be compromised), execute Jenkins adding this parameter to your JAVA_OPTS variable:

               "-Djdk.http.auth.tunneling.disabledSchemes="

            default value is

              "-Djdk.http.auth.tunneling.disabledSchemes=Basic", so Basic Schema is DISABLED by default and will return HTTP 407 error even if your credentials were OK.

             Proxy checking must use other method to validate Internet connection...

            I hope this can help you, guys.

             

            Show
            israelromero Israel Romero Fijo added a comment - - edited I respond to myself here to help anyone with same issue. The problem wasn't Jenkins. Problem comes from Java JDK8 and above. Since this version, Oracle has disabled basic auth tunnelling for Basic authentication. If you want to use Basic mode again (security will be compromised), execute Jenkins adding this parameter to your JAVA_OPTS variable:    "-Djdk.http.auth.tunneling.disabledSchemes=" default value is   "-Djdk.http.auth.tunneling.disabledSchemes=Basic", so Basic Schema is DISABLED by default and will return HTTP 407 error even if your credentials were OK.  Proxy checking must use other method to validate Internet connection... I hope this can help you, guys.  
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Removing myself as assignee. My current work assignments do not provide sufficient bandwidth to review these issues and in the majority of cases I am only assigned by virtue of being the default assignee. For the credentials-api and scm-api related plugins I have permission to allocate time reviewing changes to these APIs themselves to ensure these APIs remain cohesive, but that can be handled through PR reviews rather than assigning issues in JIRA

            Show
            stephenconnolly Stephen Connolly added a comment - Removing myself as assignee. My current work assignments do not provide sufficient bandwidth to review these issues and in the majority of cases I am only assigned by virtue of being the default assignee. For the credentials-api and scm-api related plugins I have permission to allocate time reviewing changes to these APIs themselves to ensure these APIs remain cohesive, but that can be handled through PR reviews rather than assigning issues in JIRA
            stephenconnolly Stephen Connolly made changes -
            Assignee Stephen Connolly [ stephenconnolly ]
            danielbeck Daniel Beck made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            olivergondza Oliver Gondža made changes -
            Labels Proxy lts-candidate 2.150.1-rejected Proxy lts-candidate
            Hide
            olivergondza Oliver Gondža added a comment -

            Postponing backport until 2.150.2 at least to give it some soak time.

            Show
            olivergondza Oliver Gondža added a comment - Postponing backport until 2.150.2 at least to give it some soak time.
            danielbeck Daniel Beck made changes -
            Link This issue is related to JENKINS-54903 [ JENKINS-54903 ]
            Hide
            danielbeck Daniel Beck added a comment -

            This introduced JENKINS-54903.

            Show
            danielbeck Daniel Beck added a comment - This introduced JENKINS-54903 .
            olivergondza Oliver Gondža made changes -
            Labels 2.150.1-rejected Proxy lts-candidate 2.150.1-rejected 2.150.2-rejected Proxy lts-candidate
            olivergondza Oliver Gondža made changes -
            Labels 2.150.1-rejected 2.150.2-rejected Proxy lts-candidate 2.150.1-rejected 2.150.2-rejected 2.150.3-rejected Proxy lts-candidate
            olivergondza Oliver Gondža made changes -
            Labels 2.150.1-rejected 2.150.2-rejected 2.150.3-rejected Proxy lts-candidate 2.150.1-rejected 2.150.2-rejected 2.150.3-rejected Proxy

              People

              • Assignee:
                Unassigned
                Reporter:
                go Gerald Ortner
              • Votes:
                7 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: