Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12875

"No valid crumb was included in the request" errors all around

    Details

    • Similar Issues:

      Description

      After upgrading Jenkins from 1.448 to 1.451 (with yum, upgraded plugins too) several parts of Jenkins have become unusable with error "No valid crumb was included in the request":

      Console output

      • When building, the log is unavailable, see console_log.png
      • Once the build is done, the complete log is available, but the same error appears on the side console_side.png

      Build configuration

      • Builds cannot be launched anymore from the build page (The firebug error console displays the "No valid crumb was included in the request" error). They still can be launched from the dashboard though.
      • Cannot add build steps (when choosing a build step type an error is reported in background), see add_build_step.png

      Manage Jenkins configuration

      • The page loads only partially (first half is permanently greyed out, see configuration.png). The firebug error console displays the "No valid crumb was included in the request" error.

      Manage Jenkins plugins

      • Cannot upgrade nor downgrade, see error shown plugins.png

        Attachments

        1. add_build_step.png
          add_build_step.png
          98 kB
        2. configuration.png
          configuration.png
          85 kB
        3. console_log.png
          console_log.png
          105 kB
        4. console_side.png
          console_side.png
          202 kB
        5. plugins.png
          plugins.png
          92 kB
        6. Screen Shot 2014-03-12 at 10.43.58 AM.png
          Screen Shot 2014-03-12 at 10.43.58 AM.png
          111 kB

          Issue Links

            Activity

            naloy Núria Aloy created issue -
            naloy Núria Aloy made changes -
            Field Original Value New Value
            Description After upgrading Jenkins from 1.448 to 1.451 (with yum, upgraded plugins too) several parts of Jenkins have become unusable with error "No valid crumb was included in the request":

            h4. Console output

            * When building, the log is unavailable, see !console_log.png!
            * Once the build is done, the complete log is available, but the same error appears on the side !console_side.png!

            h4. Build configuration

            * Builds cannot be launched anymore from the build page (The firebug error console displays the "No valid crumb was included in the request" error). They still can be launched from the dashboard though.
            * Cannot add build steps (when choosing a build step type an error is reported in background), see !add_build_step.png!

            h4. Manage Jenkins configuration

            * The page loads only partially (first half is permanently greyed out, see !configuration.log!). The firebug error console displays the "No valid crumb was included in the request" error.

            h4. Manage Jenkins plugins

            * Cannot upgrade nor downgrade, see error shown !plugins.png!
            After upgrading Jenkins from 1.448 to 1.451 (with yum, upgraded plugins too) several parts of Jenkins have become unusable with error "No valid crumb was included in the request":

            h4. Console output

            * When building, the log is unavailable, see [^console_log.png]
            * Once the build is done, the complete log is available, but the same error appears on the side [^console_side.png]

            h4. Build configuration

            * Builds cannot be launched anymore from the build page (The firebug error console displays the "No valid crumb was included in the request" error). They still can be launched from the dashboard though.
            * Cannot add build steps (when choosing a build step type an error is reported in background), see [^add_build_step.png]

            h4. Manage Jenkins configuration

            * The page loads only partially (first half is permanently greyed out, see [^configuration.log]). The firebug error console displays the "No valid crumb was included in the request" error.

            h4. Manage Jenkins plugins

            * Cannot upgrade nor downgrade, see error shown [^plugins.png]
            naloy Núria Aloy made changes -
            Description After upgrading Jenkins from 1.448 to 1.451 (with yum, upgraded plugins too) several parts of Jenkins have become unusable with error "No valid crumb was included in the request":

            h4. Console output

            * When building, the log is unavailable, see [^console_log.png]
            * Once the build is done, the complete log is available, but the same error appears on the side [^console_side.png]

            h4. Build configuration

            * Builds cannot be launched anymore from the build page (The firebug error console displays the "No valid crumb was included in the request" error). They still can be launched from the dashboard though.
            * Cannot add build steps (when choosing a build step type an error is reported in background), see [^add_build_step.png]

            h4. Manage Jenkins configuration

            * The page loads only partially (first half is permanently greyed out, see [^configuration.log]). The firebug error console displays the "No valid crumb was included in the request" error.

            h4. Manage Jenkins plugins

            * Cannot upgrade nor downgrade, see error shown [^plugins.png]
            After upgrading Jenkins from 1.448 to 1.451 (with yum, upgraded plugins too) several parts of Jenkins have become unusable with error "No valid crumb was included in the request":

            h4. Console output

            * When building, the log is unavailable, see [^console_log.png]
            * Once the build is done, the complete log is available, but the same error appears on the side [^console_side.png]

            h4. Build configuration

            * Builds cannot be launched anymore from the build page (The firebug error console displays the "No valid crumb was included in the request" error). They still can be launched from the dashboard though.
            * Cannot add build steps (when choosing a build step type an error is reported in background), see [^add_build_step.png]

            h4. Manage Jenkins configuration

            * The page loads only partially (first half is permanently greyed out, see [^configuration.png]). The firebug error console displays the "No valid crumb was included in the request" error.

            h4. Manage Jenkins plugins

            * Cannot upgrade nor downgrade, see error shown [^plugins.png]
            valentine Maarten van den Ende made changes -
            Link This issue is related to JENKINS-12787 [ JENKINS-12787 ]
            Hide
            naloy Núria Aloy added a comment -

            After yet another restart things seems to be back to normal (???!!)

            Show
            naloy Núria Aloy added a comment - After yet another restart things seems to be back to normal (???!!)
            naloy Núria Aloy made changes -
            Priority Blocker [ 1 ] Major [ 3 ]
            Hide
            valentine Maarten van den Ende added a comment -

            Not completely normal. The loading thing in the management section is still happening

            Show
            valentine Maarten van den Ende added a comment - Not completely normal. The loading thing in the management section is still happening
            Hide
            naloy Núria Aloy added a comment -

            In the Jenkins configuration page:

            Error: Form.findMatchingInput is not a function
            Source File: http://jenkins.rightnow.com:8080/static/6dcd23a9/scripts/hudson-behavior.js
            Line: 338

            Show
            naloy Núria Aloy added a comment - In the Jenkins configuration page: Error: Form.findMatchingInput is not a function Source File: http://jenkins.rightnow.com:8080/static/6dcd23a9/scripts/hudson-behavior.js Line: 338
            Hide
            windyroad windyroad added a comment -

            I'm not as blocked as the reporting user, but I'm having the same problem (no valid crumb..). In my case it only seems to affect the console output, though the error can be seen for "ajaxBuildQueue" and "ajaxExecuters" on the config page, the home page (and I'm assuming others).

            What's interesting is that I only get this problem through a reverse proxy. http://ve.mbqblqmj.vesrv.com:8080/ works fine, but http://ci.windyroad.org/ (which is RPed to http://ve.mbqblqmj.vesrv.com:8080/) does not.

            Show
            windyroad windyroad added a comment - I'm not as blocked as the reporting user, but I'm having the same problem (no valid crumb..). In my case it only seems to affect the console output, though the error can be seen for "ajaxBuildQueue" and "ajaxExecuters" on the config page, the home page (and I'm assuming others). What's interesting is that I only get this problem through a reverse proxy. http://ve.mbqblqmj.vesrv.com:8080/ works fine, but http://ci.windyroad.org/ (which is RPed to http://ve.mbqblqmj.vesrv.com:8080/ ) does not.
            Hide
            windyroad windyroad added a comment -

            I've offered USD100.00 at FreedomSponsors to have this fixed (http://www.freedomsponsors.org/core/issue/31/no-valid-crumb-was-included-in-the-request-errors-all-around).
            Once you solve it (according to the acceptance criteria described there), just create a FreedomSponsors account and mark it as resolved (oh, you'll need a Paypal account too)
            I'll then check it out and will gladly pay.

            Show
            windyroad windyroad added a comment - I've offered USD100.00 at FreedomSponsors to have this fixed ( http://www.freedomsponsors.org/core/issue/31/no-valid-crumb-was-included-in-the-request-errors-all-around ). Once you solve it (according to the acceptance criteria described there), just create a FreedomSponsors account and mark it as resolved (oh, you'll need a Paypal account too) I'll then check it out and will gladly pay.
            Hide
            onemanbucket onemanbucket added a comment -

            I reproduced the problem with a simple nginx reverse proxy and csrf enabled.

            The problem is that jenkins stores its' csrf token in a http header called '.crumb', AFAIK headers must only contain alphanumerics and dashes, and nginx will remove invalid headers from the request (unless configured not to).

            Sooo, a workaround is to use ignore_invalid_headers in nginx or disable csrf protection until patched.

            I'll post a pull request shortly

            Show
            onemanbucket onemanbucket added a comment - I reproduced the problem with a simple nginx reverse proxy and csrf enabled. The problem is that jenkins stores its' csrf token in a http header called '.crumb', AFAIK headers must only contain alphanumerics and dashes, and nginx will remove invalid headers from the request (unless configured not to). Sooo, a workaround is to use ignore_invalid_headers in nginx or disable csrf protection until patched. I'll post a pull request shortly
            Hide
            onemanbucket onemanbucket added a comment -

            Actually, it seems that you can set the header name with a system property to jenkins (hudson.security.csrf.requestfield). So another workaround could be to start jenkins with -Dhudson.security.csrf.requestfield=crumb

            Show
            onemanbucket onemanbucket added a comment - Actually, it seems that you can set the header name with a system property to jenkins (hudson.security.csrf.requestfield). So another workaround could be to start jenkins with -Dhudson.security.csrf.requestfield=crumb
            Hide
            onemanbucket onemanbucket added a comment -
            Show
            onemanbucket onemanbucket added a comment - Pull request: https://github.com/jenkinsci/jenkins/pull/596
            Hide
            algob Michael Decker added a comment - - edited

            I've same issue, if I try to disable a project (use Jenkins v1.502):
            http://rhodan:8080/jenkins/job/myproject_3562/disable

            It looks like, I would have the same issue, if I would try to enable it, if it is disabled: JENKINS-14501

            I tried to added these line to /opt/tomcat/bin/catalina.sh, but it didn't solve the issue:
            — SNIP —
            JAVA_OPTS="$JAVA_OPTS -Dhudson.security.csrf.requestfield=crumb"
            — SNAP —

            Show
            algob Michael Decker added a comment - - edited I've same issue, if I try to disable a project (use Jenkins v1.502): http://rhodan:8080/jenkins/job/myproject_3562/disable It looks like, I would have the same issue, if I would try to enable it, if it is disabled: JENKINS-14501 I tried to added these line to /opt/tomcat/bin/catalina.sh, but it didn't solve the issue: — SNIP — JAVA_OPTS="$JAVA_OPTS -Dhudson.security.csrf.requestfield=crumb" — SNAP —
            algob Michael Decker made changes -
            Link This issue is related to JENKINS-14501 [ JENKINS-14501 ]
            Hide
            ozawa Tsuyoshi Ozawa added a comment -

            Today, I've had also the same issue. Delete jobs or clone from github hasn't worked correctly.
            This problem is caused after upgrading 1.515.

            Show
            ozawa Tsuyoshi Ozawa added a comment - Today, I've had also the same issue. Delete jobs or clone from github hasn't worked correctly. This problem is caused after upgrading 1.515.
            Hide
            johnnygoods John Goodwin added a comment -

            Also seeing this issue when using jenkins 1.517 behind nginx reverse proxy. Setting nginx ignore_invalid_headers had no effect. Disabling CSRF did.

            Show
            johnnygoods John Goodwin added a comment - Also seeing this issue when using jenkins 1.517 behind nginx reverse proxy. Setting nginx ignore_invalid_headers had no effect. Disabling CSRF did.
            Hide
            vadakr Ranjit Vadakkan added a comment - - edited

            To disable CSRF (Screenshot) -

            Jenkins > Manage Jenkins > Configure Global Security

            At the very bottom, you will see a checkbox for "Prevent Cross Site Request Forgery exploits". Uncheck this and apply.

            Show
            vadakr Ranjit Vadakkan added a comment - - edited To disable CSRF ( Screenshot ) - Jenkins > Manage Jenkins > Configure Global Security At the very bottom, you will see a checkbox for "Prevent Cross Site Request Forgery exploits". Uncheck this and apply.
            vadakr Ranjit Vadakkan made changes -
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-7518 [ JENKINS-7518 ]
            Hide
            daemoen Marc Mercer added a comment - - edited

            Tested multiple scenarios.

            Following configs work:
            OS: Centos 6 x86_64
            Jenkins: 1.561 from Jenkins Repo

            NGINX Reverse Proxy Setup test #1 (works):
            /etc/nginx/conf.d/jenkins.conf

            #
            # Jenkins over SSL
            #
            
            upstream jenkins {
              server 127.0.0.1:8080 fail_timeout=0;
            }
            
            server {
              listen 80;
              server_name jenkins.domain.com;
              return 301 https://$host$request_uri;
            }
            
            server {
              ignore_invalid_headers off;
              listen 443;
              server_name jenkins.domain.com;
            
              ssl on;
              ssl_certificate /etc/nginx/ssl/wildcard.domain.chained.crt;
              ssl_certificate_key /etc/nginx/ssl/wildcard.domain.key;
            
              location / {
                proxy_set_header        Host $host;
                proxy_set_header        X-Real-IP $remote_addr;
                proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_redirect http:// https://;
                proxy_pass              http://jenkins;
              }
            }
            

            NGINX Reverse Proxy Setup test #2 (does NOT work):
            /etc/nginx/conf.d/jenkins.conf

            #
            # Jenkins over SSL
            #
            
            upstream jenkins {
              server 127.0.0.1:8080 fail_timeout=0;
            }
            
            server {
              listen 80;
              server_name jenkins.domain.com;
              return 301 https://$host$request_uri;
            }
            
            server {
              ignore_invalid_headers on; # Or simply leave it unspecified; This is the DEFAULT behavior of ignore_invalid_headers
              listen 443;
              server_name jenkins.domain.com;
            
              ssl on;
              ssl_certificate /etc/nginx/ssl/wildcard.domain.chained.crt;
              ssl_certificate_key /etc/nginx/ssl/wildcard.domain.key;
            
              location / {
                proxy_set_header        Host $host;
                proxy_set_header        X-Real-IP $remote_addr;
                proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_redirect http:// https://;
                proxy_pass              http://jenkins;
              }
            }
            

            NGINX Reverse Proxy Setup test #3 (works):
            /etc/nginx/conf.d/jenkins.conf

            #
            # Jenkins over SSL
            #
            
            upstream jenkins {
              server 127.0.0.1:8080 fail_timeout=0;
            }
            
            server {
              listen 80;
              server_name jenkins.domain.com;
              return 301 https://$host$request_uri;
            }
            
            server {
              ignore_invalid_headers on;
              listen 443;
              server_name jenkins.domain.com;
            
              ssl on;
              ssl_certificate /etc/nginx/ssl/wildcard.domain.chained.crt;
              ssl_certificate_key /etc/nginx/ssl/wildcard.domain.key;
            
              location / {
                proxy_set_header        Host $host;
                proxy_set_header        X-Real-IP $remote_addr;
                proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_redirect http:// https://;
                proxy_pass              http://jenkins;
              }
            }
            

            /etc/sysconfig/jenkins

            JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Dhudson.security.csrf.requestfield=crumb"
            
            Show
            daemoen Marc Mercer added a comment - - edited Tested multiple scenarios. Following configs work: OS: Centos 6 x86_64 Jenkins: 1.561 from Jenkins Repo NGINX Reverse Proxy Setup test #1 (works): /etc/nginx/conf.d/jenkins.conf # # Jenkins over SSL # upstream jenkins { server 127.0.0.1:8080 fail_timeout=0; } server { listen 80; server_name jenkins.domain.com; return 301 https: //$host$request_uri; } server { ignore_invalid_headers off; listen 443; server_name jenkins.domain.com; ssl on; ssl_certificate /etc/nginx/ssl/wildcard.domain.chained.crt; ssl_certificate_key /etc/nginx/ssl/wildcard.domain.key; location / { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_redirect http: // https://; proxy_pass http: //jenkins; } } NGINX Reverse Proxy Setup test #2 (does NOT work): /etc/nginx/conf.d/jenkins.conf # # Jenkins over SSL # upstream jenkins { server 127.0.0.1:8080 fail_timeout=0; } server { listen 80; server_name jenkins.domain.com; return 301 https: //$host$request_uri; } server { ignore_invalid_headers on; # Or simply leave it unspecified; This is the DEFAULT behavior of ignore_invalid_headers listen 443; server_name jenkins.domain.com; ssl on; ssl_certificate /etc/nginx/ssl/wildcard.domain.chained.crt; ssl_certificate_key /etc/nginx/ssl/wildcard.domain.key; location / { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_redirect http: // https://; proxy_pass http: //jenkins; } } NGINX Reverse Proxy Setup test #3 (works): /etc/nginx/conf.d/jenkins.conf # # Jenkins over SSL # upstream jenkins { server 127.0.0.1:8080 fail_timeout=0; } server { listen 80; server_name jenkins.domain.com; return 301 https: //$host$request_uri; } server { ignore_invalid_headers on; listen 443; server_name jenkins.domain.com; ssl on; ssl_certificate /etc/nginx/ssl/wildcard.domain.chained.crt; ssl_certificate_key /etc/nginx/ssl/wildcard.domain.key; location / { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_redirect http: // https://; proxy_pass http: //jenkins; } } /etc/sysconfig/jenkins JENKINS_JAVA_OPTIONS= "-Djava.awt.headless= true -Dhudson.security.csrf.requestfield=crumb"
            Hide
            daemoen Marc Mercer added a comment -

            Further testing of this shows that even with the working setups above, chrome still does not like it. Firefox works fine.

            Show
            daemoen Marc Mercer added a comment - Further testing of this shows that even with the working setups above, chrome still does not like it. Firefox works fine.
            Hide
            davea_br Dave Augustus added a comment -

            I can confirm that this also occurs on Centos 6.5 with Nginx 1.6 using Firefox.

            Show
            davea_br Dave Augustus added a comment - I can confirm that this also occurs on Centos 6.5 with Nginx 1.6 using Firefox.
            danielbeck Daniel Beck made changes -
            Link This issue is duplicated by JENKINS-24249 [ JENKINS-24249 ]
            danielbeck Daniel Beck made changes -
            Link This issue is duplicated by JENKINS-23793 [ JENKINS-23793 ]
            Hide
            jglick Jesse Glick added a comment -

            Be warned that -Dhudson.security.csrf.requestfield=crumb will cause the attempts by some REST API clients to obtain a crumb to be rejected by the fix of SECURITY-47, as crumb is a valid JavaScript identifier that would open you to attack.

            Show
            jglick Jesse Glick added a comment - Be warned that -Dhudson.security.csrf.requestfield=crumb will cause the attempts by some REST API clients to obtain a crumb to be rejected by the fix of SECURITY-47, as crumb is a valid JavaScript identifier that would open you to attack.
            jglick Jesse Glick made changes -
            Link This issue is related to SECURITY-47 [ SECURITY-47 ]
            danielbeck Daniel Beck made changes -
            Link This issue is duplicated by JENKINS-25116 [ JENKINS-25116 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 1319 (Web Link)" [ 11803 ]
            Hide
            elephantjim Jim Rath added a comment -

            I'm using Jenkins behind a reverse proxy using nginx, too. I tried to set ignore_invalid_headers to "off" on my server's nginx settings to no avail. But then I found this: http://trac.nginx.org/nginx/ticket/117

            It turns out that nginx needs to process headers before selecting a virtual server: the "Host" header determines which server to use. Unless the ignore_invalid_headers is set to off on the default server, it won't get read. The other way is to set it in nginx.conf in the http context. That worked for me.

            Show
            elephantjim Jim Rath added a comment - I'm using Jenkins behind a reverse proxy using nginx, too. I tried to set ignore_invalid_headers to "off" on my server's nginx settings to no avail. But then I found this: http://trac.nginx.org/nginx/ticket/117 It turns out that nginx needs to process headers before selecting a virtual server: the "Host" header determines which server to use. Unless the ignore_invalid_headers is set to off on the default server, it won't get read. The other way is to set it in nginx.conf in the http context. That worked for me.
            buderu Florian Storz made changes -
            Link This issue is duplicated by JENKINS-27972 [ JENKINS-27972 ]
            Hide
            blaisorblade Paolo Giarrusso added a comment - - edited

            I ran into this yesterday. What's the status? Somebody proposed an X-Jenkins-Crumb header (in JENKINS-7518), which doesn't seem a legal JavaScript identifier (though it might be "the start of a legal JavaScript statement").
            Would X-Jenkins-Crumb-0NoJS help, since 0NoJs is an illegal JS token?

            Also, SECURITY-47 isn't visible in the bug tracker, apparently on purpose — I've only found the fix (https://github.com/jenkinsci/jenkins/commit/0de3e9b14ed75f70279435e78eb9f6a3a1a179df) and maybe the advisory (https://wiki.jenkins-ci.org/display/SECURITY/Jenkins+Security+Advisory+2013-02-16).

            For my part, I tried to document this on https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+behind+an+NGinX+reverse+proxy; I only mentioned the most secure workaround, not the -Dhudson.security.csrf.requestfield= setting. Improvements are welcome.

            Show
            blaisorblade Paolo Giarrusso added a comment - - edited I ran into this yesterday. What's the status? Somebody proposed an X-Jenkins-Crumb header (in JENKINS-7518 ), which doesn't seem a legal JavaScript identifier (though it might be "the start of a legal JavaScript statement"). Would X-Jenkins-Crumb-0NoJS help, since 0NoJs is an illegal JS token? Also, SECURITY-47 isn't visible in the bug tracker, apparently on purpose — I've only found the fix ( https://github.com/jenkinsci/jenkins/commit/0de3e9b14ed75f70279435e78eb9f6a3a1a179df ) and maybe the advisory ( https://wiki.jenkins-ci.org/display/SECURITY/Jenkins+Security+Advisory+2013-02-16 ). For my part, I tried to document this on https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+behind+an+NGinX+reverse+proxy ; I only mentioned the most secure workaround, not the -Dhudson.security.csrf.requestfield= setting. Improvements are welcome.
            Hide
            timboudreau Tim Boudreau added a comment -

            Running into this too, with NginX, on https://timboudreau.com/builds

            I've set ignore_invalid_headers to off, but I'm not keen on leaving it that way, especially for an NginX instance that is proxying far more than just Jenkins - there are other forms of nastiness that would be better not to pass on to back end applications that makes possible.

            Show
            timboudreau Tim Boudreau added a comment - Running into this too, with NginX, on https://timboudreau.com/builds I've set ignore_invalid_headers to off, but I'm not keen on leaving it that way, especially for an NginX instance that is proxying far more than just Jenkins - there are other forms of nastiness that would be better not to pass on to back end applications that makes possible.
            Hide
            tomreyn Tom Reynolds added a comment - - edited

            The number of people watching this issue is pretty high compared to that of other issues. This may provide an indication as to how much of an issue it is.

            C.f. http://seclists.org/bugtraq/2015/Aug/161

            Adding a couple HTTP headers would also help to harden the security of a Jenkins installation (may break some Jenkins plugins - works for me). Assuming your domain name is EXAMPLE.ORG and you use HTTPS:

            Content-Security-Policy: "default-src 'self' EXAMPLE.ORG *.EXAMPLE.ORG; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src *; media-src *; object-src 'self'; frame-src 'self' updates.jenkins-ci.org;"
            X-Frame-Options: "sameorigin, DENY"
            X-XSS-Protection: "1; mode=block"
            X-Content-Type-Options:"nosniff"
            Strict-Transport-Security:"max-age=15768000"
            
            Show
            tomreyn Tom Reynolds added a comment - - edited The number of people watching this issue is pretty high compared to that of other issues. This may provide an indication as to how much of an issue it is. C.f. http://seclists.org/bugtraq/2015/Aug/161 Adding a couple HTTP headers would also help to harden the security of a Jenkins installation (may break some Jenkins plugins - works for me). Assuming your domain name is EXAMPLE.ORG and you use HTTPS: Content-Security-Policy: "default-src 'self' EXAMPLE.ORG *.EXAMPLE.ORG; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src *; media-src *; object-src 'self'; frame-src 'self' updates.jenkins-ci.org;" X-Frame-Options: "sameorigin, DENY" X-XSS-Protection: "1; mode=block" X-Content-Type-Options:"nosniff" Strict-Transport-Security:"max-age=15768000"
            Hide
            timboudreau Tim Boudreau added a comment -

            > as crumb is a valid Javascript identifier that would open you to attack

            Would like to understand better how a header name being a Javascript identifier opens one to attack - there are plenty of standard HTTP header names that are valid identifiers - if I make an HTTP request containing

            Date: eval("alert('Hey')")

            is something foolish enough to execute it?

            At any rate, the problem here is simple - ".crumb" is not a legal HTTP header, but there are plenty of options that are legal HTTP headers and illegal as js identifiers - one has already been suggested. I don't think any of us want to leave our NginX installs set to accept whatever garbage a client decides to turn into headers.

            Re the suggested additional headers: Some are not a bad idea, but I could write a NodeJS proxy that would strip those out in about 10 minutes and go hog wild (or just use a programmatic HTTP client) - you can't rely on that sort of thing to give you real security, just deter low-hanging-fruit bad behavior.

            Show
            timboudreau Tim Boudreau added a comment - > as crumb is a valid Javascript identifier that would open you to attack Would like to understand better how a header name being a Javascript identifier opens one to attack - there are plenty of standard HTTP header names that are valid identifiers - if I make an HTTP request containing Date: eval("alert('Hey')") is something foolish enough to execute it? At any rate, the problem here is simple - ".crumb" is not a legal HTTP header, but there are plenty of options that are legal HTTP headers and illegal as js identifiers - one has already been suggested. I don't think any of us want to leave our NginX installs set to accept whatever garbage a client decides to turn into headers. Re the suggested additional headers: Some are not a bad idea, but I could write a NodeJS proxy that would strip those out in about 10 minutes and go hog wild (or just use a programmatic HTTP client) - you can't rely on that sort of thing to give you real security, just deter low-hanging-fruit bad behavior.
            Hide
            jglick Jesse Glick added a comment -

            Would like to understand better how a header name being a Javascript identifier opens one to attack

            It is complicated but has to do with how the crumb is offered to legitimate clients.

            Anyway the fix is well understood and filed, “just” needs to be tested and reviewed by someone qualified in web security.

            Show
            jglick Jesse Glick added a comment - Would like to understand better how a header name being a Javascript identifier opens one to attack It is complicated but has to do with how the crumb is offered to legitimate clients. Anyway the fix is well understood and filed, “just” needs to be tested and reviewed by someone qualified in web security.
            Hide
            danielbeck Daniel Beck added a comment -

            Tim Boudreau You wrote:

            ".crumb" is not a legal HTTP header, but there are plenty of options that are legal HTTP headers and illegal as js identifiers

            Just curious, where did you get this information from? Neither RFC 2822 (section 2.6.8) nor RFC 7230 seem to impose any header name restrictions beyond being printable ascii and not including spaces or colons. (Not saying this shouldn't be fixed due to compatibility, but I looked repeatedly, even back when I worked on this, and couldn't find anything like what you're saying).

            Show
            danielbeck Daniel Beck added a comment - Tim Boudreau You wrote: ".crumb" is not a legal HTTP header, but there are plenty of options that are legal HTTP headers and illegal as js identifiers Just curious, where did you get this information from? Neither RFC 2822 (section 2.6.8) nor RFC 7230 seem to impose any header name restrictions beyond being printable ascii and not including spaces or colons. (Not saying this shouldn't be fixed due to compatibility, but I looked repeatedly, even back when I worked on this, and couldn't find anything like what you're saying).
            Hide
            blaisorblade Paolo Giarrusso added a comment -

            On header names: I mostly agree with Daniel Beck about the spec (after reading both specs, I've also looked at http://stackoverflow.com/a/3569667/53974 — beyond colons, other "separators" are forbidden, but not dots). I think our source is just what the nginx docs imply without giving any reference (http://nginx.org/en/docs/http/ngx_http_core_module.html#ignore_invalid_headers).

            > Re the suggested additional headers: Some are not a bad idea, but I could write a NodeJS proxy that would strip those out in about 10 minutes and go hog wild (or just use a programmatic HTTP client) - you can't rely on that sort of thing to give you real security, just deter low-hanging-fruit bad behavior.

            At least Strict-Transport-Security is there to offer some protection for clients (browsers) that do honor them (and I think some other ones too). Not sure how you propose to use the proxy, but that headers assumes Jenkins is behind HTTPS, and then attackers can't drop the header.

            However, please don't silently enable Strict-Transport-Security — that's something for the sysadmin to decide, and it's not fun if one HTTP application enables it on a host where other services don't support HTTPS.

            Anyway, this is OT here, maybe somebody (Tom Reynolds) can open a different bug for them? (I don't understand the headers well enough to do that).

            Show
            blaisorblade Paolo Giarrusso added a comment - On header names: I mostly agree with Daniel Beck about the spec (after reading both specs, I've also looked at http://stackoverflow.com/a/3569667/53974 — beyond colons, other "separators" are forbidden, but not dots). I think our source is just what the nginx docs imply without giving any reference ( http://nginx.org/en/docs/http/ngx_http_core_module.html#ignore_invalid_headers ). > Re the suggested additional headers: Some are not a bad idea, but I could write a NodeJS proxy that would strip those out in about 10 minutes and go hog wild (or just use a programmatic HTTP client) - you can't rely on that sort of thing to give you real security, just deter low-hanging-fruit bad behavior. At least Strict-Transport-Security is there to offer some protection for clients (browsers) that do honor them (and I think some other ones too). Not sure how you propose to use the proxy, but that headers assumes Jenkins is behind HTTPS, and then attackers can't drop the header. However, please don't silently enable Strict-Transport-Security — that's something for the sysadmin to decide, and it's not fun if one HTTP application enables it on a host where other services don't support HTTPS. Anyway, this is OT here, maybe somebody (Tom Reynolds) can open a different bug for them? (I don't understand the headers well enough to do that).
            jglick Jesse Glick made changes -
            Labels 2.0
            jglick Jesse Glick made changes -
            Labels 2.0 2.0 crumb
            danielbeck Daniel Beck made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            danielbeck Daniel Beck made changes -
            Epic Link JENKINS-33810 [ 169285 ]
            danielbeck Daniel Beck made changes -
            Labels 2.0 crumb 2.0 2.0-planned crumb
            Hide
            danielbeck Daniel Beck added a comment -

            We already got some enthusiastic feedback how broken Jenkins is with nginx. Looks like we actually should address this before shipping with CSRF protection enabled by default.

            Show
            danielbeck Daniel Beck added a comment - We already got some enthusiastic feedback how broken Jenkins is with nginx. Looks like we actually should address this before shipping with CSRF protection enabled by default.
            Hide
            jglick Jesse Glick added a comment -

            Assigning to Daniel Beck on the basis that he filed PR 1319 which possibly just needs to have some merge conflicts resolved and be retested, but feel free to reassign if you think someone else should pick it up.

            Show
            jglick Jesse Glick added a comment - Assigning to Daniel Beck on the basis that he filed PR 1319 which possibly just needs to have some merge conflicts resolved and be retested, but feel free to reassign if you think someone else should pick it up.
            jglick Jesse Glick made changes -
            Assignee Daniel Beck [ danielbeck ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Daniel Beck
            Path:
            core/src/main/java/hudson/security/csrf/CrumbFilter.java
            core/src/main/java/hudson/security/csrf/CrumbIssuer.java
            core/src/main/java/hudson/security/csrf/CrumbIssuerDescriptor.java
            core/src/main/java/hudson/security/csrf/DefaultCrumbIssuer.java
            test/src/test/groovy/hudson/model/SlaveTest.groovy
            http://jenkins-ci.org/commit/jenkins/ed0ea631cd4af0e678da96d7297910ba1612749d
            Log:
            [FIXED JENKINS-12875] Change default crumb name to Jenkins-Crumb

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/security/csrf/CrumbFilter.java core/src/main/java/hudson/security/csrf/CrumbIssuer.java core/src/main/java/hudson/security/csrf/CrumbIssuerDescriptor.java core/src/main/java/hudson/security/csrf/DefaultCrumbIssuer.java test/src/test/groovy/hudson/model/SlaveTest.groovy http://jenkins-ci.org/commit/jenkins/ed0ea631cd4af0e678da96d7297910ba1612749d Log: [FIXED JENKINS-12875] Change default crumb name to Jenkins-Crumb
            scm_issue_link SCM/JIRA link daemon made changes -
            Status In Progress [ 3 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Daniel Beck
            Path:
            test/src/test/java/lib/form/RowSetTest.java
            http://jenkins-ci.org/commit/jenkins/5b1642252ebd977c21832eb12cae633b3616429f
            Log:
            JENKINS-12875 Fix test

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: test/src/test/java/lib/form/RowSetTest.java http://jenkins-ci.org/commit/jenkins/5b1642252ebd977c21832eb12cae633b3616429f Log: JENKINS-12875 Fix test
            Hide
            danielbeck Daniel Beck added a comment -

            Fixed towards Jenkins 2.0 which is scheduled to be released in April.

            Show
            danielbeck Daniel Beck added a comment - Fixed towards Jenkins 2.0 which is scheduled to be released in April.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Daniel Beck
            Path:
            core/src/main/java/hudson/security/csrf/CrumbIssuer.java
            http://jenkins-ci.org/commit/jenkins/8b6d38de67154fb96c59791082d5cbe743b2b693
            Log:
            JENKINS-12875 Restrict access to constant

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/security/csrf/CrumbIssuer.java http://jenkins-ci.org/commit/jenkins/8b6d38de67154fb96c59791082d5cbe743b2b693 Log: JENKINS-12875 Restrict access to constant
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Daniel Beck
            Path:
            core/src/main/java/hudson/security/csrf/CrumbFilter.java
            core/src/main/java/hudson/security/csrf/CrumbIssuer.java
            core/src/main/java/hudson/security/csrf/CrumbIssuerDescriptor.java
            core/src/main/java/hudson/security/csrf/DefaultCrumbIssuer.java
            test/src/test/groovy/hudson/model/SlaveTest.groovy
            test/src/test/java/lib/form/RowSetTest.java
            http://jenkins-ci.org/commit/jenkins/a87c38c5052cba745127a0205c40490b30c8bdbf
            Log:
            Merge pull request #2198 from daniel-beck/JENKINS-12875

            [FIX JENKINS-12875] Change default crumb name to Jenkins-Crumb

            Compare: https://github.com/jenkinsci/jenkins/compare/4649e04cbe58...a87c38c5052c

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/security/csrf/CrumbFilter.java core/src/main/java/hudson/security/csrf/CrumbIssuer.java core/src/main/java/hudson/security/csrf/CrumbIssuerDescriptor.java core/src/main/java/hudson/security/csrf/DefaultCrumbIssuer.java test/src/test/groovy/hudson/model/SlaveTest.groovy test/src/test/java/lib/form/RowSetTest.java http://jenkins-ci.org/commit/jenkins/a87c38c5052cba745127a0205c40490b30c8bdbf Log: Merge pull request #2198 from daniel-beck/ JENKINS-12875 [FIX JENKINS-12875] Change default crumb name to Jenkins-Crumb Compare: https://github.com/jenkinsci/jenkins/compare/4649e04cbe58...a87c38c5052c
            Hide
            jknurek J Knurek added a comment -

            Maybe it's worth noting in this ticket:
            I installed Jenkins 2.0-rc-1 and created a view using Build Monitor View plugin (1.8+build.201601112328), and get this error (which leaves the UI in an infinite "sorry" loop

            Apr 14, 2016 8:16:03 AM hudson.security.csrf.CrumbFilter doFilter
            WARNING: No valid crumb was included in request for /$stapler/bound/18964be5-e86d-4242-a57b-35ee573d52c9/fetchJobViews. Returning 403.
            
            Show
            jknurek J Knurek added a comment - Maybe it's worth noting in this ticket: I installed Jenkins 2.0-rc-1 and created a view using Build Monitor View plugin (1.8+build.201601112328), and get this error (which leaves the UI in an infinite "sorry" loop Apr 14, 2016 8:16:03 AM hudson.security.csrf.CrumbFilter doFilter WARNING: No valid crumb was included in request for /$stapler/bound/18964be5-e86d-4242-a57b-35ee573d52c9/fetchJobViews. Returning 403.
            Hide
            danielbeck Daniel Beck added a comment -

            J Knurek That's unrelated to this issue and looks like a bug in the plugin that cannot handle the CSRF protection option (around since 2010 or so) being enabled.

            Show
            danielbeck Daniel Beck added a comment - J Knurek That's unrelated to this issue and looks like a bug in the plugin that cannot handle the CSRF protection option (around since 2010 or so) being enabled.
            Hide
            jknurek J Knurek added a comment -

            My first thought was that it was a bug specific to the Build Monitor View plugin, but when I run Jenkins 1.651.1, the same configuration works ok. I'm only getting this error with the 2.0-rc-1.
            (but I'm not sure what the underlying issue is in this case, so it is likely a separate issue)

            Show
            jknurek J Knurek added a comment - My first thought was that it was a bug specific to the Build Monitor View plugin, but when I run Jenkins 1.651.1, the same configuration works ok. I'm only getting this error with the 2.0-rc-1. (but I'm not sure what the underlying issue is in this case, so it is likely a separate issue)
            janek Jan Molak made changes -
            Comment [ Hey Daniel and thanks for your suggestion about Build Monitor not handling the CSRF protection option.
            The CSRF protection option works perfectly fine with Build Monitor on Jenkins 1.x.

            Has anything changed regarding CSRF and the Crumb/.crumb headers handling on Jenkins 2.0? I'd be grateful for suggestions.

            Kind regards,
            Jan (author of Build Monitor) ]
            Hide
            janek Jan Molak added a comment -

            Daniel Beck Build Monitor has supported CSRF protection since its early days

            The problem is caused by change https://github.com/jenkinsci/jenkins/compare/4649e04cbe58...a87c38c5052c , which replaced the hard-coded '.crumb' header with 'Jenkins-Crumb', that's why they no longer match.

            I can work around that in Build Monitor.

            J Knurek you can watch the progress of this issue on github at https://github.com/jan-molak/jenkins-build-monitor-plugin/issues/215, which is the official issue tracker for Build Monitor.

            Thanks!
            Jan

            Show
            janek Jan Molak added a comment - Daniel Beck Build Monitor has supported CSRF protection since its early days The problem is caused by change https://github.com/jenkinsci/jenkins/compare/4649e04cbe58...a87c38c5052c , which replaced the hard-coded '.crumb' header with 'Jenkins-Crumb', that's why they no longer match. I can work around that in Build Monitor. J Knurek you can watch the progress of this issue on github at https://github.com/jan-molak/jenkins-build-monitor-plugin/issues/215 , which is the official issue tracker for Build Monitor. Thanks! Jan
            Hide
            danielbeck Daniel Beck added a comment -

            Jan Molak

            which replaced the hard-coded '.crumb' header with 'Jenkins-Crumb', that's why they no longer match.

            The crumb name has been configurable since 1.310 (hudson.security.csrf.requestfield), and that was necessary for anyone running Jenkins behind nginx, as it dropped the .crumb header (JENKINS-12875). So we renamed the default to something nginx accepts. Maybe it's a good idea to check which crumb name is used by Jenkins in your plugin to adapt to user configuration.

            Show
            danielbeck Daniel Beck added a comment - Jan Molak which replaced the hard-coded '.crumb' header with 'Jenkins-Crumb', that's why they no longer match. The crumb name has been configurable since 1.310 ( hudson.security.csrf.requestfield ), and that was necessary for anyone running Jenkins behind nginx, as it dropped the .crumb header ( JENKINS-12875 ). So we renamed the default to something nginx accepts. Maybe it's a good idea to check which crumb name is used by Jenkins in your plugin to adapt to user configuration.
            Hide
            janek Jan Molak added a comment - - edited

            J Knurek - Fixed in Build Monitor 1.9+build.2016043001​28, which you can download here before it hits the official update centre.

            Show
            janek Jan Molak added a comment - - edited J Knurek - Fixed in Build Monitor 1.9+build.2016043001​28, which you can download here before it hits the official update centre.
            Hide
            janek Jan Molak added a comment - - edited

            Awesome, thanks Daniel Beck, that's exactly what I did. The latest Build Monitor picks up the crumb name from CrumbIssuer::getCrumbRequestField so it should be a more appropriate solution

            Show
            janek Jan Molak added a comment - - edited Awesome, thanks Daniel Beck , that's exactly what I did. The latest Build Monitor picks up the crumb name from CrumbIssuer::getCrumbRequestField so it should be a more appropriate solution
            Hide
            nvn Naveen Sharma added a comment - - edited

            Hi,
            I have Jenkins 2.7, I have created the service hook in TFS to trigger the automatic build but while testing the hook I get " No valid crumb was included in the request (403)". I can run the same build from Jenkins dashboard with no issue.
            I have also installed the downgrade Build - Plugin-Version: 1.9+build.201606171408 as suggested by Jan Molak above but I am still facing the issue.

            P.s - I have installed Jenkins on my local windows 2007 machine.

            Thanks,

            Show
            nvn Naveen Sharma added a comment - - edited Hi, I have Jenkins 2.7, I have created the service hook in TFS to trigger the automatic build but while testing the hook I get " No valid crumb was included in the request (403)". I can run the same build from Jenkins dashboard with no issue. I have also installed the downgrade Build - Plugin-Version: 1.9+build.201606171408 as suggested by Jan Molak above but I am still facing the issue. P.s - I have installed Jenkins on my local windows 2007 machine. Thanks,
            Hide
            janek Jan Molak added a comment -

            Hey Naveen - from what you're saying it seems like the problem is related to Jenkins and not the Build Monitor plugin?
            If you think it's related to Build Monitor please feel free to raise a ticket on github at https://github.com/jan-molak/jenkins-build-monitor-plugin

            Thanks,
            Jan

            Show
            janek Jan Molak added a comment - Hey Naveen - from what you're saying it seems like the problem is related to Jenkins and not the Build Monitor plugin? If you think it's related to Build Monitor please feel free to raise a ticket on github at https://github.com/jan-molak/jenkins-build-monitor-plugin Thanks, Jan
            Hide
            nvn Naveen Sharma added a comment -

            Hi Jan, yeah I think the issue is either with Jenkins or may be with Tfs-jenkins plugin, I am not sure. Pls suggest.

            Thanks

            Show
            nvn Naveen Sharma added a comment - Hi Jan, yeah I think the issue is either with Jenkins or may be with Tfs-jenkins plugin, I am not sure. Pls suggest. Thanks
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 143333 ] JNJira + In-Review [ 190506 ]
            Hide
            12niner Ryan Cocks added a comment -

            Having just wasted a few hours debugging why I was seeing this: if you re-install Jenkins you need to clear your cookies or you'll hit this.

            Show
            12niner Ryan Cocks added a comment - Having just wasted a few hours debugging why I was seeing this: if you re-install Jenkins you need to clear your cookies or you'll hit this.
            Hide
            danielbeck Daniel Beck added a comment -

            Ryan Cocks Too many cookies is tracked as JENKINS-25046.

            Show
            danielbeck Daniel Beck added a comment - Ryan Cocks Too many cookies is tracked as JENKINS-25046 .

              People

              • Assignee:
                danielbeck Daniel Beck
                Reporter:
                naloy Núria Aloy
              • Votes:
                20 Vote for this issue
                Watchers:
                43 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: