Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50154

Webhook delivery failed as X-Hub-Signature did not match calculated

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: github-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.89.4, Github Plugin 1.29.0
    • Similar Issues:

      Description

      Our users were reporting that several PR's across a number of different repositories were not being built by Jenkins. Our Webhooks are setup manually and we use Organization Folders as documented here and Freestyle/Pipeline Jobs as documented here

      We noticed that Github was showing a number of failures in both our organization and in individual repositories with messages like so:

       

      <html>
       <head>
       <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
       <title>Error 400 Provided signature [139e89df4530d9827a6a8f32c1ee28564fff178f] did not match to calculated</title>
       </head>
       <body><h2>HTTP ERROR 400</h2>
       <p>Problem accessing /github-webhook/. Reason:
       <pre> Provided signature [139e89df4530d9827a6a8f32c1ee28564fff178f] did not match to calculated</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.z-SNAPSHOT</a><hr/>
       </body>
      </html>
      

      This appeared to be limited to three repositories at first but it soon spread to other repositories. It's also intermittent and also tied to specific pull requests (i.e once a PR has had a failed webhook, subsequent updates to that PR will also fail).

      We had upgraded from Jenkins 2.64.3 and Github Plugin 1.27.0 and migrated to new instances in AWS over the weekend and had wondered if this may have made the issue worse - we've seen similar failures before now but less frequently.

      So far we have tried:

      • Rolling back to Github Plugin 1.28.1 which completely broke webhooks
      • Clearing the Github Plugin Cache (setting to 0, and back to a positive integer).
      • Changing the secret used for the webhook on Github and Jenkins
      • Changing the hook content type to `application/json`
      • Deleting and re-creating the hook

      The only workaround we have right now has been to:

      • Give our bot user Admin permissions on the affected repository
      • Enable 'Manage Hooks' and re-create hooks on all repositories
      • The created hook on the repository is, so far, delivering with 100% success

        Attachments

          Activity

          Hide
          recampbell Ryan Campbell added a comment -

          Dave, do you have any log messages with the category org.jenkinsci.plugins.github.webhook?

          Show
          recampbell Ryan Campbell added a comment - Dave, do you have any log messages with the category org.jenkinsci.plugins.github.webhook ?
          Hide
          dave_tucker Dave Tucker added a comment -

          Yes we see lots of instances of:

          Mar 14, 2018 3:12:58 PM FINEST org.jenkinsci.plugins.github.webhook.RequirePostWithGHHookPayload$Processor shouldProvideValidSignatureTrying to verify sign from header sha1=8e290f17632932bed4138eb632dd2e761f529bcfMar 14, 2018 3:12:58 PM FINEST org.jenkinsci.plugins.github.webhook.GHWebhookSignature matchesSignature: calculated=deff2fe4d0173b1f2c39d086fa6cb6ee4f85b1fd provided=8e290f17632932bed4138eb632dd2e761f529bcf
          Show
          dave_tucker Dave Tucker added a comment - Yes we see lots of instances of: Mar 14, 2018 3:12:58 PM FINEST org.jenkinsci.plugins.github.webhook.RequirePostWithGHHookPayload$Processor shouldProvideValidSignatureTrying to verify sign from header sha1=8e290f17632932bed4138eb632dd2e761f529bcfMar 14, 2018 3:12:58 PM FINEST org.jenkinsci.plugins.github.webhook.GHWebhookSignature matchesSignature: calculated=deff2fe4d0173b1f2c39d086fa6cb6ee4f85b1fd provided=8e290f17632932bed4138eb632dd2e761f529bcf
          Hide
          nusnewob Bowen Sun added a comment -

          We are seeing the same problem, but it wasn't tied to specific PR/branch, completely random and then just disappear.

          Show
          nusnewob Bowen Sun added a comment - We are seeing the same problem, but it wasn't tied to specific PR/branch, completely random and then just disappear.
          Hide
          joshm Josh McCullough added a comment - - edited

          I'm also seeing this issue this morning but only for a specific repo (1 out of 20+). Looking at the webhook history, this was working fine and then this same repo starting showing failed webhooks as of 2018-08-09 14:36:05. 

          Note that we do not have repo-level hooks, just a single org-level hook. All other builds are working fine!

          Show
          joshm Josh McCullough added a comment - - edited I'm also seeing this issue this morning but only for a specific repo (1 out of 20+). Looking at the webhook history, this was working fine and then this same repo starting showing failed webhooks as of 2018-08-09 14:36:05.  Note that we do not have repo-level hooks, just a single org-level hook. All other builds are working fine!
          Hide
          joshm Josh McCullough added a comment -

          BUMP! This is getting worse for us. We have more failed webhooks than succeeded now. We recently moved to multi-branch config, and now most of our webhook requests from GitHub are showing the "Provided signature [...] did not match to calculated" message.

          Show
          joshm Josh McCullough added a comment - BUMP! This is getting worse for us. We have more failed webhooks than succeeded now. We recently moved to multi-branch config, and now most of our webhook requests from GitHub are showing the "Provided signature [...] did not match to calculated" message.
          Hide
          medianick Nick Jones added a comment -

          In my case I was getting this randomly – for some requests, not others – but it seems to have been the result of choosing "application/x-www-form-urlencoded" as the Content Type on the GitHub side. Once I switched this to "application/json", the webhook payloads started consistently delivering successfully.

          Show
          medianick Nick Jones added a comment - In my case I was getting this randomly – for some requests, not others – but it seems to have been the result of choosing "application/x-www-form-urlencoded" as the Content Type on the GitHub side. Once I switched this to "application/json", the webhook payloads started consistently delivering successfully.

            People

            • Assignee:
              lanwen Kirill Merkushev
              Reporter:
              dave_tucker Dave Tucker
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: