Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60353

Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

    Details

    • Similar Issues:

      Description

      How to reproduce:

      Use multibranch-pipeline workflow with merge PR strategy

      1. Create a branch, develop it and make it nonmergable into the target branch
      2. Push the branch and open the Pull Request in GH
      3. Find out that the branch is not-mergeable
      4. Fix the code, push the newer revision

      Expected Result:

      The pipeline is triggered.

      Actual Result:

      1. On step #2 Jenkins gets the 404 in the request to ...refs/pull/<id>/merge/contents
      2. The 404 is cached
      3. On the step 4 it does not query GH but rather returns a cryptic error of 'Jenkins File not found' although it does not check for it at all

      Proposed solution: do not check for the contents of the "refs./..../merge" URL if the PR is not mergable.

      Versions:

      Jenkins 2.190.1
      GitHub Branch Source Plugin 2.5.8+ (some irrelevant local patches)

      Here is how we figured out what was happening:

      1. Get the repro of the bug
      2. Jump to the jenkins master node and create a copy of github cache dir, namely "${JENKINS_HOME}/org.jenkinsci.plugins.github_branch_source.GitHubSCMProbe.cache"
      3. grep for the pull request id in the *.0 files to figure out the  request id (prefix before the '.0') of the exchange with Github 
      4. zcat the file of the exchange (<reqid>.1) and see that Jenkins checks the refs/pull/<prid>/merge/contents and gets 404
      5. Jenkins never consults anything after getting this response until the cache entry with 404 is purged, but writes a cryptic message like 'Jenkinsfile not found' and (possibly) the lines saying that the PR branch is dead, thus being removed from the list of branches to run the build against,

      While surfing looking for the similar bugs, I noticed several that might be related to those ones, especially the first one.

      https://issues.jenkins-ci.org/browse/JENKINS-48571

      https://issues.jenkins-ci.org/browse/JENKINS-46290

      https://issues.jenkins-ci.org/browse/JENKINS-54126

      https://issues.jenkins-ci.org/browse/JENKINS-57206

      https://issues.jenkins-ci.org/browse/JENKINS-57411

       

       

        Attachments

        1. screenshot-1.png
          screenshot-1.png
          484 kB
        2. screenshot-2.png
          screenshot-2.png
          755 kB
        3. screenshot-3.png
          screenshot-3.png
          572 kB
        4. screenshot-4.png
          screenshot-4.png
          344 kB

          Issue Links

            Activity

            Hide
            kshultz Karl Shultz added a comment -

            I've got it recreated. Vladimir Kuklin this was so helpful. Here's a quick run-through of what I did.

            First half:

            1. Non-mergeable PR (738) gets discovered.

            2. I get a "Jenkinsfile not found" for it. Right off the bat that's wrong, since "Jenkinsfile not found" isn't the right thing to say in this case:

            3. Take a copy of the GitHub cache, and then grep for 738:

            ➜  before_fixing_merge_conflict grep -R -i 738 *                                   
            1b8c3a1c80a0c4863dff3d30a949af49.0:https://api.github.com/repos/kshultzCB/creates-and-deletes/contents/?ref=refs%2Fpull%2F738%2Fmerge
            2fb7a3744b1d3e6637744ef80f3dd8f6.0:https://api.github.com/repos/kshultzCB/creates-and-deletes/pulls/738
            eb5a189e88f6f6e47675c4de3a8e01f5.0:ETag: W/"c317b43d0986c7ea38fca51b7e76aaae41c738b9"
            

            4. The first file 1b8c3a1c80a0c4863dff3d30a949af49.0 is the merge, which is 404.

            ➜  before_fixing_merge_conflict cat 1b8c3a1c80a0c4863dff3d30a949af49.0       
            https://api.github.com/repos/kshultzCB/creates-and-deletes/contents/?ref=refs%2Fpull%2F738%2Fmerge
            GET
            0
            HTTP/1.1 404 Not Found
            23
            Server: GitHub.com
            Date: Thu, 09 Jan 2020 20:02:52 GMT
            Content-Type: application/json; charset=utf-8
            Transfer-Encoding: chunked
            Status: 404 Not Found
            X-RateLimit-Limit: 5000
            X-RateLimit-Remaining: 4982
            X-RateLimit-Reset: 1578603771
            X-OAuth-Scopes: admin:org, admin:org_hook, admin:repo_hook, gist, notifications, repo, user
            X-Accepted-OAuth-Scopes: 
            X-GitHub-Media-Type: github.v3; format=json
            Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type
            Access-Control-Allow-Origin: *
            Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
            X-Frame-Options: deny
            X-Content-Type-Options: nosniff
            X-XSS-Protection: 1; mode=block
            Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
            Content-Security-Policy: default-src 'none'
            Content-Encoding: gzip
            X-GitHub-Request-Id: E409:1B69:4DA5C5:CB461D:5E1786EC
            OkHttp-Sent-Millis: 1578600172579
            OkHttp-Received-Millis: 1578600172674
            

            Let's zcat its accompanying .1 file:

            ➜  before_fixing_merge_conflict cat 1b8c3a1c80a0c4863dff3d30a949af49.1 | zcat
            {"message":"No commit found for the ref refs/pull/738/merge","documentation_url":"https://developer.github.com/v3/repos/contents/"}%  
            

            5. As I understand it, if I fix this merge conflict, and then do a rescan, and STILL get `Jenkinsfile not found` back, that's the bug.

            5a. In Step 2 above, the "Jenkinsfile not found" thing is wrong all by itself. There might be a clue there too.

            Show
            kshultz Karl Shultz added a comment - I've got it recreated. Vladimir Kuklin this was so helpful. Here's a quick run-through of what I did. First half: 1. Non-mergeable PR ( 738 ) gets discovered. 2. I get a "Jenkinsfile not found" for it. Right off the bat that's wrong, since "Jenkinsfile not found" isn't the right thing to say in this case: 3. Take a copy of the GitHub cache, and then grep for 738: ➜ before_fixing_merge_conflict grep -R -i 738 * 1b8c3a1c80a0c4863dff3d30a949af49.0:https://api.github.com/repos/kshultzCB/creates-and-deletes/contents/?ref=refs%2Fpull%2F738%2Fmerge 2fb7a3744b1d3e6637744ef80f3dd8f6.0:https://api.github.com/repos/kshultzCB/creates-and-deletes/pulls/738 eb5a189e88f6f6e47675c4de3a8e01f5.0:ETag: W/"c317b43d0986c7ea38fca51b7e76aaae41c738b9" 4. The first file 1b8c3a1c80a0c4863dff3d30a949af49.0 is the merge, which is 404. ➜ before_fixing_merge_conflict cat 1b8c3a1c80a0c4863dff3d30a949af49.0 https://api.github.com/repos/kshultzCB/creates-and-deletes/contents/?ref=refs%2Fpull%2F738%2Fmerge GET 0 HTTP/1.1 404 Not Found 23 Server: GitHub.com Date: Thu, 09 Jan 2020 20:02:52 GMT Content-Type: application/json; charset=utf-8 Transfer-Encoding: chunked Status: 404 Not Found X-RateLimit-Limit: 5000 X-RateLimit-Remaining: 4982 X-RateLimit-Reset: 1578603771 X-OAuth-Scopes: admin:org, admin:org_hook, admin:repo_hook, gist, notifications, repo, user X-Accepted-OAuth-Scopes: X-GitHub-Media-Type: github.v3; format=json Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type Access-Control-Allow-Origin: * Strict-Transport-Security: max-age=31536000; includeSubdomains; preload X-Frame-Options: deny X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin Content-Security-Policy: default-src 'none' Content-Encoding: gzip X-GitHub-Request-Id: E409:1B69:4DA5C5:CB461D:5E1786EC OkHttp-Sent-Millis: 1578600172579 OkHttp-Received-Millis: 1578600172674 Let's zcat its accompanying .1 file: ➜ before_fixing_merge_conflict cat 1b8c3a1c80a0c4863dff3d30a949af49.1 | zcat {"message":"No commit found for the ref refs/pull/738/merge","documentation_url":"https://developer.github.com/v3/repos/contents/"}% 5. As I understand it, if I fix this merge conflict, and then do a rescan, and STILL get `Jenkinsfile not found` back, that's the bug. 5a. In Step 2 above, the "Jenkinsfile not found" thing is wrong all by itself. There might be a clue there too.
            Hide
            kshultz Karl Shultz added a comment -

            Second half - fix the PR so it can be merged, trigger a rescan, and see if the "Jenkinsfile not found" message is still there. If it is, that's the bug.

            These are screenshots of screenshots, so the resolution isn't great, but should be good enough.

            1. Fix the pull request up so it's mergeable.

            2. Perform a scan, and still get the "Jenkinsfile not found" message:

            3. The branch, and the PR, are both missing from Jenkins.

            Show
            kshultz Karl Shultz added a comment - Second half - fix the PR so it can be merged, trigger a rescan, and see if the "Jenkinsfile not found" message is still there. If it is, that's the bug. These are screenshots of screenshots, so the resolution isn't great, but should be good enough. 1. Fix the pull request up so it's mergeable. 2. Perform a scan, and still get the "Jenkinsfile not found" message: 3. The branch, and the PR, are both missing from Jenkins.
            Hide
            kshultz Karl Shultz added a comment -

            For the record, converting the plugin to use okhttp3 did not solve this problem.

            Show
            kshultz Karl Shultz added a comment - For the record, converting the plugin to use okhttp3 did not solve this problem.
            Hide
            kshultz Karl Shultz added a comment -

            Liam Newman has been able to recreate this behavior using a test in github-api. There's a draft PR demonstrating it.

            Show
            kshultz Karl Shultz added a comment - Liam Newman has been able to recreate this behavior using a test in github-api. There's a draft PR demonstrating it .
            Hide
            bitwiseman Liam Newman added a comment - - edited

            To Be Clear: This is a bug in the GitHub API. The problem is described in https://github.com/github-api/github-api/pull/669 .

            The linked PR (https://github.com/github-api/github-api/pull/665) now shows a workaround that should work for all scenarios. I think can't be done at the Jenkins plugin level. It must be fixed at the github-api library level. But this is probably a good thing as the Jenkins workarounds are only bandaids.

            We'll do the work to upgrade this dependency (already in progress) and this problem will go away.

            Show
            bitwiseman Liam Newman added a comment - - edited To Be Clear: This is a bug in the GitHub API. The problem is described in https://github.com/github-api/github-api/pull/669 . The linked PR ( https://github.com/github-api/github-api/pull/665 ) now shows a workaround that should work for all scenarios. I think can't be done at the Jenkins plugin level. It must be fixed at the github-api library level. But this is probably a good thing as the Jenkins workarounds are only bandaids. We'll do the work to upgrade this dependency (already in progress) and this problem will go away.

              People

              • Assignee:
                kshultz Karl Shultz
                Reporter:
                vkuklinatjoom Vladimir Kuklin
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: