[JIRA] (JENKINS-60353) Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

[vkuklin@joom.com (JIRA)]

Vladimir Kuklin created an issue

Jenkins / JENKINS-60353 Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404 Issue Type: Bug Assignee: Unassigned Components: github-branch-source-plugin Created: 2019-12-03 15:25 Priority: Minor Reporter: Vladimir Kuklin How to reproduce: Use multibranch-pipeline workflow Create a branch, develop it and make it nonmergable into the target branch Push the branch and open the Pull Request in GH Find out that the branch is not-mergeable Fix the code, push the newer revision # Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

[vkuklin@joom.com (JIRA)]

Vladimir Kuklin updated an issue

Jenkins / JENKINS-60353 Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

Change By: Vladimir Kuklin How to reproduce: Use multibranch-pipeline workflow with merge PR strategy # Create a branch, develop it and make it nonmergable into the target branch # Push the branch and open the Pull Request in GH # Find out that the branch is not-mergeable # Fix the code, push the newer revision Expected Result: The pipeline is triggered. Actual Result: # On step #2 Jenkins gets the 404 in the request to ...refs/pull/<id>/merge/contents # The 404 is cached # On the step 4 it does not query GH but rather returns a cryptic error of 'Jenkins File not found' although it does not check for it at all Proposed solution: do not check for the contents of the "refs./..../merge" URL if the PR is not mergable. Versions: Jenkins 2.190.1 GitHub Branch Source Plugin 2.5.8+ (some irrelevant local patches)

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz commented on JENKINS-60353

Re: Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404 Vladimir Kuklin, thanks for reporting this, and thanks for the recreate steps. I'll take a shot at this recreate in the morning. If this is the bug I'm hoping it is, this has been showing itself for a long time, and has been super annoying to try and recreate. I'm going to see if this will show itself on versions of GHBS older than 2.5.0 as well. I'm pretty sure we've been chasing this since well before that release.

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz edited a comment on JENKINS-60353

Re: Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

[~vkuklinatjoom], thanks for reporting this, and thanks for the recreate steps. I'll take a shot at this recreate in the morning. If this is the bug I'm hoping it is, this has been showing itself for a long time, and has been super annoying to try and recreate. I'm going to see if this will show itself on versions of GHBS older than 2.5.0 as well (see [JENKINS-43194|https://issues . jenkins-ci.org/browse/JENKINS-43194]). I'm pretty sure we've been chasing this since well before that release.

Add Comment

[vkuklin@joom.com (JIRA)]

Vladimir Kuklin updated an issue

Jenkins / JENKINS-60353

Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

Change By: Vladimir Kuklin

How to reproduce: Use multibranch-pipeline workflow with merge PR strategy # Create a branch, develop it and make it nonmergable into the target branch # Push the branch and open the Pull Request in GH # Find out that the branch is not-mergeable # Fix the code, push the newer revision Expected Result: The pipeline is triggered. Actual Result: # On step #2 Jenkins gets the 404 in the request to ...refs/pull/<id>/merge/contents # The 404 is cached # On the step 4 it does not query GH but rather returns a cryptic error of 'Jenkins File not found' although it does not check for it at all Proposed solution: do not check for the contents of the "refs./..../merge" URL if the PR is not mergable. Versions: Jenkins 2.190.1 GitHub Branch Source Plugin 2.5.8+ (some irrelevant local patches)

Here is how we figured out what was happening: # Get the repro of the bug # Jump to the jenkins master node and create a copy of github cache dir, namely "${JENKINS_HOME}/org.jenkinsci.plugins.github_branch_source.GitHubSCMProbe.cache" # grep for the pull request id in the *.0 files to figure out the  request id (prefix before the '.0') of the exchange with Github  # zcat the file of the exchange (<reqid>.1) and see that Jenkins checks the refs/pull/<prid>/merge/contents and gets 404 # Jenkins never consults anything after getting this response until the cache entry with 404 is purged, but writes a cryptic message like 'Jenkinsfile not found' and (possibly) the lines saying that the PR branch is dead, thus being removed from the list of branches to run the build against, While surfing looking for the similar bugs, I noticed several that might be related to those ones, especially the first one. https://issues.jenkins-ci.org/browse/JENKINS-48571 https://issues.jenkins-ci.org/browse/JENKINS-46290 https://issues.jenkins-ci.org/browse/JENKINS-54126 https://issues.jenkins-ci.org/browse/JENKINS-57206 https://issues.jenkins-ci.org/browse/JENKINS-57411

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz assigned an issue to Karl Shultz

Jenkins / JENKINS-60353 Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

Change By: Karl Shultz Assignee: Karl Shultz

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz started work on JENKINS-60353

Change By: Karl Shultz Status: Open In Progress

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz commented on JENKINS-60353

Re: Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404 Vladimir Kuklin - I want to make sure I'm doing this right. Here's what I've done. 1. Created a multibranch project on a GitHub repo. It includes a basic text file so I can easily create merge conflicts. 2. Create a new branch based off master, make changes to the text file, and push those changes to a new branch "change1" on GitHub of the same name. 3. Scan the repository from Jenkins. The branch "change1" appears on Jenkins. 4. Do a local checkout of master, intentionally change the text file in such a way that when I try to create a pull request from "change1" into "master," the PR will not be mergeable. Push the change to master. 5. Switch back to github.com and create the PR from "change1" to "master." As expected, it shows as not being mergeable. 6. Scan the repository from Jenkinis. The "change1" branch disappears, which is what I would expect, given I'm using the default Discover Branches behavior of "Exclude branches that are also filed as PRs." 7. Copy the github cache folder to someplace where I can mess around with it. I've found the correct file, knowing I'd created PR number 737 to my test repo: $ grep -R -i 737 * 141d992620ae1ba8c1e133a6649c3286.0:https://api.github.com/repos/kshultzCB/creates-and-deletes/pulls/737 3cb2ea32bdd3048a2fb77bc4e7424929.0:X-GitHub-Request-Id: E3A6:7714:117D9D6:2D87374:5DEAB8A0 54d724cbddb40d04caa4c58007667fba.0:https://api.github.com/repos/kshultzCB/creates-and-deletes/contents/?ref=refs%2Fpull%2F737%2Fmerge Based upon that, the filenames I'm after should be 54d724cbddb40d04caa4c58007667fba.0 and 54d724cbddb40d04caa4c58007667fba.1. When I zcat 54d724cbddb40d04caa4c58007667fba.1, what I get back is json. I've formatted it so it's pretty, but this is what's inside: { "message": "No commit found for the ref refs/pull/737/merge", "documentation_url": "https://developer.github.com/v3/repos/contents/" } What I'm expecting is that I'll fix the PR so it can be merged, rescan the repository from Jenkins, and hit our "Jenkinsfile not found" problem. Have I got this right so far?

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz edited a comment on JENKINS-60353

Re: Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

[~vkuklinatjoom] - I want to make sure I'm doing this right. Here's what I've done.

1. Created a multibranch project on a GitHub repo. It includes a basic text file so I can easily create merge conflicts. 2. Create a new branch based off master, make changes to the text file, and push those changes to a new branch

" {{ change1 " }} on GitHub of the same name.

3. Scan the repository from Jenkins. The branch

" {{ change1 " }} appears on Jenkins. 4. Do a local checkout of master, intentionally change the text file in such a way that when I try to create a pull request from " {{ change1 " }} into " {{ master }} , " the PR will not be mergeable. Push the change to master. 5. Switch back to github.com and create the PR from " {{ change1 " }} to " {{ master }} . " As expected, it shows as not being mergeable. 6. Scan the repository from Jenkinis. The " {{ change1 " }} branch disappears from the Branches tab , which is what I would expect, given I'm using the default Discover Branches behavior of "Exclude branches that are also filed as PRs."

7. Copy the github cache folder to someplace where I can mess around with it.  I've found the correct file, knowing I'd created PR number 737 to my test repo:

{code:none}

$ grep -R -i 737 * 141d992620ae1ba8c1e133a6649c3286.0:https://api.github.com/repos/kshultzCB/creates-and-deletes/pulls/737 3cb2ea32bdd3048a2fb77bc4e7424929.0:X-GitHub-Request-Id: E3A6:7714:117D9D6:2D87374:5DEAB8A0 54d724cbddb40d04caa4c58007667fba.0:https://api.github.com/repos/kshultzCB/creates-and-deletes/contents/?ref=refs%2Fpull%2F737%2Fmerge

{code} Based upon that, the filenames I'm after should be {{54d724cbddb40d04caa4c58007667fba.0}} and {{54d724cbddb40d04caa4c58007667fba.1}}. When I zcat {{54d724cbddb40d04caa4c58007667fba.1}}, what I get back is json. I've formatted it so it's pretty, but this is what's inside: {code:json}

{     "message": "No commit found for the ref refs/pull/737/merge",     "documentation_url": "https://developer.github.com/v3/repos/contents/" }

{code}

What I'm expecting is that I'll fix the PR so it can be merged, rescan the repository from Jenkins, and hit our "Jenkinsfile not found" problem. Have I got this right so far?

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz edited a comment on JENKINS-60353

Re: Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

[~vkuklinatjoom] - I want to make sure I'm doing this right. Here's what I've done. 1. Created a multibranch project on a GitHub repo. It includes a basic text file so I can easily create merge conflicts. 2. Create a new branch based off master, make changes to the text file, and push those changes to a new branch {{change1}} on GitHub of the same name. 3. Scan the repository from Jenkins. The branch {{change1}} appears on Jenkins. 4. Do a local checkout of master, intentionally change the text file in such a way that when I try to create a pull request from {{change1}} into {{master}}, the PR will not be mergeable. Push the change to master. 5. Switch back to github.com and create the PR from {{change1}} to {{master}}. As expected, it shows as not being mergeable.

6. Scan the repository from Jenkinis. The {{change1}} branch disappears from the Branches tab, which is what I would expect, given I'm using the default Discover Branches behavior of "Exclude branches that are also filed as PRs."

7. Copy the github cache folder to someplace where I can mess around with it.  I've found the correct file, knowing I'd created PR number 737 to my test repo: {code:none}

$ grep -R -i ' 737 . * 141d992620ae1ba8c1e133a6649c3286 merge' * .0 :https://api.github.com/repos/kshultzCB/creates-and-deletes/pulls/737

3cb2ea32bdd3048a2fb77bc4e7424929.0:X-GitHub-Request-Id: E3A6:7714:117D9D6:2D87374:5DEAB8A0

54d724cbddb40d04caa4c58007667fba.0:https://api.github.com/repos/kshultzCB/creates-and-deletes/contents/?ref=refs%2Fpull%2F737%2Fmerge {code} Based upon that, the filenames I'm after should be {{54d724cbddb40d04caa4c58007667fba.0}} and {{54d724cbddb40d04caa4c58007667fba.1}}. When I zcat {{54d724cbddb40d04caa4c58007667fba.1}}, what I get back is json. I've formatted it so it's pretty, but this is what's inside: {code:json} {     "message": "No commit found for the ref refs/pull/737/merge",     "documentation_url": "https://developer.github.com/v3/repos/contents/" } {code} What I'm expecting is that I'll fix the PR so it can be merged, rescan the repository from Jenkins, and hit our "Jenkinsfile not found" problem. Have I got this right so far?

Add Comment

[vkuklin@joom.com (JIRA)]

Vladimir Kuklin commented on JENKINS-60353

Re: Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

Pretty much that's it (the contents of the GHCache files are exactly the same that we have encountered) but there are several things and steps you need to check or follow: You should have github cache enabled with its size set to some value that the entry will not have been purged for a while (like an hour to make the pain of the developers really an issue) Step 8 should be - fix the conflict and push the new head into the source branch, so that the PR is mergeable now, but it will not get discovered as it is marked as the one that does have a Jenkinsfile/contents due to false-negative from the GHCache.  In some (or may be all) scenarios the "JenkinsFile not found" is not shown after purported step #8 as the PR branch is considered as "Dead", therefore not buildable. We can also see the logs like 'Ressurrect dead branch' or smth like this which points somewhere here:  https://github.com/jenkinsci/branch-api-plugin/blob/847348f6c741f9ac350c138d6267e41a081b89e4/src/main/java/jenkins/branch/MultiBranchProject.java Hence, I would suggest a step #9: Trigger the 'redeliver hook payload' button in github (you can do it with github webhooks page and resend the hook json once again) and see that nothing happens as if the (Multi)Branch (supposedly) plugin just skips the branch as not buildable/Dead.

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz commented on JENKINS-60353

Re: Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

Fantastic. Thanks so much! You've done some really excellent detective work on this. No real fear of the cache getting wiped out from under me, I'm just running the plugin locally via mvnDebug hpi:run. I'm using manually invoked repository scans rather than webhooks, but your comment leads me to believe that having webhooks available would be useful. I'll set that up next. Might be able to write a unit test that simulates this too. I'll look at that after I see this happen

Another question, if you don't mind. You wrote: In some (or may be all) scenarios the "JenkinsFile not found" is not shown after purported step #8 as the PR branch is considered as "Dead", therefore not buildable. We can also see the logs like 'Ressurrect dead branch' or smth like this which points somewhere here: https://github.com/jenkinsci/branch-api-plugin/blob/847348f6c741f9ac350c138d6267e41a081b89e4/src/main/java/jenkins/branch/MultiBranchProject.java

Sounds like you do not always see the "Jenkinsfile not found" message in your scan logs. Is that correct? If yes, that tracks with what we've been seeing before. The "Jenkinsfile not found" error has been very elusive and difficult to recreate.

Add Comment

[vkuklin@joom.com (JIRA)]

Vladimir Kuklin commented on JENKINS-60353

Re: Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

You are vey welcome, sir. Wrt the question I think (still not 100% sure) that the "Jenkinsfile not found" is shown the first time it misses the contents of the merge head. All the consequent attempts to redeliver the payload or rescan the branches end with the branch just considered "dead"/not buildable.

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz commented on JENKINS-60353

Re: Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

Now that everyone's back from various holidays, I'm going to pick this back up.

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz edited a comment on JENKINS-60353

Re: Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

Now that everyone's back from various holidays, I'm going hoping to pick this back up.

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz updated an issue

Jenkins / JENKINS-60353

Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

Change By: Karl Shultz Attachment: screenshot-1.png

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz commented on JENKINS-60353

Re: Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404 I've got it recreated. Vladimir Kuklin this was so helpful. Here's a quick run-through of what I did. First half: 1. Non-mergeable PR (738) gets discovered. 2. I get a "Jenkinsfile not found" for it. Right off the bat that's wrong, since "Jenkinsfile not found" isn't the right thing to say in this case: 3. Take a copy of the GitHub cache, and then grep for 738: ➜ before_fixing_merge_conflict grep -R -i 738 * 1b8c3a1c80a0c4863dff3d30a949af49.0:https://api.github.com/repos/kshultzCB/creates-and-deletes/contents/?ref=refs%2Fpull%2F738%2Fmerge 2fb7a3744b1d3e6637744ef80f3dd8f6.0:https://api.github.com/repos/kshultzCB/creates-and-deletes/pulls/738 eb5a189e88f6f6e47675c4de3a8e01f5.0:ETag: W/"c317b43d0986c7ea38fca51b7e76aaae41c738b9" 4. The first file 1b8c3a1c80a0c4863dff3d30a949af49.0 is the merge, which is 404. ➜ before_fixing_merge_conflict cat 1b8c3a1c80a0c4863dff3d30a949af49.0 https://api.github.com/repos/kshultzCB/creates-and-deletes/contents/?ref=refs%2Fpull%2F738%2Fmerge GET 0 HTTP/1.1 404 Not Found 23 Server: GitHub.com Date: Thu, 09 Jan 2020 20:02:52 GMT Content-Type: application/json; charset=utf-8 Transfer-Encoding: chunked Status: 404 Not Found X-RateLimit-Limit: 5000 X-RateLimit-Remaining: 4982 X-RateLimit-Reset: 1578603771 X-OAuth-Scopes: admin:org, admin:org_hook, admin:repo_hook, gist, notifications, repo, user X-Accepted-OAuth-Scopes: X-GitHub-Media-Type: github.v3; format=json Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type Access-Control-Allow-Origin: * Strict-Transport-Security: max-age=31536000; includeSubdomains; preload X-Frame-Options: deny X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin Content-Security-Policy: default-src 'none' Content-Encoding: gzip X-GitHub-Request-Id: E409:1B69:4DA5C5:CB461D:5E1786EC OkHttp-Sent-Millis: 1578600172579 OkHttp-Received-Millis: 1578600172674 Let's zcat its accompanying .1 file: ➜ before_fixing_merge_conflict cat 1b8c3a1c80a0c4863dff3d30a949af49.1 | zcat {"message":"No commit found for the ref refs/pull/738/merge","documentation_url":"https://developer.github.com/v3/repos/contents/"}% 5. As I understand it, if I fix this merge conflict, and then do a rescan, and STILL get `Jenkinsfile not found` back, that's the bug. 5a. In Step 2 above, the "Jenkinsfile not found" thing is wrong all by itself. There might be a clue there too.

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz updated an issue

Jenkins / JENKINS-60353

Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

Change By: Karl Shultz Attachment: screenshot-2.png

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz updated an issue

Jenkins / JENKINS-60353 Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

Change By: Karl Shultz Attachment: screenshot-3.png

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz updated an issue

Jenkins / JENKINS-60353 Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

Change By: Karl Shultz Attachment: screenshot-4.png

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz commented on JENKINS-60353

Re: Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404 Second half - fix the PR so it can be merged, trigger a rescan, and see if the "Jenkinsfile not found" message is still there. If it is, that's the bug. These are screenshots of screenshots, so the resolution isn't great, but should be good enough. 1. Fix the pull request up so it's mergeable. 2. Perform a scan, and still get the "Jenkinsfile not found" message: 3. The branch, and the PR, are both missing from Jenkins.

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz commented on JENKINS-60353

Re: Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

For the record, converting the plugin to use okhttp3 did not solve this problem.

Add Comment

[kshultz@cloudbees.com (JIRA)]

Karl Shultz commented on JENKINS-60353

Re: Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

Liam Newman has been able to recreate this behavior using a test in github-api. There's a draft PR demonstrating it.

Add Comment

[bitwiseman@gmail.com (JIRA)]

Liam Newman commented on JENKINS-60353

Re: Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

To Be Clear: This is a bug in the GitHub API. The linked PR (https://github.com/github-api/github-api/pull/665) now shows a workaround that should work for all scenarios. I think can't be done at the Jenkins plugin level. It must be fixed at the github-api library level. But this is probably a good thing as the Jenkins workarounds are only bandaids. We'll do the work to upgrade this dependency (already in progress) and this problem will go away.

Add Comment

[bitwiseman@gmail.com (JIRA)]

Liam Newman edited a comment on JENKINS-60353

Re: Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

To Be Clear: This is a bug in the GitHub API.

The problem is described in https://github.com/github-api/github-api/pull/669 .

The linked PR (https://github.com/github-api/github-api/pull/665) now shows a workaround that should work for all scenarios.  I think can't be done at the Jenkins plugin level. It must be fixed at the github-api library level.  But this is probably a good thing as the Jenkins workarounds are only bandaids.   We'll do the work to upgrade this dependency (already in progress) and this problem will go away.

Add Comment

[bitwiseman@gmail.com (JIRA)]

Liam Newman updated JENKINS-60353

The fix for this issue has been merged and will be released in the next day or two in github-branch-source v2.6.0. If you want to try it out now: install github-api-plugin 1.6.0 and then install the hpi from: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/github-branch-source/2.5.9-rc1028.3059575bf1cc/

Jenkins / JENKINS-60353

Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

Change By: Liam Newman Status: In Progress Fixed but Unreleased Resolution: Fixed

Add Comment

[bitwiseman@gmail.com (JIRA)]

Liam Newman commented on JENKINS-60353

Re: Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404 Vladimir Kuklin Github-branch-source-plugin v2.6.0

Add Comment

[bitwiseman@gmail.com (JIRA)]

Liam Newman updated JENKINS-60353

Jenkins / JENKINS-60353

Non-mergable Pull-request Contents Get into GH Cache with False-Positive 404

Change By: Liam Newman Status: Fixed but Unreleased Closed

Add Comment