[JIRA] [core] (JENKINS-29962) Found invalid crumb

[hany@vmfarms.com (JIRA)]

Hany Fahim created an issue

Jenkins / JENKINS-29962 Found invalid crumb Issue Type: Bug Assignee: Unassigned Components: core Created: 14/Aug/15 9:53 PM Environment: Jenkins 1.620 with nginx as proxy, SSL enabled. Labels: jenkins gui Priority: Blocker Reporter: Hany Fahim When trying to configure a new job, adding a new parameter using the drop-down results in a 403 error message being returned: 403 No valid crumb was included in the request The logs show: WARNING: Found invalid crumb <CRUMB_ID>, <CRUMB_ID>. Will check remaining parameters for a valid one... Aug 14, 2015 5:32:06 PM hudson.security.csrf.CrumbFilter doFilter WARNING: No valid crumb was included in request for /jenkins//$stapler/bound/dd7670cf-db32-481d-b6f3-6fcdfde6e658/render. Returning 403. Curiously, when examining the request headers, the crumb is actually being duplicated: Crumb:<CRUMB_ID>, <CRUMB_ID> Which is what the logs indicate as well, and claims it's invalid. Steps to replicate: 1. Go to the configuration of a job. 2. Click on a drop-down like "Add Parameter" 3. Choose a param type, and check request in network dev tools At the same time, this setup is also running into this bug when deleting projects: https://issues.jenkins-ci.org/browse/JENKINS-18032 Not sure if they are related. Add Comment

This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)

[dbeck@cloudbees.com (JIRA)]

Daniel Beck resolved as Not A Defect

Access Jenkins using the URL you specified in its global configuration.

Jenkins / JENKINS-29962 Found invalid crumb

Change By: Daniel Beck Status: Open Resolved Resolution: Not A Defect

Add Comment

[hany@vmfarms.com (JIRA)]

Hany Fahim reopened an issue

Hi, I'm not sure why this was closed so quickly, but we are accessing it from the same URL. Under "Jenkins URL" in the Jenkins Location header, we have: https://ci.hostname.com/jenkins/ And the server is being accessed from this URL. We are still getting the same error. Can you clarify what you mean?

Jenkins / JENKINS-29962 Found invalid crumb

Change By: Hany Fahim Resolution: Not A Defect Status: Resolved Reopened

Add Comment

[hany@vmfarms.com (JIRA)]

Hany Fahim commented on JENKINS-29962

Re: Found invalid crumb I've confirmed via developer tools that the request is being made to the right URL: https://ci.hostname.com/jenkins/$stapler/bound/8a619a33-bce4-4c9f-81fb-5c98ddd556c7/render Any ideas?

Add Comment

[dbeck@cloudbees.com (JIRA)]

Daniel Beck updated JENKINS-29962

Jenkins / JENKINS-29962 Found invalid crumb

Change By: Daniel Beck Status: Reopened Open

Add Comment

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-29962

Re: Found invalid crumb All instances of this issue I've seen are related to broken config – my apologies for being too quick. Does the /manage URL show a reverse proxy configuration warning? Could you provide a screenshot of the security configuration screen?

Add Comment

[hany@vmfarms.com (JIRA)]

Hany Fahim commented on JENKINS-29962

Re: Found invalid crumb Hi there, When navigating to /manage, there is no warning or other notice about reverse proxies. I've attached a screenshot of the relevant section about "Prevent Cross Site Request Forgery exploits" here. Unable to render embedded object: File (attachment-name.jpg) not found.

Add Comment

[hany@vmfarms.com (JIRA)]

Hany Fahim updated an issue

Jenkins / JENKINS-29962 Found invalid crumb

Change By: Hany Fahim Attachment: Screen Shot 2015-08-15 at 9.55.10 AM.png

Add Comment

[hany@vmfarms.com (JIRA)]

Hany Fahim edited a comment on JENKINS-29962

Re: Found invalid crumb

Hi there, When navigating to /manage, there is no warning or other notice about reverse proxies. I've attached a screenshot of the relevant section about "Prevent Cross Site Request Forgery exploits" here.

! attachment Screen Shot 2015 - name 08-15 at 9 . jpg 55.10 AM.png |thumbnail!

Add Comment

[mwarkentin@gmail.com (JIRA)]

Michael Warkentin commented on JENKINS-29962

Re: Found invalid crumb Hi Daniel Beck, I have access to the same Jenkins instance as Hany Fahim. I don't see any warnings about the reverse proxy in /jenkins/manage: https://s3.amazonaws.com/snaps.michaelwarkentin.com/Manage_Jenkins_Jenkins_2015-08-15_09-53-04.png Here's a screenshot of the CSRF section of the security page: https://s3.amazonaws.com/snaps.michaelwarkentin.com/Configure_Global_Security_Jenkins_2015-08-15_09-55-45.png

Add Comment

[hany@vmfarms.com (JIRA)]

Hany Fahim edited a comment on JENKINS-29962

Re: Found invalid crumb

Hi there, When navigating to /manage, there is no warning or other notice about reverse proxies.

I've attached a screenshot of the relevant section about "Prevent Cross Site Request Forgery exploits" here.  Are there any other relevant sections you need to see? Obviously there is sensitive information on this page, so let me know what you need. !Screen Shot 2015-08-15 at 9.55.10 AM.png|thumbnail!

Add Comment

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-29962

Re: Found invalid crumb Could you provide the headers and POST parameters sent by your browser for some request that gets rejected?

Add Comment

[mwarkentin@gmail.com (JIRA)]

Michael Warkentin commented on JENKINS-29962

Re: Found invalid crumb Here's an example AJAX request when trying to add a new parameter to a job (removed cookies): Accept:text/javascript, text/html, application/xml, text/xml, */* Accept-Encoding:gzip, deflate Accept-Language:en-US,en;q=0.8 Connection:keep-alive Content-Length:2 Content-type:application/x-stapler-method-invocation;charset=UTF-8 Crumb:3a19f039c1048c7144cb4412f5cc87f6, 3a19f039c1048c7144cb4412f5cc87f6 Host:ci.hostname.com Origin:https://ci.hostname.com Referer:https://ci.hostname.com/jenkins/job/test/configure User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36 X-Prototype-Version:1.7 X-Requested-With:XMLHttpRequest POST payload was empty: http://snaps.michaelwarkentin.com.s3.amazonaws.com/test_Config_Jenkins_2015-08-17_09-29-11.png

Add Comment

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-29962

Re: Found invalid crumb In the HTML header of the page should be a script section that initializes the crumb value. What value gets set there? Look for crumb.init. Could you set up a JS break point at appendToForm in hudson-behavior.js to see whether it's called repeatedly? Are you using plugins such as Simple Theme Plugin and are customizing the UI?

Add Comment

[mwarkentin@gmail.com (JIRA)]

Michael Warkentin commented on JENKINS-29962

Re: Found invalid crumb Looks like it's initialized with a single value: <script>crumb.init("crumb", "3a19f039c1048c7144cb4412f5cc87f6"); crumb.appendToForm appears to be called twice on page load. I don't believe that we have any UI customization plugins installed here are screenshots showing what plugins we've got installed / enabled: http://snaps.michaelwarkentin.com.s3.amazonaws.com/Update_Center_Jenkins_2015-08-17_10-02-36.png http://snaps.michaelwarkentin.com.s3.amazonaws.com/Update_Center_Jenkins_2015-08-17_10-02-53.png http://snaps.michaelwarkentin.com.s3.amazonaws.com/Update_Center_Jenkins_2015-08-17_10-03-07.png

Add Comment

[mwarkentin@gmail.com (JIRA)]

Michael Warkentin edited a comment on JENKINS-29962

Re: Found invalid crumb

Looks like it's initialized with a single value: {{<script>crumb.init("crumb", "3a19f039c1048c7144cb4412f5cc87f6");}} {{crumb.appendToForm}} appears to be called twice on page load.

I don't believe that we have any UI customization plugins installed  here . Here  are screenshots showing what plugins we've got installed / enabled: * http://snaps.michaelwarkentin.com.s3.amazonaws.com/Update_Center_Jenkins_2015-08-17_10-02-36.png * http://snaps.michaelwarkentin.com.s3.amazonaws.com/Update_Center_Jenkins_2015-08-17_10-02-53.png * http://snaps.michaelwarkentin.com.s3.amazonaws.com/Update_Center_Jenkins_2015-08-17_10-03-07.png

Add Comment

[mwarkentin@gmail.com (JIRA)]

Michael Warkentin commented on JENKINS-29962

Re: Found invalid crumb I noticed that there seems to be two scripts in the page source which are calling appendToForm: <script>function confirmPOST_id1957(post, href, message) { if (confirm(message)) { var form = document.createElement('form'); form.setAttribute('method', post ? 'POST' : 'GET'); form.setAttribute('action', href); if (post) { crumb.appendToForm(form); } document.body.appendChild(form); form.submit(); } return false; }</script> <a onclick="confirmPOST_id1958(true, '/jenkins/job/test/doDelete', 'Are you sure about deleting the Project ‘test’?')" class="task-link" href="#">Delete Project</a><script>function confirmPOST_id1958(post, href, message) { if (confirm(message)) { var form = document.createElement('form'); form.setAttribute('method', post ? 'POST' : 'GET'); form.setAttribute('action', href); if (post) { crumb.appendToForm(form); } document.body.appendChild(form); form.submit(); } return false; }</script>

Add Comment

[mwarkentin@gmail.com (JIRA)]

Michael Warkentin commented on JENKINS-29962

Re: Found invalid crumb Never mind, those appear to be functions which get called when clicking on various links.

Add Comment

[mwarkentin@gmail.com (JIRA)]

Michael Warkentin commented on JENKINS-29962

Re: Found invalid crumb Based on the js call stack, it's being called twice from behavior.js. startNode._each(function (node) { var list = findElementsBySelector(node, registration.selector, includeSelf); if (list.length > 0) { //console.log(registration.id + ':' + registration.selector + ' @' + registration.priority + ' on ' + list.length + ' elements'); list._each(registration.behavior); } }); list is an array with 2 elements: form and form.no-json

Add Comment

[mwarkentin@gmail.com (JIRA)]

Michael Warkentin commented on JENKINS-29962

Re: Found invalid crumb I used Ajax.Responders.register to peek into the AJAX requests being sent, and noticed that there are actually two separate crumb headers in the options: crumb and Crumb. They both contain the same crumb value. Using this hacky, hopefully temporary snippet, fixed the issue by deleting one of those crumb headers and allowed me to configure the job: Ajax.Responders.register({ onCreate: function(a){ delete a.options.requestHeaders.Crumb; } });

Add Comment

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-29962

Re: Found invalid crumb Could this be related to your customizing the crumb name? It's .crumb by default.

Add Comment

[mwarkentin@gmail.com (JIRA)]

Michael Warkentin commented on JENKINS-29962

Re: Found invalid crumb Daniel Beck We'll try removing our custom crumb name and see if that fixes things.

Add Comment

[mwarkentin@gmail.com (JIRA)]

Michael Warkentin commented on JENKINS-29962

Re: Found invalid crumb Hey Daniel Beck, looks like things are working without the custom crumb name, however I took a look at the request headers, and we're still sending the extra Crumb header - just that Jenkins isn't using it anymore. Let me know if you have any ideas for figuring out where that's coming from.

Add Comment

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-29962

Re: Found invalid crumb Michael Warkentin No idea. Maybe a plugin you installed? A user-script in your browser?

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick updated an issue

Jenkins / JENKINS-29962 Found invalid crumb

Change By: Jesse Glick Labels: crumb gui jenkins

Add Comment

[DZwarg@massmutual.com (JIRA)]

David Zwarg commented on JENKINS-29962

Re: Found invalid crumb Encountering the same problem. "deadbeef" is the sanitized version of my crumb in all cases below. I'm using Postman to test requests. In one tab, I get a crumb: GET https://(jenkins)/crumbIssuer/api/xml?xpath=//crumb --- <crumb>deadbeef</crumb> Then I use that identical crumb to issue a request (in this case, to a pipeline input): POST https://(jenkins)/job/(pipeline name)/(job number)/input/(input id)/submit Jenkins-Crumb=deadbeef proceed=Proceed --- HTTP 403 Title: Error 403 No valid crumb was included in the request And I can verify that the crumb is being passed through all proxies to get there. In the logs, I can see the crumb: Nov 18, 2016 2:35:36 PM WARNING hudson.security.csrf.CrumbFilter doFilter Found invalid crumb deadbeef. Will check remaining parameters for a valid one... Nov 18, 2016 2:35:36 PM WARNING hudson.security.csrf.CrumbFilter doFilter No valid crumb was included in request for /job/(pipeline name)/(job number)/input/(input id)/submit. Returning 403. I can verify that if I omit the crumb, the error doesn't mention anything about invalid crumbs: Nov 18, 2016 2:38:26 PM WARNING hudson.security.csrf.CrumbFilter doFilter No valid crumb was included in request for /job/(pipeline name)/(job number)/input/(input id)/submit. Returning 403. Please advise.

Add Comment

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

[DZwarg@massmutual.com (JIRA)]

David Zwarg commented on JENKINS-29962

Re: Found invalid crumb Okay, I hope this will help someone in the future. After much experimentation (and red herrings by reading the logs), I discovered that I had to log in with my credentials (using Basic Auth) to make the request work. Also, I needed to submit it like a form, not url encoded. The full request that worked is: POST /job/(job name)/(job number)/input/(input id)/proceedEmpty HTTP/1.1 Host: (jenkins) Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryBUILDR4EVA Jenkins-Crumb: deadbeef Authorization: Basic (encoded "username:password") ------WebKitFormBoundaryBUILDR4EVA Content-Disposition: form-data; name="proceed" Proceed ------WebKitFormBoundaryBUILDR4EVA

Add Comment

[sajj.farahani@nab.com.au (JIRA)]

Sajj Farahani commented on JENKINS-29962

Re: Found invalid crumb I still have this issue. Any Idea how I may fix this? I saw David Zwarg response. However, I am new to Jenkins and that response does not make much sense to me. Any assistance will be greatly appreciated.

Add Comment