[JIRA] (JENKINS-42450) Efficiently add a single repo to a github org without requiring a full scan

[mneale@cloudbees.com (JIRA)]

Michael Neale created an issue

Jenkins / JENKINS-42450 Efficiently add a single repo to a github org without requiring a full scan Issue Type: Improvement Assignee: Kohsuke Kawaguchi Components: blueocean-plugin, github-branch-source-plugin, github-organization-folder-plugin Created: 2017/Mar/03 2:08 AM Priority: Minor Reporter: Michael Neale In some cases when you are using a github org folder, you know you want to add one specific repository to it (it may be brand new for example, or a user has changed the pattern). In this case, it would be nice to not do a full org scan but instead just look for that specified repository, and save the api quota. Add Comment

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

[mneale@cloudbees.com (JIRA)]

Michael Neale assigned an issue to Stephen Connolly

Jenkins / JENKINS-42450 Efficiently add a single repo to a github org without requiring a full scan

Change By: Michael Neale Assignee: Kohsuke Kawaguchi Stephen Connolly

Add Comment

[jdumay@cloudbees.com (JIRA)]

James Dumay updated an issue

Jenkins / JENKINS-42450 Efficiently add a single repo to a github org without requiring a full scan

Change By: James Dumay Sprint: 1.0

Add Comment

[jdumay@cloudbees.com (JIRA)]

James Dumay commented on JENKINS-42450

Re: Efficiently add a single repo to a github org without requiring a full scan Vivek Pandey Stephen Connolly it would be fantastic if we could get a resolution for this one. What we are doing today is quite slow

Add Comment

[stephen.alan.connolly@gmail.com (JIRA)]

Stephen Connolly commented on JENKINS-42450

Re: Efficiently add a single repo to a github org without requiring a full scan

You should be firing an event through the SCMEvent API.

Add Comment

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)

[jdumay@cloudbees.com (JIRA)]

James Dumay commented on JENKINS-42450

Re: Efficiently add a single repo to a github org without requiring a full scan

Stephen Connolly could you give us an example of how to do that please? cc Vivek Pandey

Add Comment

[mneale@cloudbees.com (JIRA)]

Michael Neale assigned an issue to Vivek Pandey

Jenkins / JENKINS-42450

Efficiently add a single repo to a github org without requiring a full scan

Change By: Michael Neale Assignee: Stephen Connolly Vivek Pandey

Add Comment

[vivek.pandey@gmail.com (JIRA)]

Vivek Pandey started work on JENKINS-42450

Change By: Vivek Pandey Status: Open In Progress

Add Comment

[vivek.pandey@gmail.com (JIRA)]

Vivek Pandey commented on JENKINS-42450

Re: Efficiently add a single repo to a github org without requiring a full scan Stephen Connolly I guess I need github-branch-source-2.0.4-beta-1? Because in 2.0.3, I don't see this optimization as its calling GitHub.listRepositories(). So we need to do both, create single repo in OrganizationFolder and update to OrganizationFolder. For creation, I think I should be using SCMSourceEvent and for update SCMNavigatorEvent, right? An example of each use case will be very helpful as its not clear what would go for example as payload when creating an SCMEvent. Looks like I may have to create GHRepository to use GHEventPayload.Create, but then I need to create instance of GHRepository, right? this will be few round trips to GitHub, but I guess it will be used as it is later on? I think it will be lot simpler if you could provide samples on how to go about it.

Add Comment

[stephen.alan.connolly@gmail.com (JIRA)]

Stephen Connolly commented on JENKINS-42450

Re: Efficiently add a single repo to a github org without requiring a full scan

No, you are over thinking.   Just create your own implementation of SCMHeadEvent. Have the isMatch methods match your project only and return an SCMHead with a dummy SCMRevision. Then rely on the fact that we don't trust events - only using them as a hint. The SCMHeadEvent will trigger creation of a SCMSource providing that the SCMHead has the Jenkinsfile and then the SCMSource will trigger creation of the branch project for you.   Now if you need to create the project before it has a Jenkinsfile in SCM... well that's a different story.

Add Comment

[vivek.pandey@gmail.com (JIRA)]

Vivek Pandey commented on JENKINS-42450

Re: Efficiently add a single repo to a github org without requiring a full scan

Ok, thanks. I will give it a try.

Add Comment

[vivek.pandey@gmail.com (JIRA)]

Vivek Pandey commented on JENKINS-42450

Re: Efficiently add a single repo to a github org without requiring a full scan

Stephen Connolly So SCMHeadEvent for multi-branch project where SCMHead represents a branch. What would be equivalent for repo, SCMSourceEvent? That is when we are creating organization folder from GitHub org and we want to scan only one repo.

Add Comment

[vivek.pandey@gmail.com (JIRA)]

Vivek Pandey edited a comment on JENKINS-42450

Re: Efficiently add a single repo to a github org without requiring a full scan

[~stephenconnolly] So SCMHeadEvent for multi-branch project where SCMHead represents a branch. What would be equivalent for I need to fire an event to scan just one repo , SCMSourceEvent? That is when we are creating /updating a GitHub organization folder from GitHub org and we want . In this case, I don't have branch related info to scan only one repo create SCMHead .

Add Comment

[stephen.alan.connolly@gmail.com (JIRA)]

Stephen Connolly commented on JENKINS-42450

Re: Efficiently add a single repo to a github org without requiring a full scan

Vivek, the Org Folder will process the SCMHeadEvent: https://github.com/jenkinsci/branch-api-plugin/blob/d7f50ea7911a5363ed1972ffe7d5c5576789e968/src/main/java/jenkins/branch/OrganizationFolder.java#L924

Add Comment

[vivek.pandey@gmail.com (JIRA)]

Vivek Pandey commented on JENKINS-42450

Re: Efficiently add a single repo to a github org without requiring a full scan

Stephen Connolly Ok, I tried implementing it in PR https://github.com/jenkinsci/blueocean-plugin/pull/893. It doesn't work though, repo is fetched but MBP scan doesn't happen. MultibranchProjectFactory.recognize() returns false. Reason being that scmSource.fetch().getRevision() returns null. Since no factory is found the MBP is not created even though there is Jenkinsfile. Maybe there is something missing, PTAL. Also noticed, If there are multiple Jenkins organization folders for a given GitHub org, this repo is fetched for each org folder as SCMHeadEvent.isMatch(navigator) returns true as navigators match. Wonder if there is way to avoid this unnecessary fetch of this repo for each existing org folders for given GitHub org?

Add Comment

[vivek.pandey@gmail.com (JIRA)]

Vivek Pandey commented on JENKINS-42450

Re: Efficiently add a single repo to a github org without requiring a full scan

According to chat with Stephen Connolly, for what we are doing, that is org/repo level, SCMSourceEvent is more appropriate. Implementing using SCMSourceEvent.

Add Comment

[vivek.pandey@gmail.com (JIRA)]

Vivek Pandey updated JENKINS-42450

Jenkins / JENKINS-42450

Efficiently add a single repo to a github org without requiring a full scan

Change By: Vivek Pandey Status: In Progress Review

Add Comment

[vivek.pandey@gmail.com (JIRA)]

Vivek Pandey updated JENKINS-42450

Jenkins / JENKINS-42450 Efficiently add a single repo to a github org without requiring a full scan

Change By: Vivek Pandey Status: In Review Resolved Resolution: Fixed

Add Comment

[leandro.lucarella@sociomantic.com (JIRA)]

Leandro Lucarella commented on JENKINS-42450

Re: Efficiently add a single repo to a github org without requiring a full scan Any news on this one? I'm bumping into this problem, specially because scanning the full org will make Jenkins hit the API quota limit.

Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[leandro.lucarella@sociomantic.com (JIRA)]

Leandro Lucarella commented on JENKINS-42450

Re: Efficiently add a single repo to a github org without requiring a full scan

BTW, this is marked as resolved but I can't find what's the solution and how to use it.

Add Comment

[vivek.pandey@gmail.com (JIRA)]

Vivek Pandey commented on JENKINS-42450

Re: Efficiently add a single repo to a github org without requiring a full scan

This was fixed long time back. PR https://github.com/jenkinsci/blueocean-plugin/pull/893. However, pipeline creation using GitHub org folder was removed in later version of blue ocean. Basically, there is no pipeline creation using Github org folder in blueocean now.

Add Comment

[leandro.lucarella@sociomantic.com (JIRA)]

Leandro Lucarella commented on JENKINS-42450

Re: Efficiently add a single repo to a github org without requiring a full scan

How was this fixed exactly? I still can't find any way to trigger a discovery on a new project without going through a whole organization scan, which for us is horribly slow and it exhausts all the GitHub API rate limit. Each time a new project is created, we basically have to wait for a day to see it in Jenkins...

Add Comment