[JIRA] (JENKINS-54495) Better handling of GitHub Organization folder scan to avoid API quota

[leandro.lucarella@sociomantic.com (JIRA)]

Leandro Lucarella created an issue

Jenkins / JENKINS-54495 Better handling of GitHub Organization folder scan to avoid API quota Issue Type: Improvement Assignee: Unassigned Components: github-branch-source-plugin Created: 2018-11-06 15:32 Environment: Jenkins ver. 2.138.2 GitHub Branch Source Plugin 2.4.1 Labels: performance user-experience github api quota Priority: Major Reporter: Leandro Lucarella When having a big GitHub organization, with hundreds of repos, each with hundreds of branches and tags, refreshing the whole organization is not possible (or it takes ages) due to GitHub API quota being hit. This is particularly bad when trying to add a new repo, it could take days, which is completely impractical. There are several solutions to this issue that I can think of: Use GitHub GraphQL API to query the whole thing in one (or very few) request(s) Make a "shallow scan", that only discovers repos. Then each repo can be refreshed separately, which can 1. enable the quick addition of new repos and 2. distribute the refresh API bursts in time making hitting the API quota less likely Add a separate function to only discover one repo specified by the user Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[bitwiseman@gmail.com (JIRA)]

Liam Newman commented on JENKINS-54495

Re: Better handling of GitHub Organization folder scan to avoid API quota Any of these seems like interesting options. Targeted scan - This would involve some working with Jelly and Jenkins UI, but it might be easier to implement due to the narrow target. Shallow scan - At very least, doing a breadth first scan that then requested scans from the child repo's over time. The GraphQL option would be a massive undertaking. But maybe if you switched just to top level repo scan or some other targeted scenario. They're all viable in different ways. Perhaps you could file a separate issue for each one and then work on them separately?

Add Comment

[kivagant@gmail.com (JIRA)]

Eugene G commented on JENKINS-54495

Re: Better handling of GitHub Organization folder scan to avoid API quota

This is very important. If an organization has tons of old repos and more than 50 repos with Jenkinsfiles it becomes crazy to maintain the list of repos in this small filter field.

Add Comment