GSoC 2020 idea: Git plugin automatic caching for faster clone

[Mark Waite]

Faster git fetch performance using existing local workspaces as a cache

The Jenkins git plugin clones remote git repositories into Jenkins workspaces on agents.  The Jenkins agents frequently have other copies of the remote git repository which could be used as a local cache to reduce network data transfer from the remote git server to the local workspace. Those local copies could be in workspaces for previous builds of this job or they could be in workspaces for other branches of the same repository. Those local copies might also be maintained by administrators with periodic updates from the central repository.

Identify and use a well-chosen existing copy of the remote git repository as a reference repository for the git fetch that is being performed in the workspace. Default to dissociate the workspace repository from the reference repository during the fetch so that deletion of the reference repository will not harm the workspace repository.

Refer to https://docs.google.com/document/d/1M9Na1eJxwGZ7fY1L-PsBGv62jL_pPBtqmGePpwogUBw/edit# as a draft location for the project proposal and for refinement of the proposal.

Mark Waite

[RAJAT BANSAL]

Hi Mark and others,

I am Rajat Bansal, a 4th-year Computer Science and Applied Mathematics student at IIIT Delhi and would love to contribute to an esteemed organization like Jenkins. I recently joined the mailing list and really liked the idea you have proposed above. I would like to be assigned to this task if possible however I think that it's already been done since I found this. Also, I am planning to apply for GSoC 2020 under Jenkins and would like to know about any other pending projects/issues to get familiar with the code-base. 

Looking forward to some help and guidance. 

Regards,

Rajat Bansal  

[Mark Waite]

Hi Rajat!

Thanks for your interest.  That's great!

The project definitely has not been done.  Using a reference repository as described in that article requires that the reference repository be updated by some process, usually outside Jenkins.  That is a lot of work that the Jenkins user or administrator should not need to do.  The project idea is seeking ways to make the use of reference repositories much more automatic and much more effective for Jenkins users.

I'm personally inexperienced with how the Google Summer of Code student selection process works.  I believe it is described at https://google.github.io/gsocguides/mentor/selecting-a-student .  I know that the Google Summer of Code subproject of Jenkins has lots of experience with the process and will help and coach as projects are discussed, evaluated, refined, selected, and assigned. 

If you're interested in starting on a project even before being selected as a Google Summer of Code student, you may be better served by starting with one of the "friendly issues" in the Jenkins project.  Friendly issues will give you experience with the Jenkins project and its infrastructure and take much less time than the Google Summer of Code project ideas.  

The participate/code page and an excellent blog post on becoming a Jenkins contributor will get you started.  The Jenkins contributing page also provides a link to friendly issues or you can use the two dimensional filter to find friendly issues by project and severity of the issue.

--
You received this message because you are subscribed to the Google Groups "jenkinsci-gsoc-all-public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-gsoc-all...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-gsoc-all-public/0905c63c-09c5-44d6-b192-3e39baf5323f%40googlegroups.com.

--

Thanks!

Mark Waite

[Jeff]

Hi Rajat.

I'm one of the org admins for GSoC in Jenkins. I appreciate your enthusiasm, and I hope you'll apply when we're ready to start taking applications in a few months.

At this point, we're not ready to accept applications. We expect to start engaging with students February 20, 2020, and the application period will be March 16 - 31. You can consult the GSoC Timeline for full information.

I encourage you to get involved in one of the "newbie friendly issues" to get acquainted with the Jenkins project.  The ultimate goal of GSoC is to encourage students to get involved in open source, so we'd definitely welcome any contributions.

Best 

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-gsoc-all-public/CAO49JtFc69-gZ9%2BPoOCpvmxemi7RE%3DJu2Y2shhr0Da%2BdhUKLbQ%40mail.gmail.com.

[RAJAT BANSAL]

Thanks a lot Mark and Jeff for the guidance. I will start working on some of the friendly issues then to get better acquainted with Jenkins and hope to be of much help. 

Regards,

Rajat Bansal

[rishabhb...@gmail.com]

Before asking questions regarding this project, please be patient with my understanding of this project. 

Each SCMHead and SCMRevision has a SCMFileSystem associated to it. This Filesystem is created by locking the git repository using a cache. 
The objective of this project seems to use that cache as a reference repository (in the local machine:agent node) which would basically reduce network and local storage costs as fewer objects would have to be copied from the repo being cloned.

My queries regarding the project are: 

1) The parameter to choose an existing copy of the remote git repo is how many git objects the "local copy" and the remote git repo share? The repo which shares the most would be the latest or the preferable candidate for this job?

2) I have seen the PR: 502. The comment made by Jesse Glick about Mercurial Plugin's way of storing two levels of cache -- on the master and on the agents seems like a good way to implement caching for the git-plugin. Would you consider that as a good choice?

Also, I want to make the case of creating this functionality into a separate extension plugin which then can be an opt-in feature as using cache could be a potential downside that may affect some users.

[Gyanesha Prajjwal]

Hi Everyone,

I am Gyanesha Prajjwal. I am really interested to contribute to Jenkins and work for this esteemed organisation in GSoC 2020. 

I went through the ideas list and I found "Git Repository Caching" really interesting. 

Before submitting any proposal, I would really like to get familiarised with the Jenkins code-base. I have made some pull requests on git-plugin and git-client-plugin before that I have contributed to jenkinsci/analysis-model and jenkinsci/warnings-ng-plugin. 

Also, I would like to understand more about the aforementioned project.

Looking forward for some guidance.

Thank you and Regards,

Gyanesha Prajjwal

[Marky Jackson]

Thank you and welcome!

If you would like to get familiar with the Jenkins core codebase, do look at the following areas:

Jenkins core repository: https://github.com/jenkinsci/jenkins

Good first issues: https://github.com/search?q=org%3Ajenkinsci+org%3Ajenkins-infra+is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22

Beginners guide to contributing: https://wiki.jenkins.io/display/JENKINS/Beginners+Guide+to+Contributing#BeginnersGuidetoContributing-Areyouinterestedinwritingcode%3F

Contributing to Jenkins: https://docs.google.com/presentation/d/1JHgVzWZAx95IsUAZp8OoyCQGGkrCjzUd7eblwd1Y-hA/edit#slide=id.p

Hope this helps and good luck!

--
You received this message because you are subscribed to the Google Groups "jenkinsci-gsoc-all-public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-gsoc-all...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-gsoc-all-public/5d83fa14-4c97-4d51-b5d1-955146257370%40googlegroups.com.