Download the git repo and use it for the downstream job

[sravan]

Hi All,

My git repo is really big and I am running job parallel on two different machine.Sometimes I see an issue with downloading from git . I get this error "failed on download from git" Error.

ERROR: Timeout after 10 minutes
ERROR: Error cloning remote repo 'origin'
returned status code 128:
stdout: 
stderr: remote: Enumerating objects: 522, done.        
remote: Counting objects:   0% (1/522)        
remote: Counting objects:   1% (6/522)        

Could anyone help me with this please

[Nitish Goel]

Have you tried git lfs?
If you have large files, then git lfs is your solution.

Thanks,
Nitish Goel

[Gianluca]

Hi,

it's hard to help with so little context.

For big what you mean? what is the actual size of the repository?

Git .. exactly where is the remote? GitHub, Gitlab, other?

The error is intermittent or constant? (i.e. could a simple retry do the trick?)

[Mark Waite]

You might watch https://youtu.be/jBGFjFc6Jf8?t=6434 and https://youtu.be/TsWkZLLU-s4?t=138 in case they contain hints that might help in your case.

Those videos suggest that you might:

• Use a reference repository on the agent machine to reduce data transfer between the git server and the agent
• Narrow the refspec in the job definition to request only the items you need
• Use a shallow clone to request less history of the repository

You might also consider placing the git server "nearer" to the agents on the network.

You might also assess the capabilities of the git server as suggested by Gianluca.  If it is heavily loaded while providing the data, then it may need better I/O services or more processor power or more network bandwidth.

You might also confirm that the git repository on the git server is not needing a garbage collection run.

Many opportunities to tune and improve at many different locations.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/113403d9-abd0-4d73-9ec0-6b684e893315o%40googlegroups.com.

[Ivan Fernandez Calvo]

we have a similar use case, where we need to checkout a repo and download a bunch of dependencies, then we have to share this with several Agents. Because stash and unstash will kill the performance of our Jenkins instance we use a custom step similar to stash/unstash that uses a GCP bucket to store the repo and the dependencies, it adds a little time to make the checkout because we have to upload everything to the bucket, but increases a lot the next checkouts+dependencies.

If you only need the source code, you could use a reference repo. basically your Agents have the code (or part of it) before they make the checkout.