Copy Artefact between master and slave node is very slow. (Windows Jenkins version 2.107.3)

[panpaliamahen]

Hi,

I am using Jenkins version 2.107.3, latest one and on windows. I am using
Jenkins Slave node to run a Job which require to copy an artefact of size
~6GH. And we found when jenkins copy this it takes ~10 minutes. (Note
artefact are copied from master).

Also observed it takes ~10 minutes when jenkins job archived back to master.

Where as same can be copied in ~1 minute in same network without Jenkins.

Please can someone help me?
What should I do?
Do I need to install/upgrade any additional plugin on Jenkins?

Please it is very urgent and reducing efficiency with in our development
teams.

Thanks and regards,
Mahendra




--
Sent from: http://jenkins-ci.361315.n4.nabble.com/Jenkins-users-f361316.html

[Mark Waite]

I thought I remembered reading advice that archiving artifacts was known to be slow with large files.  I couldn't find the reference, so I may be incorrect in this case.

I'd suggest using a different technique to store your large artifacts, rather than having Jenkins perform the copy.  If on Unix, consider rsync or other copy program.  If on Windows, consider robocopy.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1527665457279-0.post%40n4.nabble.com.
For more options, visit https://groups.google.com/d/optout.

[niristotle okram]

 Copying files from jenkins workspace , is always slow. I have seen that in the past. 

If u r using jenkins to checkout an artifact, and then copy to a path on a remote. Just use wget/rsync (as Mark suggested) . U can trigger that from jenkins too

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtHYp8p3civwUYu4_pAyi63PB4O8exWjt4YS0qimgLa8tg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Sent from mobile device, excuse typos if any.

[Tim Black]

Yes, archiving artifacts is very slow, this has apparently always been the case with Jenkins - there are numerous jira issues that have come and been resolved without fixes to this. It is what it is..

As I commented here, I'm also seeking massive performance gains by replacing copyArtifact with a shell call in my pipelines. In lieu of installing a proper artifact management system and replacing all archive/copyArtifact with calls to its REST API (which I'll be doing later this year), I'm hoping to find a quick alternative. 

HTTP/wget/curl is problematic when you want to fetch anything but a single artifact or all artifacts, bc HTTP doesn't support a notion of a directory, so you have to fetch the index and preprocess before fetching what you really want. With scp I could just use unix glob pattern matching to fetch what I desire in a single simple call.

HTTP/wget/curl is also problematic bc you have to use Jenkins API tokens and authentication. I'm using ansible to setup my jenkins infra inside a firewalled LAN, and the jenkins users on all nodes are already set up to be able to freely ssh back and forth without password. 

So, SCP would be a slam dunk for me if I could construct the source path correctly. The problem is that Jenkins uses an algorithm to create a unique folder name, both for workspace names, and for branch job names, but I'm not sure if that's consistent, and therefore I do not know if it would be safe to attempt to re-construct and reference job paths on the controller's disk.

E.g. to fetch artifacts from the corresponding branch of an upstream multibranch pipeline job with Full project name of "ProjectFolder/MyProject/feature%2Ffoo", in the downstream multibranch pipeline, I would do something like:

scp -r jenkins-controller:<JENKINS_HOME>/jobs/ProjectFolder/jobs/MyProject/branches/<HOW_DO_I_COMPUTE_THE_BRANCH_PORTION_OF_PATH?>/lastSuccessfulBuild/artifact/<GLOB>

[Tim Black]

To whom it may concern, I ended up finding the code in Jenkins branch-api plugin that's creating that branch path segment (the NameMangler), however it turned out to be completely unnecessary since I can just get the build directory (to construct the path to the artifacts on the controller) from the project/job object, obtained by name from the jenkins instance in groovy. So my shared library function is even simpler now, works for any project type and safer because it doesn't need to use any core or plug in classes.