[JIRA] (JENKINS-61936) AWS Artifact-Manager-S3-plugin uploads in series not parallel

beryl.snyder@factset.com (JIRA)

unread,

Apr 16, 2020, 5:05:02 PM4/16/20

to jenkinsc...@googlegroups.com

Beryl Snyder created an issue

Jenkins /

JENKINS-61936

AWS Artifact-Manager-S3-plugin uploads in series not parallel

Issue Type:	Improvement
Assignee:	Unassigned
Components:	artifact-manager-s3-plugin
Created:	2020-04-16 21:04
Environment:	Artifact-Manager-S3-plugin Version
Labels:	plugin performance
Priority:	Minor
Reporter:	Beryl Snyder

Looking at https://github.com/jenkinsci/artifact-manager-s3-plugin and the uploads appear to happen in series

for 10K, 5KB files upload time is around 20 mins, for a 100MB zip file it is a few second.

I’ve tried to fix it, but I’m not a Java developer or have any Jenkins plugin development experience , and have no idea what I’m doing.

The original line,

https://github.com/jenkinsci/artifact-manager-s3-plugin/blob/master/src/main/java/io/jenkins/plugins/artifact_manager_jclouds/JCloudsArtifactManager.java#L189

for (Map.Entry<String, URL> entry : artifactUrls.entrySet())

{ client.uploadFile(new File(f, entry.getKey()), contentTypes.get(entry.getKey()), entry.getValue(), listener); }

My change,

Map.Entry<String, URL> entry : artifactUrls.entrySet().parallelStream().forEach(

client.uploadFile(new File(f, entry.getKey()), contentTypes.get(entry.getKey()), entry.getValue(), listener));

Add Comment

This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)

hbetts@factset.com (JIRA)

unread,

Apr 16, 2020, 5:37:03 PM4/16/20

to jenkinsc...@googlegroups.com

Hutson Betts stopped work on

JENKINS-61936

Change By:	Hutson Betts
Status:	In Progress Open

Add Comment

hbetts@factset.com (JIRA)

unread,

Apr 16, 2020, 5:37:03 PM4/16/20

to jenkinsc...@googlegroups.com

Hutson Betts started work on

JENKINS-61936

Change By:	Hutson Betts
Status:	Open In Progress

Add Comment

hbetts@factset.com (JIRA)

unread,

Apr 16, 2020, 5:43:02 PM4/16/20

to jenkinsc...@googlegroups.com

Hutson Betts commented on

JENKINS-61936

Re: AWS Artifact-Manager-S3-plugin uploads in series not parallel

As an example, teams I work with often set the Archive Artifacts include list to be `.`. They archive hundreds, and quite often, thousands of small files consisting of compiled assets; JavaScript, CSS, images, etc.

There is a often cited solution to pre-zip files into a single archive file before using the Archive Artifacts feature to store the artifact.

However, in practice, we find the extra step of pre-zipping files to be a point of friction/contention with our user base.

As AWS S3 allows a high rate of simultaneous uploads, it seems reasonable to avoid burdening our users with the added friction, while also leveraging S3's support for a large number of simultaneous uploads.