Publishing many small files to raw repository is slow

[Christian Ward-Garrison]

Hi,

I have a raw repository that I'm using to serve the Javadoc for my project. Once the files are published, it works great!

The problem? It takes over 17 minutes to publish the 3525 documents that comprise the Javadoc. I'm using the REST API, so that's 3525 authenticated PUTs.

How can I make the process faster? The total size of the Javadoc is only 117 MB, so it's definitely the individual PUTs that are slowing me down. Is there a better way to transfer all of those files? If I recall correctly, there was a plugin for NXRM 2 that allowed you to upload an archive. Then, Nexus would extract the archive and add each file within to the repo individually. Only one PUT! Doesn't something like that exist for version 3?

Thanks,

Christian

[ds...@sonatype.com]

Hi Christian,

You can try uploading multiple files at per call as this is supported for raw repositories.
Here's the documentation that may help you: https://help.sonatype.com/display/NXRM3/Components+API#ComponentsAPI-Raw

[Peter Lynch]

On Fri, Apr 20, 2018 at 7:42 PM, Christian Ward-Garrison <cwar...@ucar.edu> wrote:

Hi,

I have a raw repository that I'm using to serve the Javadoc for my project. Once the files are published, it works great!

The problem? It takes over 17 minutes to publish the 3525 documents that comprise the Javadoc. I'm using the REST API, so that's 3525 authenticated PUTs.

You are describing something that is much slower than what we would normally expect. However this can depend on a lot of factors - too many to enumerate on this mailing list.

If you can reproduce the problem, please file an issue in the NEXUS project at https://issues.sonatype.org and include a support zip generated from your instance after the deployment of these javadoc files completes.

 

How can I make the process faster? The total size of the Javadoc is only 117 MB, so it's definitely the individual PUTs that are slowing me down. Is there a better way to transfer all of those files? If I recall correctly, there was a plugin for NXRM 2 that allowed you to upload an archive. Then, Nexus would extract the archive and add each file within to the repo individually. Only one PUT! Doesn't something like that exist for version 3?

Thanks,

Christian

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/5dcbf932-f0f6-477b-9031-5c68cf136aaf%40glists.sonatype.com.
For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.

[jhe...@sonatype.com]

Depending on how deep you want to go, I could likely help lead you down the path of creating a plugin to accomplish that. I'm sure it would be well received in the community. Let me know if you (or anyone) would like to do that!

[Christian Ward-Garrison]

Hi Dawid,

This did the trick for me, thanks. Constructing the multipart POST requests was a little tricky, but the OkHttp client (http://square.github.io/okhttp/) was up to the task.

Cheers,

Christian

--
You received this message because you are subscribed to a topic in the Google Groups "Nexus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/a/glists.sonatype.com/d/topic/nexus-users/5Ww7rOm0yBE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nexus-users+unsubscribe@glists.sonatype.com.

To post to this group, send email to nexus...@glists.sonatype.com.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/e5eeba3b-4aa7-48b2-b1e4-5210c6b5d503%40glists.sonatype.com.

[Christian Ward-Garrison]

Hi Peter,

After a bit more investigation, I believe that the long upload times were caused by erratic I/O performance of the machine that Nexus is running on. When I performed the same benchmark against a Nexus instance running locally, the upload (using individual PUTs) completed in about 40 seconds.

So yeah, this issue probably has nothing to do with the Nexus software.

Cheers,

Christian