troubleshooting stash slowness

[Nathaniel Irons]

Jenkins is building an iOS project with five similarly-sized targets.
The git repository is 251 MB, of which the .git dir is 130 MB. A
pipeline stage handles the checkout, injects build numbers, and stashes
the modified sources, minus the unnecessary .git dir, for the other
build nodes to consume.

stage ("Checkout and modify") {
agent { label 'xcode-9' }
steps {
checkout scm
sh ("agvtool new-version -all $BUILD_ID")
stash (excludes: '.git', name: "modified-sources")
}
}

This stash operation consistently takes 15 minutes. Unstashing it on any
of the build nodes then takes about ten minutes per node. Something is
wrong, but I'm not sure what steps to take.

The build environment is vsphere on an older mac pro, with some Mac VMs
for Xcode and a (dockerized) jenkins master in a linux VM. The storage
is all SSD, and we're not seeing any issues with I/O performance other
than this stash/unstash problem. Zipping this same source directory
takes six seconds, and tar+bzip2 takes 15 seconds.

An individual build of this project runs in under three minutes, so the
unstash alone quadruples our build times. Any suggestions for how to
investigate?

Thanks,

-nat

[Mark Waite]

On Thu, Dec 14, 2017 at 4:13 PM Nathaniel Irons <na...@bumppo.net> wrote:

Jenkins is building an iOS project with five similarly-sized targets.
The git repository is 251 MB, of which the .git dir is 130 MB. A
pipeline stage handles the checkout, injects build numbers, and stashes
the modified sources, minus the unnecessary .git dir, for the other
build nodes to consume.

stage ("Checkout and modify") {
        agent { label 'xcode-9' }
        steps {
                checkout scm
                sh ("agvtool new-version -all $BUILD_ID")
                stash (excludes: '.git', name: "modified-sources")
        }
}

This stash operation consistently takes 15 minutes. Unstashing it on any
of the build nodes then takes about ten minutes per node. Something is
wrong, but I'm not sure what steps to take.

Unfortunately, I don't have a solution to offer.

The online help for stash (in the Pipeline syntax when "Stash - Stash some files to be used later in the build" is selected), hints that you may be at the limits of the stash implementation.  It says:

Note that the stash and unstash steps are designed for use with small files. For large data transfers, use the External Workspace Manager plugin, or use an external repository manager such as Nexus or Artifactory. This is because stashed files are archived in a compressed TAR, and with large files this demands considerable on-master resources, particularly CPU time. There's not a hard stash size limit, but between 5-100 MB you should probably consider alternatives.

Mark Waite

 

The build environment is vsphere on an older mac pro, with some Mac VMs
for Xcode and a (dockerized) jenkins master in a linux VM. The storage
is all SSD, and we're not seeing any issues with I/O performance other
than this stash/unstash problem. Zipping this same source directory
takes six seconds, and tar+bzip2 takes 15 seconds.

An individual build of this project runs in under three minutes, so the
unstash alone quadruples our build times. Any suggestions for how to
investigate?

Thanks,

  -nat

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1513293192.3586726.1205509440.0B960302%40webmail.messagingengine.com.
For more options, visit https://groups.google.com/d/optout.

[Nathaniel Irons]

When a stash is being performed on a non-master agent, is the stash operation really CPU-intensive on the master? I wasn't sure from that description whether it assumed the master was also executing the stash phase. We did try tripling the CPU allocated to the master's VM, and saw no change in stash performance.

As recently as three weeks ago, this same stash pipeline stage on the same repository was executing in 30-45 seconds. The repo's only grown fractionally since then. If I'm at the limit of the stash implementation, I hit a heck of a wall.

Thanks,

  -nat

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtE0k5MswmNzRhTRfcHCdq3Ry%2BXbognyX6km1GQ_c8-eEw%40mail.gmail.com.

[Mark Waite]

On Thu, Dec 14, 2017 at 6:53 PM Nathaniel Irons <na...@bumppo.net> wrote:

When a stash is being performed on a non-master agent, is the stash operation really CPU-intensive on the master? I wasn't sure from that description whether it assumed the master was also executing the stash phase. We did try tripling the CPU allocated to the master's VM, and saw no change in stash performance.

As recently as three weeks ago, this same stash pipeline stage on the same repository was executing in 30-45 seconds. The repo's only grown fractionally since then. If I'm at the limit of the stash implementation, I hit a heck of a wall.

That sounds like something else has changed.  If it was working well before, and has only grown a little since then, it seems like it is worth investigating other things that might have changed.

Has the disc I/O capability on the master or the agent changed significantly?

If you perform a smaller stash from the same workspace, does it perform significantly better?

Are there competing processes which might be slowing throughput?

Has the Jenkins memory use increased significantly?

Mark Waite

 

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1513302770.3633224.1205623960.21FCC9DC%40webmail.messagingengine.com.