sporatic git-path Pluggable SCM error

[Sifu Tian]

Hi Folks,

I keep getting this error on various pipelines.  Its not consistent but frequent enough to be annoying.

I confirmed the materials are parsed and up to date.

Any thoughts on why this is happening?

Material qa.cypress.automation checkout failed: The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: '"checkout failed due to [RuntimeException: Exception (Process exited with an error: 128 (Exit value: 128)) Occurred: [git, clone, --branch=master, https://github.com/Cypressrx/cypress, /go/pipelines/Cypress-Qa-Build] - null], rootCause [ExecuteException: Process exited with an error: 128 (Exit value: 128)]"'

The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: '"checkout failed due to [RuntimeException: Exception (Process exited with an error: 128 (Exit value: 128)) Occurred: [git, clone, --branch=master, https://github.com/cypressrx/cypress, /go/pipelines/Cypress-Qa-Build] - null], rootCause [ExecuteException: Process exited with an error: 128 (Exit value: 128)]"'

[Chad Wilson]

In my experience (as maintainer of that plugin and a frequent user until a year or so ago) those errors are usually caused by networking issues or rate limits on the remote repository manager.

Unfortunately the plugin does not report back to the GoCD API the details of what is going on so that it can appear in the server console logs - so you get these generic errors that the process fork returned an error code, but not the stderr or log stream which is not so helpful.

See https://github.com/TWChennai/gocd-git-path-material-plugin/issues/26 for the context and a suggested workaround or check out the plugin-specific log files from an agent when it happens (it's usually a good idea to look in the plugin repo GitHub issues for plugin-specific challenges)

What you may find if you have many materials using this plugin in a polling mode and many parallel jobs is that you are getting rate limited by GitHub. Just a guess though.

-chad

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/64296719-5516-4ba1-bf27-5e274c011b55n%40googlegroups.com.

[Sifu Tian]

Hi Chad,

Thanks for the response as it was extremely helpful.  We recently moved to CodeCommit and I suspect we are being throttled.

I will make some adjustments and test.

Thanks again!

[Chad Wilson]

I previously used CodeCommit and it was very problematic with the git path plugin (as each path combination is a separate material) in particular due to its rate limiting.

We found we needed to restructure pipelines to avoid lots of parallel clones, particularly parallel jobs which all kick off at the same time and clone the same repo in parallel. We did things like using shallow clones to reduce requests, restructuring some pipelines to 'artifact' code/config within GoCD in an initial stage and then fetching that artifact in later stages/pipelines - but ultimately moving away from CodeCommit as the rate limits were too severe to support our parallel builds and AWS didn't seem to be improving things. This was a couple if years ago now, so not sure if things have relaxed or are different region-by-region.

Proxying/caching on top of CodeCommit is also possibly an option with something like nginx, but that is extra complexity and probably only scales if you don't have too many different repos.

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/25ed45e7-2fbb-484a-a004-7d0c865dfb47n%40googlegroups.com.