git clone failures in gocd pipelines after sonoma upgrade

[SATYA R]

Hi Team,

Recently, we upgraded all of our go-agents to macos Sonoma from ventura.
GoCD also runs on Sonoma version.
After the upgrade, we are seeing intermittent git clone failures in the pipelines.

Each pipeline has 3 or more repos to clone, when multiple pipelines around 20 pipelines try to clone the 3 or more repositories from git, we are facing this git clone issue.

sometimes the pipeline are able to clone all the repos but sometimes they fail.

is there anything that we need to particularly look into in gocd and go-agents configuration due to sonoma upgrade. do we need to make any changes in go-agent service  in order to support the sonoma version?

Kindly help us.

Thank you.

[Chad Wilson]

GoCD doesn't do anything special with Git - it just uses whatever git version you have available to the agent/server on the path. If something has changed after the Sonoma upgrade, it is probably affecting all Git usage on the machine.

• What are the specific clone errors you are getting?
• Have you tried running clones manually to see if you can replicate the problem? That way you can enable git debugging flags to debug the issue. (e.g GIT_CURL_VERBOSE=1)
• Which exact git version are you using? A Mac pre-installed one, or something from Brew or MacPorts?
  
• Are the errors always on agents or always on the server, or both?
• What's the filesystem path that GoCD is cloning into when there are failures?
  

-Chad

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/63efcffd-c537-4855-8039-9de9eafb885en%40googlegroups.com.

[SATYA R]

Hi Chad,

Thank you for responding to my issue.

Just wanted to give some basic info on the GoCD version.

We are using GoCD 23.1.0 aarch version, all of our servers are Mac OS M1 servers(for both GoCD and agents)

• What are the specific clone errors you are getting?

we get the below error when the job tried to clone more the repositories, at first we thorught the git server is closing the connection, the gitlab team took the logs from ec2 instance where it says the source ip which is go-agent is requesting the disconnection.
pipelines logs:
STDERR: Cloning into '/Users/****/go-agent-23.1.0/pipelines/Git_Clone/daf-mock-server'...
STDERR: kex_exchange_identification: Connection closed by remote host
STDERR: Connection closed by 10.112.205.90 port 22
STDERR: fatal: Could not read from remote repository.
STDERR:
STDERR: Please make sure you have the correct access rights
STDERR: and the repository exists.
Failed to run git clone command

Logs from Git server: the Ip in the logs is go-agent ip.
Mar  5 23:30:23 ip-10-112-205-125 sshd[3315655]: pam_unix(sshd:session): session closed for user git
Mar  5 23:30:23 ip-10-112-205-125 sshd[3315640]: Received disconnect from 10.254.56.146 port 63239:11: disconnected by user

• Have you tried running clones manually to see if you can replicate the problem? That way you can enable git debugging flags to debug the issue. (e.g GIT_CURL_VERBOSE=1)

I have not done this part, I can try this on the agent.

• Which exact git version are you using? A Mac pre-installed one, or something from Brew or MacPorts?
  

Before this issue, we used pre-installed git version on all go-agents. In order to resolve this issue, I have installed git with brew on the go-agents and started using it. the version is 2.44.0. But the issue still exists. on the GoCD,  we are still using pre-installed git, do I need install git with brew on the GoCD as well.

• Are the errors always on agents or always on the server, or both?

This error is occurring on the agents when the job starts cloning the repos mentioned in the materials.

• What's the filesystem path that GoCD is cloning into when there are failures?

The failures are happening on the agents. the agent path is /Users/****/go-agent-23.1.0/

We installed GoCD in /Applications/.

The artifacts path is /Volumes/Artifacts.

Please let me know if you need more information.

Thank you.

[Chad Wilson]

Is it always "kex_exchange_identification: Connection closed by remote host" as the failure reason?

If so, it usually means the remote host is closing it before exchanging SSH keys (maybe because of connection limits, or authentication problems), or perhaps there is a firewall/network proxy or something else in between your agent and the git repository host that is closing the connection from both the git client and the git server's perspective.

I can't think of anything obvious that would have changed due just to the MacOS upgrade, unless something has changed with firewall settings, or other such software you have installed on the agents. Are you sure nothing changed in the network environment your agents are deployed, or on the Gitlab instance at the same time?

It looks like you are using SSH Git materials, so once you can replicate with git commands directly on the agent you might need to use environment variable GIT_SSH_COMMAND="ssh -vvv" to see all of the raw SSH debug information (GIT_CURL_VERBOSE is for HTTPS connections). If you discover something here you can probably google to debug it further.

But the issue still exists. on the GoCD,  we are still using pre-installed git, do I need install git with brew on the GoCD as well.

If the problem is only on the agents, then no need to change things on the servers, especially if using homebrew Git didn't change anything.

-Chad

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/f725ced6-703d-480d-a801-049fc13f474fn%40googlegroups.com.