Material update is currently running but has not shown any activity in the last 15 minute(s).

[Aslhey]

Hi, from yesterday all pipelines in gocd got freezed, looking for logs, I found go/api/process_list return 8 process/commands for git clone, and running inside gocd server host ps -ef | grep git return that git clones, looks like git got stuck. Then how I could fix it?. Killing stuck git process does not work. gocd is showing so many errors infos and sometimes warnings:

=================
Go Server Version
=================

15.2.0(2248-c1f90a6aa3192a)

=========================
Server health information
=========================

Server Health Messages: [51 messages]

[Ketan Padegaonkar]

This is because of two possible reasons —

• The git process hung
• The git process is waiting for input on standard input — try running `git ls-remote YOUR_GIT_URL` as the go user on your server. You seem to be using an ssh url for git. Have you setup the SSH keys on your server ~/.ssh/id_rsa.pub?

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Varsha Varadarajan]

Hey,

I had seen this issue happen once on our build server while using the github PR plugin. Doing a kill of all the git processes spawned by the go server seemed to help as for the next material update check, the go server would start new git processes and continue.  

Out of the errors shown in the screenshot, it seems like 3 different git materials were picked up during the material update check and 1 material specified for the Github PR plugin. So, I think we can say that its a generic git problem and not anything to do with the plugin. It does look like git clone is taking a long time for all these materials. I cannot say for sure, but it could just be a problem with the network or github's server or the fact that they might be large repositories. 

Can you please try to do a git clone for these git repositories specified as git materials and  on the GoCD server machine (as a 'go' user if you are using Linux, as the local system administrator if you are using Windows, or as the user who installed the GoCD server on Mac) ?

Thanks,

Varsha

[Aslhey]

Hi,

here is git ls-remote ouput:

go@b4e33f719330:/tmp$ git ls-remote g...@github.com:(user)/(repo).git
ea100646b78d166402ffcbe767d31191044b342a        HEAD
84563d0c9c0f17eaceddf18bb628dc5c57c2aa36        refs/heads/OR-387
ea100646b78d166402ffcbe767d31191044b342a        refs/heads/develop
299960c57297d6bf5fa8c7454442d176bc481afd        refs/heads/master
1759cb00de08a9934de62d364b7a3821ad54911e        refs/pull/1/head
763dd1b2adbd1448a000efd7f9122b4eb164f632        refs/pull/1/merge
9dc3b2797c0c76d97599fef461247dcfae08fa30        refs/pull/10/head
4d16002349933286689181fbd642512b4a02872a        refs/pull/11/head
...[more]
299960c57297d6bf5fa8c7454442d176bc481afd        refs/tags/v1.0

ssh keys are right, I can clone and push.

[Aslhey]

Hi,

I can clone and push as 'go' user, so ssh keys are right.

look at the 'ps aux | grep git' out

root@b4e33f719330:/tmp# ps aux | grep git
go         395  0.0  0.0  18076  1352 ?        S    15:24   0:00 git clone --branch=master g...@github.com:(my user)/(repo1).git /var/lib/go-server/pipelines/flyweight/bab38460-49da-46fd-bf6d-511d6d368c72
go         401  0.0  0.0  46216  2912 ?        S    15:24   0:00 ssh git@github.com git-upload-pack '(my user)/(repo1).git'
go         410  0.0  0.0  18076  1352 ?        S    15:24   0:00 git clone --branch=master git@github.com:(my user)/(repo2).git /var/lib/go-server/pipelines/flyweight/79b579d3-e722-410c-9603-2111f25e8337
go         416  0.0  0.0  18124  1300 ?        S    15:24   0:00 git fetch origin +refs/pull/*/head:refs/remotes/origin/pull-request/*
go         419  0.0  0.0  46216  2916 ?        S    15:24   0:00 ssh g...@github.com git-upload-pack '(my user)/(repo8).git'
go         420  0.0  0.0  46216  2912 ?        S    15:24   0:00 ssh g...@github.com git-upload-pack '(my user)/(repo2).git'
go         421  0.0  0.0  18076  1348 ?        S    15:24   0:00 git clone --branch=master g...@github.com:(my user)/(repo5).git /var/lib/go-server/pipelines/flyweight/417b65f5-1acb-46f1-9f92-a62c3399bbaf
go         425  0.0  0.0  46216  2912 ?        S    15:24   0:00 ssh g...@github.com git-upload-pack '(my user)/(repo5).git'
go         426  0.0  0.0  18076  1344 ?        S    15:24   0:00 git clone --branch=master g...@github.com:(my user)/(repo2).git /var/lib/go-server/pipelines/flyweight/9f11fae9-364a-4c3f-b5ed-afd5a9ecbd32
go         430  0.0  0.0  46216  2912 ?        S    15:24   0:00 ssh g...@github.com git-upload-pack '(my user)/(repo2).git'
go         431  0.0  0.0  18076  1348 ?        S    15:24   0:00 git clone --branch=master g...@github.com:(my user)/(repo3).git /var/lib/go-server/pipelines/flyweight/71e704a0-33b1-441f-b5ea-f448e11c2a2c
go         435  0.0  0.0  46216  2916 ?        S    15:24   0:00 ssh g...@github.com git-upload-pack '(my user)/(repo3).git'
go         441  0.0  0.0  18076  1348 ?        S    15:24   0:00 git clone --branch=master g...@github.com:(my user)/CD-HelloWorld-backend.git /var/lib/go-server/pipelines/flyweight/944418b3-3a02-4a6c-8990-ed13f29ddf37
go         444  0.0  0.0  46216  2912 ?        S    15:24   0:00 ssh g...@github.com git-upload-pack '(my user)/CD-HelloWorld-backend.git'
go         464  0.0  0.0  18080  1296 ?        S    15:24   0:00 git fetch origin
go         467  0.0  0.0  46216  2908 ?        S    15:24   0:00 ssh g...@github.com git-upload-pack '(my user)/(repo4).git'
go         481  0.0  0.0  18120  1296 ?        S    15:24   0:00 git fetch origin
go         484  0.0  0.0  46216  2908 ?        S    15:24   0:00 ssh g...@github.com git-upload-pack '(my user)/(repo3).git'
go         625  0.0  0.0  18172  1312 ?        S    15:25   0:00 git fetch origin +refs/pull/*/head:refs/remotes/origin/pull-request/*
go         628  0.0  0.0  46216  2908 ?        S    15:25   0:00 ssh g...@github.com git-upload-pack '(my user)/(repo5).git'

[Aslhey]

plugin-github.pr.log is showing:

2016-12-28 15:59:15,839  WARN [88@MessageListener for MaterialUpdateListener] GitHubPRBuildPlugin:67 - get latest revisions since:
java.lang.RuntimeException: Exception Occurred: [git, clone, --branch=master, git@github.com:(my user)/(my repo1).git, /var/lib/go-server/pipelines/flyweight/2acf2b24-9ed9-4487-af18-bace6b6bdee1] - null
        at com.tw.go.plugin.cmd.Console.runOrBomb(Console.java:33)
        at com.tw.go.plugin.git.GitCmdHelper.runAndGetOutput(GitCmdHelper.java:397)
        at com.tw.go.plugin.git.GitCmdHelper.cloneRepository(GitCmdHelper.java:55)
        at com.tw.go.plugin.GitHelper.cloneOrFetch(GitHelper.java:38)
        at in.ashwanthkumar.gocd.github.GitHubPRBuildPlugin.handleLatestRevisionSince(GitHubPRBuildPlugin.java:244)
        at in.ashwanthkumar.gocd.github.GitHubPRBuildPlugin.handle(GitHubPRBuildPlugin.java:121)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager$1.execute(DefaultPluginManager.java:172)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager$1.execute(DefaultPluginManager.java:167)
        at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:315)
        at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:245)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:167)
        at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:32)
        at com.thoughtworks.go.plugin.access.scm.SCMExtension.latestModificationSince(SCMExtension.java:156)
        at com.thoughtworks.go.server.service.materials.PluggableSCMMaterialPoller.modificationsSince(PluggableSCMMaterialPoller.java:77)
        at com.thoughtworks.go.server.service.materials.PluggableSCMMaterialPoller.modificationsSince(PluggableSCMMaterialPoller.java:46)
        at com.thoughtworks.go.server.service.MaterialService.modificationsSince(MaterialService.java:110)
        at com.thoughtworks.go.server.materials.ScmMaterialUpdater.insertLatestOrNewModifications(ScmMaterialUpdater.java:52)
        at com.thoughtworks.go.server.materials.PluggableSCMMaterialUpdater.insertLatestOrNewModifications(PluggableSCMMaterialUpdater.java:61)
        at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.insertLatestOrNewModifications(MaterialDatabaseUpdater.java:155)
        at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.updateMaterialWithNewRevisions(MaterialDatabaseUpdater.java:147)
        at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater$2.doInTransaction(MaterialDatabaseUpdater.java:110)
        at com.thoughtworks.go.server.transaction.TransactionCallback.doWithExceptionHandling(TransactionCallback.java:24)
        at com.thoughtworks.go.server.transaction.TransactionTemplate$3.doInTransaction(TransactionTemplate.java:53)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
        at com.thoughtworks.go.server.transaction.TransactionTemplate.executeWithExceptionHandling(TransactionTemplate.java:49)
        at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.updateMaterial(MaterialDatabaseUpdater.java:107)
        at com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:48)
        at com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:29)
        at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:65)
        at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:50)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 128 (Exit value: 128)
        at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
        at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166)
        at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:153)
        at com.tw.go.plugin.cmd.Console.runOrBomb(Console.java:25)
        ... 30 more