Automatic pipelines waiting on manual kickoff!!

[orlando kelly]

For some reasons, some of my pipelines that are configured for automatic scheduling seems to be waiting for a manual start. I have many pipelines with 3/4 stages that should run one after the other. In some instances the stage seems to stop and I have to manually go into the tool and kick it off. Has anyone experienced this before. Is there some condition that is occurring that would mean a stage changes from being automatic to manual? All my stage types are set to run 'On Success'

Regards

Orlando

[orlando kelly]

Here are some screenshots of the behaviour

[orlando kelly]

Here is the latest stage to be stuck, but it should run as the last stage was successful

[Ketan Padegaonkar]

So we can help you further, what is the version of Go you are on? You will find the version in the footer of every page.

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[orlando kelly]

I'm on the 15.1 release, the penultimate release to the latest one that came out recently.

[Jyoti Singh]

I have seen this happen if I pause the pipeline while one of the stages is still running. 

On stage completion, Go would check if it can schedule the next stage. Since the pipeline has been paused, it decides not to schedule it. Now upon un-pausing the pipeline, the next set of stages would *not* be auto-triggered. This behaviour exists by design.

--

Cheers,
Jyoti

[orlando kelly]

Hi Jyoti,

Thanks for the info, but we have not paused the any of the pipelines, it seems that random stages stop triggering automatically, and we can't see why this is. So we now have to constantly monitor our pipelines and keep them going manually!! I'll try restarting the server and see if this makes a difference. Is there any other conditions that may cause the stage to change from automatic to manual?

Regards

Orlando 

[Jeff Dunbar]

Hi All,

I am having the same issue as Mr. Kelly. I have a rather large Project that is configured for Auto-scheduling that randomly started to require admin approval  to kick of the unit tests. This has then held up my development deployments.

I too have not paused the build in the past. I am using version 15.1 and seem to have a similar set up as Mr. Kelly by looking at his screenshots.

If any of you could provide further pointers and help it would be much appreciated.

Thanks!

Jeff

[orlando kelly]

Hi Jyoti,

I've sent you a copy of the log and config files to take a look.

Regards

Orlando

[Carl Reid]

We have had similar issues.

We have pipelines with two stages where occasionally only the first stage fires.

When you look at the pipeline the second stage is still grey and it says "Awaiting approval" even though the stage is configured to run "On Success".

Because it is an automatic stage there is nowhere in the user interface to manually trigger it.

I am at a loss to explain why this is happening and what can be done about it. The only thing I can do is cancel that pipeline and run it again from start which may or may not work as expected.

[Aravind SV]

Needs investigating. Maybe it has something to do with agents which lose contact or somehow related to material polling, or something like that.

On Wed, Jul 22, 2015 at 7:34 AM, Carl Reid <carland...@gmail.com> wrote:

We have pipelines with two stages where occasionally only the first stage fires.

When you look at the pipeline the second stage is still grey and it says "Awaiting approval" even though the stage is configured to run "On Success".

Because it is an automatic stage there is nowhere in the user interface to manually trigger it.

Aren't you able to click on the three arrows in between the stages?   

[orlando kelly]

I'm able to do that, but we don't get a notification the stage has stopped, so we have to constantly monitor the board. It really is breaking the whole 'continuous' aspect of the toolset.

[Aravind SV]

On Wed, Jul 22, 2015 at 1:06 PM, orlando kelly <orland...@gmail.com> wrote:

I'm able to do that, but we don't get a notification the stage has stopped, so we have to constantly monitor the board. It really is breaking the whole 'continuous' aspect of the toolset.

Oh yes, of course. I was referring to Carl's point, where he said: "Because it is an automatic stage there is nowhere in the user interface to manually trigger it."

[Orlando Kelly]

Yep we do use that option, but it's is tucked away. I wonder if this could be moved to the main dashboard screen as well.

--
You received this message because you are subscribed to a topic in the Google Groups "go-cd" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd/bc8duGw-89E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Orlando Kelly

Oracle Development Manager

Global IT - Application Services

AMEC

717 Terrace Gardens, Calgary, Alberta, Canada

Tel +1403 513  6524 

Cell +1403  605 3337

orland...@amec.com
amec.com

[marius....@betfair.com]

Can you take a screenshot of the Agents tab, and of the Job tab to check resources and available agents?

[orlando kelly]

Hi Marius,

I'm attaching screenshots of those tabs. They seem to have enough resources.

Regards

Orlando

[marius....@betfair.com]

Thanks, 

I don't think the problem is in the config. We have a lot of pipelines with automatic trigger and no issues like this. The only thing I see different for you are number of agents and how they are allocated in environments. I'll try to get some tests to simulate something similar to your setup, maybe I can reproduce it. 

If you have space, can you try adding more agents to have available in each environment?

Marius

[Aravind SV]

Just for the record: I came across this happening on build.go.cd a few minutes ago, and decided to take a quick look. Here's what I found:

1. It was in the "Awaiting manual approval" mode for a few minutes, and then went away and the next state started.

2. While it was in that mode, the output of /go/api/support showed a stack-trace like the one at the end of this mail. When the next stage started, the thread had finished.

So, I think, the issue is that the stage is not yet complete. It is green, though, and the agent is reporting that it is completed. The server is taking some time to parse the artifacts (particularly test artifacts, I think) and has acquired a lock on the pipeline, which is preventing the next stage from starting. Crucially, given a little time, the next stage should start. I don't know if this is true for everyone. If it is, you should see whether this happens on stages with big test artifacts, and look for this stack trace in /go/api/support when it is happening:

4337308, qtp1027532228-4337308, RUNNABLE
Locked Monitors:
java.lang.String@38451a94 at com.thoughtworks.go.server.service.ScheduleService.updateJobStatus(ScheduleService.java:508)java.lang.String@40d40768 at com.thoughtworks.go.server.service.ScheduleService.updateJobStatus(ScheduleService.java:508)Locked Synchronizers:
Stacktrace:
  org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
  org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
  org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
  org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
  org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
  org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
  org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
  org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
  org.dom4j.io.SAXReader.read(SAXReader.java:465)
  org.dom4j.io.SAXReader.read(SAXReader.java:343)
  com.thoughtworks.studios.shine.cruise.stage.details.XMLArtifactImporter.importFile(XMLArtifactImporter.java:61)
  com.thoughtworks.studios.shine.cruise.stage.details.JobResourceImporter.importArtifactsForJob(JobResourceImporter.java:97)
  com.thoughtworks.studios.shine.cruise.stage.details.JobResourceImporter.importJob(JobResourceImporter.java:71)
  com.thoughtworks.studios.shine.cruise.stage.details.StageResourceImporter.importJobs(StageResourceImporter.java:124)
  com.thoughtworks.studios.shine.cruise.stage.details.StageResourceImporter.load(StageResourceImporter.java:84)
  com.thoughtworks.studios.shine.cruise.BackgroundStageLoader.handle(BackgroundStageLoader.java:71)
  com.thoughtworks.studios.shine.cruise.stage.feeds.StageAtomFeedsReader.readFromLatest(StageAtomFeedsReader.java:51)
  com.thoughtworks.studios.shine.cruise.BackgroundStageLoader.load(BackgroundStageLoader.java:61)
  com.thoughtworks.studios.shine.cruise.BackgroundStageLoaderStageStatusListener.stageStatusChanged(BackgroundStageLoader.java:104)
  com.thoughtworks.go.server.service.StageService$4.afterCommit(StageService.java:233)
  org.springframework.transaction.support.TransactionSynchronizationUtils.invokeAfterCommit(TransactionSynchronizationUtils.java:133)
  org.springframework.transaction.support.TransactionSynchronizationUtils.triggerAfterCommit(TransactionSynchronizationUtils.java:121)
  org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerAfterCommit(AbstractPlatformTransactionManager.java:950)
  org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:796)
  org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
  org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:147)
  com.thoughtworks.go.server.transaction.TransactionTemplate.executeWithExceptionHandling(TransactionTemplate.java:49)
  com.thoughtworks.go.server.service.ScheduleService.updateJobStatus(ScheduleService.java:508)
  com.thoughtworks.go.server.service.BuildRepositoryService.updateStatusFromAgent(BuildRepositoryService.java:58)
  com.thoughtworks.go.remote.BuildRepositoryRemoteImpl$3.call(BuildRepositoryRemoteImpl.java:96)
  com.thoughtworks.go.remote.BuildRepositoryRemoteImpl.handleFailuresDuringReporting(BuildRepositoryRemoteImpl.java:106)
  com.thoughtworks.go.remote.BuildRepositoryRemoteImpl.reportCompleted(BuildRepositoryRemoteImpl.java:88)
  com.thoughtworks.go.server.messaging.BuildRepositoryMessageProducer.reportCompleted(BuildRepositoryMessageProducer.java:76)