Pipeline locking and Awaiting Approval

[Darren Rowley]

 

I’d like a clearer understanding of how pipeline locking works. Say I have 2 pipelines:

 

pipeline-1:  stage1:stage2:stage3

and

pipeline-2: stage1:stage2

pipeline-2 is “locked” and also “value streamed” to pipeline-1:stage1

According to the documentation:

“If a pipeline is locked then Go will not allow any other instance of that pipeline to be scheduled until the currently running one has been completed.”

So my understanding is a new instance of pipeline2 will not be created until the previous instance has finished but this should NOT stop new instances of pipeline-1 from running in parallel.

 

The weird thing is since we have been stringing more pipelines together to form Value Streams we’ve started getting sporadic “Awaiting Approval” appearing on the “Pipeline Activity” page (i.e. for the example above after completing pipeline-1: stage1). This seems to come and go (i.e. it will go away if you approve everything and restart the go server).

So what is "Awaiting Approval" ?

Thanks,

Darren.

[Jyoti Singh]

Hi Darren,

Awaiting approval normally comes up for stages which are marked for manual trigger, check the value of "Stage type" on stage settings tab in the admin section. It should be set to "On Success" for Go to automatically schedule a pipeline. Having said that, it can be seen in many scenarios, such as a new stage being added when you already have a few pipeline runs, or if the previous stage fails then the next stage will not be automatically triggered, a user could choose to manually approve such a stage though. 

>> So my understanding is a new instance of pipeline2 will not be created until the previous instance has finished but this should NOT stop new instances of pipeline-1 from running in parallel.

That is correct, pipeline locking for an upstream pipeline will not lock a downstream pipeline. However, there is one detail that you should know. Pipeline1 can have multiple instances running at the same time, but the same stage of two different pipeline instances will never run in parallel. Lets say, pipeline1-stage2 is running with pipeline counter 10. At the same time a new instance of pipeline1 could be triggered say counter 11. Now stage1 of counter-11 finishes, but stage2 of counter-10 is still running. Go will wait for stage2 of counter-10 to finish before it schedules stage2 of counter-11. 

Also, the config you speak about is fairly simple ie. Pipeline2 triggers pipeline1. So as Go finds new revisions for pipeline1, builds would be triggered. However, things would not be the same if the configuration is not exactly how you described above. For instance, if there was a common SCM material feeding into both Pipeline1 and Pipeline2, then Pipeline1 would indeed wait for Pipeline2 to complete. The same would be true if the common material happens to be an upstream pipeline instead/along with a SCM material.

>> The weird thing is since we have been stringing more pipelines together to form Value Streams we’ve started getting sporadic “Awaiting Approval” appearing on the “Pipeline Activity” page (i.e. for the example above after completing pipeline-1: stage1). 

I am sorry, I do not understand which pipeline and stage do you see this message for. I have attached two screenshots, which of the two do you see? Some more details would help.

>> This seems to come and go (i.e. it will go away if you approve everything and restart the go server). 

While we are yet to know where and why you see the awaiting approval message, you shouldn't ever have to restart your Go server for the message to go away. Just manually approving the action should be good enough.

Cheers,

Jyoti

[Aravind SV]

Hi Darren,

Can you also check whether the "Awaiting approval" goes away by itself after a while? I saw this happening on another Go Server, and tracked it down to uploading of test artifacts. Maybe that's what's happening? If so, it should go away by itself.

When you see this happening the next time, take a look at the output of /go/api/support and see if you can find a thread trying to upload artifacts.

Cheers,

Aravind

[Darren Rowley]

Thanks for the response...

On Tuesday, October 6, 2015 at 10:32:31 PM UTC+13, Jyoti Singh wrote:

Hi Darren,

>Awaiting approval normally comes up for stages which are marked for manual trigger, check the value of "Stage type" on stage settings tab in the admin section. It should be set to "On Success" for Go to automatically schedule a pipeline

Definitely set to "On success" for every stage on both of the pipelines.
 
 

. Having said that, it can be seen in many scenarios, such as a new stage being added when you already have a few pipeline runs,

This isn't the case we have changed stages in these 2 pipelines for a couple of week.
 

or if the previous stage fails then the next stage will not be automatically triggered, a user could choose to manually approve such a stage though. 

I guess that makes sense but but I don't see why it would stop new pipelines starting or why the upstream pipeline being paused show make the downstream pipeline pause.
 

>> So my understanding is a new instance of pipeline2 will not be created until the previous instance has finished but this should NOT stop new instances of pipeline-1 from running in parallel.

That is correct, pipeline locking for an upstream pipeline will not lock a downstream pipeline. However, there is one detail that you should know. Pipeline1 can have multiple instances running at the same time, but the same stage of two different pipeline instances will never run in parallel. Lets say, pipeline1-stage2 is running with pipeline counter 10. At the same time a new instance of pipeline1 could be triggered say counter 11. Now stage1 of counter-11 finishes, but stage2 of counter-10 is still running. Go will wait for stage2 of counter-10 to finish before it schedules stage2 of counter-11. 

Ok that's good to know but I don't think it is related to our problem. 
 

Also, the config you speak about is fairly simple ie. Pipeline2 triggers pipeline1. So as Go finds new revisions for pipeline1, builds would be triggered. However, things would not be the same if the configuration is not exactly how you described above. For instance, if there was a common SCM material feeding into both Pipeline1 and Pipeline2, then Pipeline1 would indeed wait for Pipeline2 to complete. The same would be true if the common material happens to be an upstream pipeline instead/along with a SCM material.

Ok will the upstream pipeline does also have SCM materials and these are from the same SVN repo but they are NOT the exact same materials set. Also this doesn't happen every time. The whole build process works fine for a few hours but it appears once something starts failing (we think the upstream smoke test) then we start to see this locking.
 

>> The weird thing is since we have been stringing more pipelines together to form Value Streams we’ve started getting sporadic “Awaiting Approval” appearing on the “Pipeline Activity” page (i.e. for the example above after completing pipeline-1: stage1). 

I am sorry, I do not understand which pipeline and stage do you see this message for. I have attached two screenshots, which of the two do you see? Some more details would help.

Ok looks like you didn't attach the screenshots. However we actually get both pipelines stages start going into "awaiting approval" but it is the first upstream pipeline where the problem lies as this is our main CI build process.

Some screenshots of my own attached.

The first one go1.png. This one is a bunch of build stage pipelines all "awaiting" approval even though everything is green. This pipeline is our main CI process and has no "locking" configured. In this screen shot as soon as the #143 "build" stage completes it will sit and NOT go into the "tell" stage. 

The 2nd shows the overview of our upstream "smoke test" pipline - this one does have pipeline "locking" enabled. This one sometimes get the mysterious "Padlock without an X" in it however when pipelines start going into "Awaiting approval" mode we don't always get this padlock.

The 3rd screenshot again shows the history of our "deploy-smoke" pipeline (the one with locking enabled). This strange thing about this one was this morning, I have had the pipeline "paused" for 2 days since build #6, the immediately when I un-pause it build #7 starts and sits at the 'awaiting approval" point again....
 

>> This seems to come and go (i.e. it will go away if you approve everything and restart the go server). 

While we are yet to know where and why you see the awaiting approval message, you shouldn't ever have to restart your Go server for the message to go away. Just manually approving the action should be good enough.

Ok yes we are clutching at straws the problem is once it gets itself into this state the you have to keep manaully approving all the time.

Thanks Darren.

 

[Darren Rowley]

Thanks Arvind, no this never seems to go away at least not for "hours"... Just ran the API you suggested. Lots and lots of Locked Monitors but I guess that's expected but nothing that appears to do with artifact uploads. We do upload large artifacts but that happens as part of the first stage in the first pipeline and that seems to work fine. We also download those artifacts in the first stage of the 2nd pipeline but again that bit is working fine, if it was blocking here the deployments would be working..

Thanks
Darren.

[Jyoti Singh]

Hi Darren,

>> Ok will the upstream pipeline does also have SCM materials and these are from the same SVN repo but they are NOT the exact same materials set. Also this doesn't happen every time. The whole build process works fine for a few hours but it appears once something starts failing (we think the upstream smoke test) then we start to see this locking.

Failure of an upstream pipeline can cause some of the descendents to stop getting autoscheduled in case of fan-in. But it will not cause the second stage to be waiting for approval while first stage ran automatically.

>> Ok looks like you didn't attach the screenshots. 

Sorry about that, but I think you got an idea of what I was referring to.

>> The first one go1.png. This one is a bunch of build stage pipelines all "awaiting" approval even though everything is green. This pipeline is our main CI process and has no "locking" configured. In this screen shot as soon as the #143 "build" stage completes it will sit and NOT go into the "tell" stage.

I haven't seen the schedule and completed times for your stages, so I cannot tell for sure. But looking at the screenshot, I think this is what happened. Lets start with counter 138. "build" finishes off and triggers "tell". Meanwhile "build" of 139 triggers and finishes. By this time if "tell" of 138 hasn't finished, "tell" of 139 will not be triggered.

Now, "build" of 140 triggers and finishes and even now "tell" of 138 hasn't finished. So, at this point "tell" of both 139 and 140 are waiting to be scheduled. Once "tell" of 138 completes, it will look for the most recent run of this pipeline and trigger the "tell" stage of that run. So, in your case, "tell" of 140 gets executed. "tell" of 139 will not be automatically executed, but could be triggered manually if desired. This behavior exists by design to avoid wasted builds.

>> The 2nd shows the overview of our upstream "smoke test" pipline - this one does have pipeline "locking" enabled. This one sometimes get the mysterious "Padlock without an X" in it however when pipelines start going into "Awaiting approval" mode we don't always get this padlock.

"Padlock with an X" means the pipeline is locked and you will not be able to unlock it. "Padlock without an X" on the other hand is an "unlock" button. That would show up when none of the stages are currently running but pipeline instance hasn't completed ie, not all stages have run or gone green. In this case since the second stage is waiting for manual approval, the pipeline instance could not complete. On manually unlocking the pipeline, Go will start scheduling newer runs for this pipeline. As to why the pipeline did not run all its stages, see if any of the conditions mentioned in a similar thread applies. There are multiple responses raising different points, so you need to go through the entire thread.

>> The 3rd screenshot again shows the history of our "deploy-smoke" pipeline (the one with locking enabled). This strange thing about this one was this morning, I have had the pipeline "paused" for 2 days since build #6, the immediately when I un-pause it build #7 starts and sits at the 'awaiting approval" point again....

Did you go back to pausing the pipeline right after you triggered it? If so, that is an expected behavior. 

Cheers,

Jyoti

[Darren Rowley]

>> I haven't seen the schedule and completed times for your stages, so I cannot tell for sure. But looking at the screenshot, I think this is what happened. Lets start with counter 138. "build" finishes off and triggers "tell". Meanwhile "build" of 139 triggers and finishes. By this time if "tell" of 138 hasn't finished, "tell" of 139 will not be triggered.

Now, "build" of 140 triggers and finishes and even now "tell" of 138 hasn't finished. So, at this point "tell" of both 139 and 140 are waiting to be scheduled. Once "tell" of 138 completes, it will look for the most recent run of this pipeline and trigger the "tell" stage of that run. So, in your case, "tell" of 140 gets executed. "tell" of 139 will not be automatically executed, but could be triggered manually if desired. This behavior exists by design to avoid wasted builds.

<<
Hmm, don' think this is the issue because build takes approx 20mins. tell takes approx 20 seconds and checks takes approx 15mins. However this "tell" stage is the upstream materials trigger for the deploy / smoke test and again I wonder if the upstream pipeline locking has something to do with it.
 

Did you go back to pausing the pipeline right after you triggered it? If so, that is an expected behavior. 

No absolutely not, I enabled this build and it ran and then immediately put itself into "Awaiting Approval" after the first stage completed even though the previous run had passsed (before I paused it).

Thanks.

D.