Fan-In from different SCM material with deployment gate

[Joe H]

My application has several libraries each with their own git repository. The repos provide material to individual pipelines that build the libraries. 

The build pipelines then feed to a revision packager pipeline that bundles the libraries into a revision package which then feeds into a deployment pipeline.

    REPO-A -> BUILD-A

    REPO-B -> BUILD-B

    REPO-C -> BUILD-C

    BUILD-A, BUILD-B, BUILD-C -> PACKAGE -> DEPLOY

Let's say BUILD-A and BUILD-B take 20 seconds to complete, but BUILD-C pipeline takes a few minutes to complete.

What I'm seeing is that when commits come in on multiple repos somewhat simultaneously, this results in multiple runs of the PACKAGE -> DEPLOY pipelines.  I'm trying to set up a gate that waits for all running builds to complete before continuing with PACKAGE.

I understand why that's the default behavior if you view each path through the graph as a unique run. The documentation also indicates that Fan-In only works if the SCM material for the entry pipelines is the exact same. So it seems I'm observing the documented behavior.

Is there a way to achieve what I'm trying to do where I can Fan-In from different SCM material?  I'm thinking like a way to override the "Fan-In Key" where it could use an arbitrary key to drive the Fan-In behavior instead of the repo URL.

If not, is there an obvious way I'm not seeing to break up my graph such that I can wait for all BUILD-* to complete with some kind of quiet period gate before continuing with PACKAGE -> DEPLOY? For example, once any of the BUILD-* pipelines have run, wait 45 seconds and if nothing else starts, then run PACKAGE -> DEPLOY.

Maybe the PACKAGE pipeline has to be manual trigger and ignores the BUILD-* pipelines for scheduling, then the "gate" is really just a script or something that uses the API to determine whether or not it's time to run?

Looking for any ideas I'm not seeing...

Thanks

 

[Jason Smyth]

Hi Joe,

I think you are on the right track with your manual trigger idea. It should be pretty straight-forward to put together a simple script/program that checks every minute or 2 to see:

1. Is there a new version of BUILD-A, -B or -C?

2. Is there currently an instance of BUILD-A, -B or -C running?

Then, if the answer to the first question is "yes" and the second question is "no", kick off the PACKAGE Pipeline.

Another option might be to design the PACKAGE process in such a way that it checks to see whether there is an instance of any of the BUILD Pipelines running and, if so, it fails the PACKAGE Pipeline. But depending on how often that situation comes up it might cause frustration in the user base if there are a lot of "failures" that aren't really failures.

Hope this helps somewhat.

Regards,

Jason