Conditional Pipeline Trigger

[Anant Agarwal]

Hi 

We have a bunch of micro services in separate code repositories.   The pipelines we have setup look like the diagram below.

At the moment whenever we make changes to any one service let's say service1,   integration test pipeline runs and then triggers deployment of all services. 

What we want is that Integration test pipeline should run if changes to any of the dependent services but it should trigger deployment only for the particular service which originated the change.  Is that possible?

so we are kind of looking to achieve "AND" condition in pipeline trigger.  i.e.  Deployment service 1 pipeline should trigger only when Service1 Artefact was built AND integration test passed.   and this should not trigger Deployment service2 pipeline. 

Appreciate your help. 

Thanks

Anant Agarwal

[Aravind SV]

No, there's no way, as far as I know. I remember that this question (or a very similar one) has come up before. I think what's making this hard to do is the fact that the information shown in the build cause popup of the "IntegrationTest" pipeline is lost. I mean, if you click on "Changes" in that pipeline, it'll show you, in yellow, which upstream pipeline(s) caused it ("IntegrationTest" pipeline) to trigger. But, once that pipeline has finished, that information is no longer accessible to the downstream pipelines (DeploymentService1, ...).

I'm wondering: If GoCD provided an environment variable (say, "GO_material_name_CHANGED" like "GO_material_name_LABEL" etc), then a deployment descriptor (a file describing what happened) can be created and pushed down as an artifact to all three downstream pipelines, which can figure out whether to re-deploy or not.

I know it's not a pipeline modeling solution but it is a solution which can enable this kind of a workflow to be supported. Thoughts?

I'm thinking of a code change like this (just to clarify):

diff --git a/common/src/com/thoughtworks/go/config/materials/PackageMaterial.java b/common/src/com/thoughtworks/go/config/materials/PackageMaterial.java
index 2513d68..3847bf2 100644
--- a/common/src/com/thoughtworks/go/config/materials/PackageMaterial.java
+++ b/common/src/com/thoughtworks/go/config/materials/PackageMaterial.java
@@ -168,6 +168,7 @@ public class PackageMaterial extends AbstractMaterial {
     @Override
     public void populateEnvironmentContext(EnvironmentVariableContext context, MaterialRevision materialRevision, File workingDir) {
         context.setProperty(upperCase(format("GO_PACKAGE_%s_LABEL", escapeEnvironmentVariable(getName().toString()))), materialRevision.getRevision().getRevision(), false);
+        context.setProperty(upperCase(format("GO_PACKAGE_%s_CHANGED", escapeEnvironmentVariable(getName().toString()))), Boolean.toString(materialRevision.isChanged()), false);
         for (ConfigurationProperty configurationProperty : getPackageDefinition().getRepository().getConfiguration()) {
             context.setProperty(getEnvironmentVariableKey("GO_REPO_%s_%s", configurationProperty.getConfigurationKey().getName()),
                     configurationProperty.getValue(), configurationProperty.isSecure());
diff --git a/common/src/com/thoughtworks/go/config/materials/PluggableSCMMaterial.java b/common/src/com/thoughtworks/go/config/materials/PluggableSCMMaterial.java
index 391bfae..1190cf9 100644
--- a/common/src/com/thoughtworks/go/config/materials/PluggableSCMMaterial.java
+++ b/common/src/com/thoughtworks/go/config/materials/PluggableSCMMaterial.java
@@ -265,6 +265,7 @@ public class PluggableSCMMaterial extends AbstractMaterial {
     @Override
     public void populateEnvironmentContext(EnvironmentVariableContext context, MaterialRevision materialRevision, File workingDir) {
         context.setProperty(getEnvironmentVariableKey("GO_SCM_%s_%s", "LABEL"), materialRevision.getRevision().getRevision(), false);
+        context.setProperty(getEnvironmentVariableKey("GO_SCM_%s_%s", "CHANGED"), Boolean.toString(materialRevision.isChanged()), false);
         for (ConfigurationProperty configurationProperty : scmConfig.getConfiguration()) {
             context.setProperty(getEnvironmentVariableKey("GO_SCM_%s_%s", configurationProperty.getConfigurationKey().getName()),
                     configurationProperty.getValue(), configurationProperty.isSecure());
diff --git a/common/src/com/thoughtworks/go/config/materials/ScmMaterial.java b/common/src/com/thoughtworks/go/config/materials/ScmMaterial.java
index d3c86db..aa9509e 100644
--- a/common/src/com/thoughtworks/go/config/materials/ScmMaterial.java
+++ b/common/src/com/thoughtworks/go/config/materials/ScmMaterial.java
@@ -129,13 +129,14 @@ public abstract class ScmMaterial extends AbstractMaterial {
         String toRevision = materialRevision.getRevision().getRevision();
         String fromRevision = materialRevision.getOldestRevision().getRevision();

-        setGoRevisionVariables(environmentVariableContext, fromRevision, toRevision);
+        setGoRevisionVariables(environmentVariableContext, materialRevision, fromRevision, toRevision);
     }

-    private void setGoRevisionVariables(EnvironmentVariableContext environmentVariableContext, String fromRevision, String toRevision) {
+    private void setGoRevisionVariables(EnvironmentVariableContext environmentVariableContext, MaterialRevision materialRevision, String fromRevision, String toRevision) {
         setVariableWithName(environmentVariableContext, toRevision, GO_REVISION);
         setVariableWithName(environmentVariableContext, toRevision, GO_TO_REVISION);
         setVariableWithName(environmentVariableContext, fromRevision, GO_FROM_REVISION);
+        setVariableWithName(environmentVariableContext, Boolean.toString(materialRevision.isChanged()), "GO_CHANGED");
     }

     protected void setVariableWithName(EnvironmentVariableContext environmentVariableContext, String value, String propertyName) {
diff --git a/common/src/com/thoughtworks/go/config/materials/dependency/DependencyMaterial.java b/common/src/com/thoughtworks/go/config/materials/dependency/DependencyMaterial.java
index 016e049..ea3681a 100644
--- a/common/src/com/thoughtworks/go/config/materials/dependency/DependencyMaterial.java
+++ b/common/src/com/thoughtworks/go/config/materials/dependency/DependencyMaterial.java
@@ -160,6 +160,7 @@ public class DependencyMaterial extends AbstractMaterial {
         DependencyMaterialRevision revision = (DependencyMaterialRevision) materialRevision.getRevision();
         context.setPropertyWithEscape(format("GO_DEPENDENCY_LABEL_%s", getName()), revision.getPipelineLabel());
         context.setPropertyWithEscape(format("GO_DEPENDENCY_LOCATOR_%s", getName()), revision.getRevision());
+        context.setPropertyWithEscape(format("GO_DEPENDENCY_CHANGED_%s", getName()), Boolean.toString(materialRevision.isChanged()));
     }

     public boolean isAutoUpdate() {

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Anant Agarwal]

Hi Aravind

Thank you very for your response.  I like the idea you have suggested but unable to comment on the code snippet you have attached.  

Could you clarify following in bold  "as an artefact to all three downstream pipelines, which can figure out whether to re-deploy or not."      does the pipeline has ability to figure out to run or not based on artefact or are you suggesting that deployment code can decide whether to deploy or not in which case all deployment pipeline will run but deployment code will deploy conditionally.

I was thinking that may be I have to rethink how I setup the pipeline,  I am finding it hard to believe that such a common scenario in today's word about building microservices,  running integration test and deployment of what has changed should be easy.   

Thanks
Anant Agarwal

To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.

[Anant Agarwal]

May be https://api.go.cd/current/#scheduling-pipelines    is the answer for controlling the deployment pipelines trigger.

[Jyoti Singh]

Just thinking out loud, what if the integration test pipeline was refactored to have two stages. First stage to run the integration-tests and second to publish the changed installers to a package repository (https://www.go.cd/plugins/). 

'DeploymentService1' pipeline could now be changed to depend on this package repo instead with appropriate package spec (say Service1) specified. The same goes for the other two deployment pipelines - DeploymentService2 and DeploymentService3.   

With this, the deployment pipelines would get triggered only if a new package matching the specified specs was published. Won't that help simply the scenario?

[Anant Agarwal]

Thanks Jyoti so there are two key points here:

1. publish the changed installers to a package repository 

2. DeploymentService1' pipeline could now be changed to depend on this package repo instead with appropriate package spec (say Service1) specified.

Your idea looks good however I am not sure about point2 "appropriate package spec" I will explore this configuration and will get back.

Thanks a lot.

Sent from my iPhone

You received this message because you are subscribed to a topic in the Google Groups "go-cd" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd/z1t0fy1Ldck/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd+un...@googlegroups.com.

[Sriram Narayanan]

Consider this:

a. Various pipelines publish their artifacts as packages (e.g. RPMs).

b. You run integration tests using the "latest" of the various pipelines.

c. When a given set of artifacts work together (as per the integration tests), you publish this information somewhere.

d. You run a "deploy" task on all the agents for that environment.

e. The deploy task just runs a configuration management tool like Ansible, which uses the version numbers provided, and then "ensures" that those versions are present.

This way, if a given computer will always have the version that you want it to have.

-- Ram

To unsubscribe from this group and all its topics, send an email to go-cd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "go-cd" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscribe@googlegroups.com.