Modify Fan in behaviour

[Manikkumar K]

We do have a fan-in where number upstream has exploded in the count, 

Instead of waiting for all upstream to pass, can we pick most recent successful upstream pipeline along with new pipeline instance of one(or more) of upstream?

Thanks,

Mani

[Chad Wilson]

Doing that means removing the guarantees fan-in offers you as to consistency of the material versions on upstream pipelines when triggering a downstream.

It's possible to turn off fan-in globally (although I have never done so personally) but to my knowledge not for individual pipelines.

Alternatively, if your upstreams are some sort of monorepo which are segregated by upstream pipelines (e.g each pipeline operates on different subdirectories or parts of the repo) you could consider using something like https://github.com/TWChennai/gocd-git-path-material-plugin to define separate materials for each of the upstreams that share a repo with different subsets of the repo that they operate on so GoCD doesn't decide they need fan-in.

-Chad

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/4bcad889-0820-427b-ac85-d91e50f24e43n%40googlegroups.com.

[Manikkumar K]

Ours is not monorepo. Is there any other alternative? 

You received this message because you are subscribed to a topic in the Google Groups "go-cd" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd/VVTYzU5OhV4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH-Pub%3DUtDr0QEc-yS5DoR3BbKbagC0Jk8rfSYFDbiXe5Q%40mail.gmail.com.

[Chad Wilson]

If it's "not monorepo", can you clarify if you have multiple upstreams that point to one repository/material? That is the scenario that triggers fan-in, and that does at least indicate that the repo is somewhat being treated as a monorepo in the sense that you have multiple pipelines directly dependent on that repository.

Otherwise you might need to share more specific examples of pipelines, dependencies and the materials they have, along with the behaviour you observe vs. desire.

To successfully trigger a downstream, all upstreams only need to pass (on a given material version) if they share a material. If they do not share 1 or more materials then the behaviour should be as you describe - the downstream will trigger with the last successful version of each upstream stage (no fan-in required).

-Chad

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CACfO_cKcqovDwHrmD0G_yBP1cCYJJT%2BMs5hxV9KeTKrQmuf-OQ%40mail.gmail.com.

[Manikkumar K]

You are spot on. We do share one of the repositories for the all microservice(aka upstreams). 

And was able to simulate and see where the fan-in didn't happen when upstreams do not share any of the materials.

Thanks for prompt response,

Mani

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH8n%2BOm%3D5L13B3srVAhNZAPnHdrPZKJMh2sC7ZQrCB%2BJBw%40mail.gmail.com.

[Chad Wilson]

So sometimes with one repo and multiple pipelines you want fan-in behaviour, e.g if you are splitting the pipelines to run different logical testing blocks in parallel and either

• don't want to model them as parallel jobs within one pipeline or
• where pipelines require multiple source code material inputs, and you don't want excessive triggering of unrelated pipeline work

Basically the classic cases documented at https://www.gocd.org/getting-started/part-3/#fan_out_fan_in or https://www.gocd.org/2017/04/17/build-propagation-using-fan-in-fan-out.html

Other times you have multiple things in one repo for ease of management, but things are sufficiently segregated and you are OK with treating the pieces independently. In this case fan-in might not be what you want/need, despite sharing a repository/material.

Let's say you have some more classic mono repo A:

./deploy-scripts/
./service-common/

./service-A/

./frontend/

Perhaps you then have pipelines like

repoA --fanout--> frontend-build --> frontend-functional-test --fanin--> frontend-plus-service-A-smoke-test --fanout--> (deploy-frontend-to-UAT + deploy-service-A-to-UAT)

repoA --fanout--> service-A-build --> service-A-functional-test --fanin--> frontend-plus-service-A-smoke-test --fanout--> (deploy-frontend-to-UAT + deploy-service-A-to-UAT)

In a normal setup with GoCD, frontend-plus-service-A-smoke-test will only trigger when all of the upstreams are built on the same revision from repoA. If the frontend tests are perhaps far slower, this may unnecessarily slow progress down for promoting service-A code changes to UAT, and require triggering of everything.

You may be confident enough that they can be built and deployed independently, but still want some confidence that the frontend and service-A play nicely together which is why you have "frontend-plus-service-A-smoke-test". In this case you may want to take more responsibility on yourselves, and consider something like https://github.com/TWChennai/gocd-git-path-material-plugin to allow the deploys to happen independently, but without excessive triggering of unrelated pieces, without splitting your repository. With something like this you might instead model like

repoA (path: deploy-scripts/ + frontend/) --> frontend-build --> frontend-functional-test --> frontend-plus-service-A-smoke-test --fanout--> (deploy-frontend-to-UAT + deploy-service-A-to-UAT)

repoA (path: deploy-scripts/ + service-common/ + service-A/)  --> service-A-build --> service-A-functional-test --> frontend-plus-service-A-smoke-test --fanout--> (deploy-frontend-to-UAT + deploy-service-A-to-UAT)

This would logically split the materials while allowing you to commit together in one repo, essentially working around the limitations of filtering noted here.

Anyway, these are just different tools in the arsenal, depending on what you are trying to achieve, and how you are modelling your pipelines and splitting your source code, test/deploy scripting etc.

-Chad

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CACfO_c%2BrsZviV4sMiy6%2BnTquWqWi7L6AafDc3%2B5Z%2BbhCrihzjg%40mail.gmail.com.