Why is Go picking an old version of an upstream pipeline when a newer version is available...

[Tom Duckering]

Hi Folks,

I'm seeing an odd problem whereby a Go pipeline is picking an older copy of a pipeline when there is a new version available.

Can't for the life of me work out why. Does any one have any ideas?

Tom

[Aravind SV]

On Mon, Sep 8, 2014 at 11:21 AM, Tom Duckering <tom.du...@gmail.com> wrote:

I'm seeing an odd problem whereby a Go pipeline is picking an older copy of a pipeline when there is a new version available.

Can't for the life of me work out why. Does any one have any ideas?

It's probably because of fan-in. This is going to be a bit of an involved explaination. Stay with me. If you already understand fan-in well, then, you can skip to the bottom, where I talk about possible options to find out what is going on in your case.

1. Let's take a simple case, with a git material and a pipeline (P1). You make a commit (g1), and pipeline P1 runs off of that commit.

2. You make another commit, g2 and pipeline P1 runs again, with g2. The run label is now 2.

3. Let's add a new pipeline (P2). It is also set to trigger off of the same git material. This situation is called a "fan-out". Within a few seconds, pipeline P2 runs with the same commit, g2. Its run label is 1.

4. You make a new commit, g3, and both pipelines pick it up and they run it. Everyone is still happy, because new commits are being built against.

5. You add another pipeline (called a "downstream pipeline"), P3. It is set to trigger off of pipelines P1 and P2. This situation is called "fan-in". It can happen with multiple pipelines or a combination of multiple pipelines and (typically SCM) materials, like Git.

6. Within a few seconds, Go sees the new pipeline and triggers it, using P1 (run 3) and P2 (run 2).

7. A new commit, g4, is similarly seen by all the pipelines. P1, P2 and P3.

8. However, if we look at the timing of how those pipelines ran, you'd see something like this.

You can see that pipeline P3 was waiting for both P1 and P2 to finish (successfully) and then, it started running. It did NOT run immediately after P1 finished. This is a very simplified case of fan-in.

9. Let's say you now add a direct dependency from the git material to P3. It doesn't change anything according to Go. The sequence of triggers will be exactly the same.

Even though you added a direct dependency to P3, Go will not trigger P3 till every dependecy of it (P1, P2 and git) are in a compatible revision. It looks at its dependencies, and their parent dependencies and so on, however many levels there are, and then decides whether P3 should run of not. This is what prevents incompatible builds from running.

10. Suppose one of those dependencies fails. Then, P3 will NOT trigger. That's the case shown below:

11. Suppose one of those pipeline dependencies is paused. Then, P3 will NOT trigger. That's the case shown below:

Now, if you forcibly trigger P3, it will pick up the older commit g4, since there is a valid revision that ends up with a compatible revision of all its dependencies (ie, "g4", "P1 - run 4" and "P2 - run 3").

Having said all of that, if your dependency structure is not as easy as this, it'll be difficult to find out exactly why. You should use the VSM feature to do that. The image below is probably small. You can right-click and open it in a new tab or go to https://build.go.cd/go/pipelines/value_stream_map/qa-packages/135 and use the username "view" and password "password". But, in the image, you'll see that pipeline "regression" is not triggered and it will not. Even if I had gone to that pipeline and forcibly triggered it, it would have picked up an older run, and hence an older commit.

You'd then compare this with the VSM for the "regression" pipeline and see why it is picking up an older revision.

Maybe with more information about your config, someone here can help.

Cheers,

Aravind

[Tom Duckering]

Hey Aravind - thanks for your careful explanation. I'm going to try to digest that and take a look. It's certainly the kind of situation we're in.

An additional point - which I fear will not help - is that when we forcibly push the pipeline it picks the latest. Then when it's kicked off by an upstream change then it reverts to an older version.

Will find a moment to give a more detailed explanation soon.

Tom

[Marius Ciotlos]

You might also want to take a look at this https://groups.google.com/forum/m/#!topic/go-cd/rBouLywo3a4

[Yahya Poonawala]

Hi Tom, we are also facing the same issue and although Aravind's explanation is pretty useful it does not seem to be the setup we are having.

Were you able to find a solution for this? 

[srinivas upadhya]

Yahya's team was facing the same issue. We went on a call & we could see that this issue was due to the older commit.

 +----> P1 ----+

 |             |

git            +---> P3

 |             |

 +----> P2 ----+

Scenario:

commit "g1" made at May 1 - 10 am. P1/1, P2/1 & P3 has not run for this.

commit "g2" made at May 1 - 11 am. P1/2, P2/2 & P3/2 run with "g2"

Now commit "g3" made at May 1 - 10.30 am. P1/3 goes green. So P1/3 & & P2/2 are available for P3 to run. But they are not compatible (g3 & g2). Now when Go tries to resolve upstream for P3 it looks for instance of P1 which has an older commit than "g3" & its "g1" & not "g2" (based on time) & hence upstream revisions resolve to "g1" - P1/1 & P2/1 (compatible with g1) & since P3 has not run for P1/1 & P2/1 it runs for older revisions than before.

The older commit than "g3" is the culprit. While it works for centralised SCMs like svn, tfs & perforce, it causes this issue with distributed SCMs git & hg. We could stop P3 from scheduling with P1/1 & P2/1 since P3 has already run with P1/2 & P2/2 (newer revisions) but thats kind of a hack. Ideally we should use "push" times & not "commit" times for distributed SCMs which isn't available.

[Dominik Psenner]

For completeness, we've faced the same issue while working with svn. So this is not exclusively limited to distributed SCMs.

[Carl Reid]

This is excellent documentation but  could do with being moved into the GO docs for reference rather than being in here where it is hard to find.