GoCD 19.1.0 Git material unwanted clean

[Denis O]

Hi,

I'm facing to a strange behavior with Git material pipeline.

In fact even:

- I check "fetch material" on the first stage only

- I check "newer cleanup artifacts" on ALL stages

- I uncheck "clean working directory" on ALL stages

As a result each pipeline execution the workspace is entirely deleted and replaced by Git revision content files. Is this behavior is normal, something wrong in my configuration ?

Any help wold be apreciated :)

GoCD Version: 19.1.0 (8469-3885582184c6f7c4bbbeb94239e5dba6f5f772f4).

Regards

[Aravind SV]

Hello Denis,

The working directory will always be cleaned using something like git clean -dff (or git clean -dffx maybe) before any job starts. If you really need something across runs, then you should move it out of the workspace directory. However, please note that this is not recommended. One of the concepts of CI is repeatability. It shouldn’t depend on the environment of a previous run of a job or pipeline.

The GoCD way of keeping state across stages within a pipeline (for instance) is to use artifacts.

Cheers,
Aravind

[Denis O]

Hi,

Thanks for your response. But how can I setup an incremental build pipeline? I have an msbuild daily pipeline that I want to build as fast as possible to get a quick feedback to the team. MSbuild is based on file timestamp to detect changes so as a result each run is a full rebuild instead of incremental. Is it possible to achieve this with GoCD?

Many thanks,

Denis

[Denis O]

Any clue?

[Aravind SV]

Hello Denis,

It depends. GoCD just does a git clean as I mentioned, which shouldn’t change modified times of your codebase, I mean, of files which haven’t changed, right?

If you’ve asked it to clean working directory, it will remove and re-clone. Otherwise, it shouldn’t change. Of course, if you have multiple agents, you can’t guarantee that the same agent will pick up your job across days without using GoCD’s job resources.

Cheers,
Aravind

[Denis O]

Hi,

That's the point. Even I chose to NOT clean workspace it was happened. See the log below:

---

[go] Keeping folder pipelines\xxx\workspace
[go] Start to update materials.

[go] Start updating workspace at revision dd9b36482ca1d2a73e032e75770c34f3d0134146 from http://xxxx.git
[GIT] Fetching changes
STDERR: From http://xxx/mainline
STDERR:  - [deleted]           (none)     -> origin/xxx
STDERR:    3fc491a5..dd9b3648  xxx         -> origin/xxx
STDERR:    59726306..3bc01051  xxx         -> origin/xxx
STDERR:  * [new branch]        xxx -> origin/xxx
[GIT] Performing git gc
[GIT] Reset working directory pipelines\xxx\workspace
[GIT] Cleaning all unversioned files in working copy
[GIT] Cleaning submodule configurations in .git/config
[GIT] Updating working copy to revision dd9b36482ca1d2a73e032e75770c34f3d0134146
HEAD is now at dd9b3648 [xxx]:xxx
[GIT] Removing modified files in submodules
[GIT] Cleaning all unversioned files in working copy
[go] Done.

---

So I cannot benefit from incremental build behavior :(

Regards,

Denis

[Aravind SV]

Hello Denis,

It’s hard to say for sure, just from that, but these are some behaviors I believe are true:

1. If there are files left behind inside a git repository checked out folder, then it will run a git clean. No way to stop that. It will need to make sure every job run starts with a clean slate.
   
2. If you haven’t asked it to clean working directory, then it won’t delete and re-clone. So, timestamps of unmodified files should be the same (depending on how git handles them).
   
3. If you’re able to use the checked out files, but have your workspace (which gets “dirty”) in a location outside the repository checkout location, then GoCD shouldn’t touch those directories outside the repo location.
   

I mean: Suppose you configure the material to check out into folder “src1” and then run your build from the parent dir of “src1”, then git clean should run only within “src1”, leaving your workspace in the parent untouched. It isn’t recommended to depend on something left behind, as I said, but that’s one way that might work in this case, assuming it works for your build system to work that way.

Cheers,
Aravind