GoCD - Deploy using an artefact rather using git checkout - URGENT please

[Prakash Tirumalareddy]

Hello Everyone

Is there anyway we can configure a particular stage not to do git-checkout and instead download the artefacts from external repository (such jfrog artifactory) which was built and pushed in previous stage?

Currently what have seen that at every stage it does the git-checkout for the commit hash. 

I know there is a stage configuration which says not to checkout the material. But when we enable this option then there is no way in that stage to do required task.

Your help will be greatly appreciated.

Regards

Prakash

[Helge Walter]

Hello Prakash,

there are 2 ways to achieve this:

1. You can store the created artifact from the previous stage as a resulting artifact (https://docs.gocd.org/20.4.0/configuration/managing_artifacts_and_reports.html) and then fetch the artifact in the next stage with the "Fetch Artifact" task for further use.

2. Upload the artifact to the Artifactory in the previous stage and retrieve the artifact in next stage using a simple curl task.

I would prefer the first option because all the artifact handling is done by GoCD internally.

Regards,

Helge

[Prakash Tirumalareddy]

Hello Helge

Thanks for your kind reply. 

Either of the option is not solving the git-checkout at every stage.

Is there way we can achieve to run stage by executing download the artefacts from Artifactory. (Can we put curl command in the pipeline.yaml itself? if so can you please give me some example?)

Thanks again!!!

Regards

Prakash

[Jason Smyth]

Hello Prakash,

You wrote in your initial post:

I know there is a stage configuration which says not to checkout the material. But when we enable this option then there is no way in that stage to do required task.

Why can GoCD not do the required task if materials are not fetched from git?

I think once you know the answer to this question you should be well on your way to solving the issue.

Hope this helps,

Jason

[Prakash Tirumalareddy]

Thanks for your email Jason.

When we try this option, there is NO files to execute the command. My make file is not there so that stage can't do anything.

What I am after is, the stage should be download the artefact from Artifactory and then run the make command.

Regards

Prakash

[Helge Walter]

Hello Prakash,

I think you should describe your setup and relation of the files you are talking about a little more in detail:

1. What is this artifact on the Artifactory?
2. Where is this make file located? In the artifact or in the source code repository?
3. If it's in the source code repository and you don't want to check this out from the repository-- Why do you expect it to work in GoCD?
4. Why do you store the artifact in the Artifactory in the first stage and then want to get it from there in the second stage and not use the GoCD artifact management features?
5. You can put the curl command as a regular "Custom Task" (which is basically as OS command line call) in the yaml.

Regards,

Helge

[Prakash Tirumalareddy]

Hello Helge 

Thanks for your email and help.

We don't want to use the GoCD inbuilt artefact management hence we are using Artifactory for artefacts management.

Makefile is part of the artefact and in the second stage we want to download the artefact and run the make command to do next steps i.e. make deploy

Regarding your no.5 point about curl command, is there example to specify in pipeline yaml file.

Regards

Prakash

[Helge Walter]

Hello Prakash,

for an example see here: https://github.com/tomzo/gocd-yaml-config-plugin#example

tasks: # ordered list of tasks to execute
- exec: # indicates type of task
command: make
arguments:
- "VERBOSE=true"

You can specify your curl command here instead of the "make" call. The curl call is specific to your artifactory and the repository type you use -- so it's difficult for me to give you a neaningfull example.

Regards,
Helge

[Prakash Tirumalareddy]

Hello Helge

We tried using the curl command:

6-DeploySIT1: 

    fetch_materials: no

    clean_workspace: yes

    approval: manual

    environment_variables:

        app_environment: nonprod

        verify_repo: verify-repo

    jobs:

        deploy:

        tasks: 

        - exec:

            command: /usr/bin/curl -L -k -v -u <username>:<password> -X GET "https://artifactory.xys.com/service/rest/v1/search/assets/download?group=/<artefact_name>/master/${GO_PIPELINE_LABEL}&name=<artefact_name>/master/${GO_PIPELINE_LABEL}/${GO_REVISION_MYGIT}.zip" -o "${GO_REVISION_MYGIT}.zip" && unzip -o "${GO_REVISION_MYGIT}.zip" -d .'

        - exec: 

            arguments:

            - deploy

            command: make

Error is:
Check your command whether it can be executed on your agent.

Regards

Prakash

[Jason Smyth]

Hello Prakash,

What do you see in the console output beyond the error message? I believe the raw command should get printed to console which would make troubleshooting easier.

Here is an example from the raw output of one of our Pipelines that uses Custom Commands:

!!|10:43:38.980 [go] Task: cmd /c ""C:\Windows\Microsoft.NET\Framework\v4.0.30319\msbuild.exe" "tfs\path\to\project.proj" "/t:Release\""

I suspect that the issue you are having is that GoCD tries to expand the environment variables used in your command before they have been defined but, again, you should be able to see that from the console output.

If the console output looks right, including all of the environment variables, try running it from the command line on the agent to see what the error message is there.

Hope this helps,

Jason

[Prakash Tirumalareddy]

Hello Jason

Thanks for your email.

If I updated variable manually the command works in agent.

I am getting the following error: which does not tell me that is related with populating the variable or not.. 

[go] Job Started: 2020-08-10 19:53:21 AEST

[go] Start to prepare xx-prj/77/6-DeploySIT1/1/deploy on test.internal [/var/lib/go-agent]

[go] Cleaning working directory "/var/lib/go-agent/pipelines/ xx-prj" since stage is configured to clean working directory

[go] Skipping material update since stage is configured not to fetch materials

[go] Start to build xx-prj /77/6-DeploySIT1/1/deploy on test.internal [/var/lib/go-agent]

[go] Task: /usr/bin/curl -L -k -v -u <username>:<password> -X GET "https://artifactory.com/service/rest/v1/search/assets/download?group=/xx-prj/master/${GO_PIPELINE_LABEL}&name=xx-prj/master/${GO_PIPELINE_LABEL}/${GO_REVISION_MYGIT}.zip" -o "${GO_REVISION_MYGIT}.zip" && unzip -o "${GO_REVISION_MYGIT}.zip" -d .'took: 0.2s

Error happened while attempting to : /usr/bin/curl -L -k -v -u <username>:<password> -X GET "https://artifactory.com/service/rest/v1/search/assets/download?group=/xx-prj/master/${GO_PIPELINE_LABEL}&name=xx-prj/master/${GO_PIPELINE_LABEL}/${GO_REVISION_MYGIT}.zip" -o "${GO_REVISION_MYGIT}.zip" && unzip -o "${GO_REVISION_MYGIT}.zip" -d .] can be executed on this agent. 

[Debug Information] Environment variable PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

[go] Task status: failed, took: 0.2s

[go] Current job status: failed

[go] Start to create properties xx-prj/77/6-DeploySIT1/1/deploy on test.internal [/var/lib/go-agent]

[go] Start to upload xx-prj/77/6-DeploySIT1/1/deploy on test.internal [/var/lib/go-agent]

[go] Job completed xx-prj/77/6-DeploySIT1/1/deploy on test.internal [/var/lib/go-agent]

Regards

Prakash

[Jason Smyth]

Hello Prakash,

What happens if you add "echo" to the start of the command? Assuming it works as intended, the output of echo should make it clear whether or not the environment variables are expanding properly.

Another potential issue is with the way the command is constructed in the Pipeline definition. GoCD may be trying to execute a command called '/usr/bin/curl -L -k -v -u <username>:<password> -X GET "https://artifactory.com/service/rest/v1/search/assets/download?group=/xx-prj/master/${GO_PIPELINE_LABEL}&name=xx-prj/master/${GO_PIPELINE_LABEL}/${GO_REVISION_MYGIT}.zip" -o "${GO_REVISION_MYGIT}.zip" && unzip -o "${GO_REVISION_MYGIT}.zip" -d .', rather than running curl with the rest of the string as arguments.

We've also had trouble getting quoting just right in our custom commands. Here's the XML definition of the command from my previous message:

              <exec command="cmd">

                <arg>/c</arg>

                <arg>""#{msbuild}"</arg>

                <arg>"#{projectRootDirectory}\#{projectName}\#{projectName}.proj"</arg>

                <arg>/t:Release"</arg>

                <runif status="passed" />

              </exec>

Note that each space-delimited argument is in its own <arg> element, which translates to each on its own line if editing in the GUI. Note as well that each argument that could contain a space is enclosed in double quotes. Lastly, to get this to work, we had to add a double quote to the start of the second argument (first one after "/c") and to the end of the last argument so that the final argument list becomes /c ""msbuild" "path/to/project.proj" /t:Release".

Hope this helps,

Jason

[Aravind SV]

Hello,

In case it helps, I see Jason had responded earlier to something like this:

https://groups.google.com/d/msg/go-cd/fjOz211CIhY/dThcC3P2AQAJ

Cheers,
Aravind

PS: Thanks for helping people, Jason. :)

[Prakash Tirumalareddy]

Thanks Jason.

Your suggestion worked!!!

Thank you very much!!!

Regards

Prakash