Can someone please explain environmental variables work in GOCD?

[Gabriel Callaghan]

Hi,

I have a script which does an API call. To authenticate myself, I use a token.

$ curl 'https://ci.example.com/go/api/stages/mypipeline/defaultStage/history' \ 

 -H "Authorization: Bearer ${GOCD_ACCESS_TOKEN}" \

-H 'Accept: application/vnd.go.cd.v3+json'

When I use this api, if in my script I use my personal access token generated by GOCD, the call works. However, if I have the same API call with the same token, but the token is inside the secure variable section of the pipeline, and called using ${GOCD_ACCESS_TOKEN}, I am unable to get authenticated. 

I have tried to hash the token to see if it matches. When doing cksum on my personal token inside my script, I get a different result from if I did cksum <<<${GOCD_ACCESS_TOKEN}.

I am a junior, so I have much to learn and I would apperciate any help in understanding how to resolve this issue.

Thank you :)
Gabes.

[Marques Lee]

Can you post your task config? Just a guess before actually seeing your config is that you might be putting ${GOCD_ACCESS_TOKEN} as a raw argument. Since the arguments are effectively shell-escaped, `curl` might be sending the literal text ${GOCD_ACCESS_TOKEN} instead of interpolating your secret variable.

The solution to that is to either:

• Wrap your `curl` call with `bash -c`, or
  
• Wrap this in a script and call your script

In either case above, curl will receive the interpolated value.

But, let's take a look at your task config to confirm what's going on.

Best,

Marques

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/2a219c35-6c78-4ae3-8070-1e2a54118638n%40googlegroups.com.

[Gabriel Callaghan]

Hi, 

Thank you for your reply - sorry I got pulled away to another task. 

For the task config:

Command: /bin/sh

Arguments: ./{companyName}/game-deploy/checkToken.sh

Inside the script:

#! /bin/bash

 curl "https://{COMPANY NAME}/go/api/stages/testStage/Deploy/history" \
      -H "Authorization: Bearer $GOCD_ACCESS_TOKEN" \
      -H 'Accept: application/vnd.go.cd.v3+json'

echo cksum $GOCD_ACCESS_TOKEN

I have taken your suggestion of wrapping this in a script rather than having it as an argument. It now passes but in the console it shows "cksum *******". How would I be able to compare the cksum of the token if it is hidden from me? I already have the cksum from when I tested on VS Code, I am not sure how to proceed in GOCD. 

From some testing, I found that if I use my personal access token in this script on Notepad++, it will return an answer. If I use  GOCD_ACCESS_TOKEN in Notepad, it will fail, telling me I am not authenticated. However it will pass in the same script in GOCD. 

If I am correct and that when "cksum ****" appears, that means I have passed the auth check, I am still not sure how I managed to fix the problem for when this happens in the future. 

What would your advice for my next steps be?

Thank you,
Gabes.

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAPKX9jZAnKGrupURs%3DrmR5FXceBowfG_WLQFUFdVPEYnqut70A%40mail.gmail.com.

[Sriram Narayanan]

I the past, I've faced a similar issue where some special characters were escaped (or should have been escaped) and I faced such a mismatch.

My solution at that time was to change the passphrase and use other special characters.

Could you share the non-alphabetic non-numeric characters in the GOCD_ACCESS_TOKEN?

Since you mentioning these Windows-based text editors, I wonder if there's some unicode related activity going on. 

Could you also try generating a new API token and using that instead? (You could set up GoCD locally and try, too).

-- Ram

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAKV4bkeVVUaObxjv1S-j478UzuTDjv1d8TEhRJ%3D1U%2BCkkX69cA%40mail.gmail.com.

[Marques Lee]

Hi there, no worries on the delay.

When you say `echo cksum $GOCD_ACCESS_TOKEN` it's probably not doing what you want. You’re literally echoing the work “cksum” followed by the access token. You probably don’t want the `echo` If your intent was to display the checksum of the secret. GoCD will print whatever is on stdout or stderr from `cksum` to the console.

Which brings me to my next point. GoCD masks secrets it knows about from the console. Meaning, if it finds some console text that matches a known secure environment variable, it replaces it with a series of asterisks of a fixed, arbitrary length.

Thus, the fact that it masked the value that you echoed means it printed a secure variable. So I would conclude that your script is receiving the right value from the environment.

So, I’d say that solves the problem you were facing?

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CANiY96YQDRTgzpMPDiG6kMkNDU5kaN1MP9O2wiixm%2BKhzHKLpQ%40mail.gmail.com.

[Marques Lee]

Sorry for the typos in my previous email. Just to make clear what I was saying about “echo”:

1. echo doesn’t shell-eval the rest of its arguments; it prints them, so you’re never actually invoking `cksum`, just printing the word “cksum”

2. the masking is evidence that you’re getting the real value of the secure variable because GoCD is hiding it for security reasons

3. If you really want to print the checksum, then change your conmand to just `cksum $GOCD_ACCESS_TOKEN` (i.e., without the preceding `echo`)

[Fenn]

GoCD borrows from Java standards / defacto standards and also from
Linux standards / defacto standards. It may not be convenient but
learning something about shell notation and Java prior art (e.g. Ant)
can be useful. The best books I can recommend to learn Linux / Unix
shell are Unix Power Tools and Unix in a Nutshell.

https://www.goodreads.com/book/show/172314.UNIX_Power_Tools

https://www.goodreads.com/book/show/331673.UNIX_in_a_Nutshell

Everything you ever wanted to know about the Linux shell (bash) may be
found in the man page (Linux Manual Page) a copy of which is here:

https://linux.die.net/man/1/bash

But it is not an easy read for a beginner.

You might also want to google for "bash tutorial" or "bash beginner
tutorial" to get the hang of things. GoCD uses bash on Linux.

Once you understand how bash works some of these GoCD behaviors might
make more sense.

It also helps to understand Java prior art such as Apache Ant. Ant
documentation on how properties (and environment variables) are used
is here:

https://ant.apache.org/manual/properties.html

https://ant.apache.org/manual/Tasks/property.html

I found this quote particularly instructive:
"When Ant started to support setting properties from environment
variables it ran on Java 1.2 where System.getEnv didn't work. So we
decided to start a command in a new process which prints the
environment variables, analyzes the output and creates the properties.
Once Java 5 became our baseline we could have switched to getEnv but
it returned different results on some platforms so we stuck with the
command approach to remain backwards compatible."

What this basically says is that at one time Java made environment
variables very hard - so people tended to avoid them in the Java world
- and used Properties instead (while the whole rest of the world, and
all other programming languages, used Environment Variables). GoCD
uses both. Sort of. You will see there there are places you can set
Environment Variables in GoCD where you cannot set Properties.

You will quickly notice how Ant uses properties is similar to how Bash
uses environment variables. CoCD tries to use both.

On Windows I think GoCD uses CMD.EXE - some documentation for that is here:

https://ss64.com/nt/cmd.html

Once you understand Environment Variables in Bash and CMD.EXE and
understand Properties in Ant, GoCD makes more sense.

General Note on GoCD Documentation:

The behavior of GoCD is controlled by the XML configuration. The Web
UI documentation assumes that you already understand the XML, so it
doesn't always explain exactly what the Web UI does.

I have often found it helpful to read the "Configuration Reference"
for segments of the "Config XML" which are related to the GoCD Web UI
changes that I am trying to make.

https://docs.gocd.org/current/configuration/configuration_reference.html

Brian Fennell

[Gabriel Callaghan]

Hi,

Thank you all for your helpful comments - I have written them down in my notebook to make sure that I can retain what I've learned here. The authentication problem is fixed, it seems that there may have been a whitespace in there. Have used a GOCD token which works.

My next problem is "Permission denied". I had originally thought this meant that my authentication was wrong, but after I went thru and put the new token in and checked, it seems that the auth isn't the main problem. I am running a script to pull the pipelines from a group, and to approve all of them (if they haven't been approved already). If you guys have some free time, could I please ask for your help on this?

My first script (pushGames.sh):

#! /bin/bash

curl "https:// {companyName} /go/api/admin/pipeline_groups/${PIPELINE_GROUP}" \

       -H "Authorization: Bearer ${GOCD_ACCESS_TOKEN}" \

       -H 'Accept: application/vnd.go.cd.v1+json' \
| jq -r '.pipelines[] | select(.name | contains("Games-")).name'   > testEnvGamesList.txt

cat testEnvGamesList.txt | while read line

do

echo The current pipeline is: $line
./gocd/game-deploy/pushOneGame.sh $line

done

rm testEnvGamesList.txt


My second script (pushOneGame.sh):

#! /bin/bash

fetchPipelineHistory=$( curl " {companyName} /go/api/stages/$1/Approve/history" \

      -H "Authorization: Bearer ${GOCD_ACCESS_TOKEN}" \

      -H 'Accept: application/vnd.go.cd.v2+json')


getResultOfCurrentStage=$( echo $fetchPipelineHistory | jq -r '.stages[].result')

counter=$( echo $fetchPipelineHistory | jq -r '.stages[0].counter')

if [ "$getResultOfCurrentStage" != "Passed" ]
then

echo "https:// {companyName} /go/api/stages/$1/$counter/Approve/run"

curl "https:// {companyName} /go/api/stages/$1/$counter/Approve/run" \

      -H "Authorization: Bearer ${GOCD_ACCESS_TOKEN}" \

      -H 'X-GoCD-Confirm: true' \
      -H 'Accept: application/vnd.go.cd.v2+json' \
      -X POST

fi

I am running these in the task config:

Command: /bin/sh

Arguments: ./{companyName}/game-deploy/pushGames.sh

Both scripts are in the game-deploy folder.

When running this in VS Code (I have it set up to use LF for my end of line so should work with GOCD and Bash), it works. In VScode, I run the "pushOneGame.sh" script, replacing $1 with the name of one pipeline. This works. However, when leaving it as code inside GOCD, it gives me the "permission denied". What did I miss here? I went to Admin > Security > Role config and checked that I was assigned the dev role. There is a message saying the plugins do not support role based authorization. Does this mean I need to request to be given Admin role to check?

@fenn, thanks for linking me to some learning resources, I've saved them and will try to look into them when I am not on work hours.

Cheers,
Gabes.

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAGJwi9gftjW9%3DDZnbQTsKoitEwSXxp8JHC1_RH1SakPaRT1_bw%40mail.gmail.com.

[Marques Lee]

Can you change your task config invoke the script with bash -x and paste the exact output?

That is:

Command: /bin/bash

Args:

  -x

  /path/to/your/script.sh

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAKV4bkehET%3DvU40h8xZpF3zy5vx67Zhee-fdsAM%3DZQyBdvOH1Q%40mail.gmail.com.

[Gabriel Callaghan]

[go] Task: /bin/sh -x ./gocd/game-deploy/pushGames.shtook: 0.130s

+ curl https://{COMPANY NAME}/go/api/admin/pipeline_groups/PHY-Test-Deploy -H Authorization: Bearer ****** -H Accept: application/vnd.go.cd.v1+json

+ jq -r .pipelines[] | select(.name | contains("Games-")).name

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0

100 11098 0 11098 0 0 99089 0 --:--:-- --:--:-- --:--:-- 99089

+ + readcat line

testEnvGamesList.txt

+ echo The current pipeline is: Games-AcesHigh-PHY-Test

+ ./gocd/game-deploy/pushOneGame.sh Games-AcesHigh-PHY-Test

The current pipeline is: Games-AcesHigh-PHY-Test

./gocd/game-deploy/pushGames.sh: 13: ./gocd/game-deploy/pushGames.sh: ./gocd/game-deploy/pushOneGame.sh: Permission denied

+ read line

The current pipeline is: Games-CandyCash-PHY-Test

+ echo The current pipeline is: Games-CandyCash-PHY-Test

+ ./gocd/game-deploy/pushOneGame.sh Games-CandyCash-PHY-Test

./gocd/game-deploy/pushGames.sh: 13: ./gocd/game-deploy/pushGames.sh: ./gocd/game-deploy/pushOneGame.sh: Permission denied (this goes on to list all the other games, i've removed their output to reduce clutter but they all say the same things as the above 2 game output).

+ rm testEnvGamesList.txt
[go] Task status: passed, took: 0.130s
[go] Current job status: passed

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAPKX9jbeF4J0Pg9HuW-f8eysFxtyc0N%3D7wW-pCkPDVnW4mfxfQ%40mail.gmail.com.

[Sriram Narayanan]

Your pushOneGame.sh should be marked executable by the shell account running the script - most likely the “go” user. It currently isn’t.

— Ram

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAKV4bkeddTykAiwoVK7kOU%2B7LfVqws9rorNaNsnirihahqxShg%40mail.gmail.com.

[Fenn]

This might help:

https://linoxide.com/linux-shell-script/make-bash-script-executable-using-chmod/

If not try these:

https://linuxize.com/post/chmod-command-in-linux/
https://www.lifewire.com/uses-of-command-chmod-2201064
https://linux.die.net/man/1/chmod
https://linux.die.net/man/1/chown
https://linux.die.net/man/8/groupadd
https://linux.die.net/man/8/usermod
https://linux.die.net/man/5/group

[Marques Lee]

You should be able to commit an executable permissions change in git.

But you could also side step all of these by running your script explicitly with the shell. That will make your script robust and not rely on exec permissions.

What I mean is do:

/bin/bash ./gocd/game-deploy/pushOneGame.sh

Inside of your script. It will always run and doesn’t depend on exec bit being set.

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAGJwi9iE4DH4zV4mciG%3DVhPuF117iikitHvwbZAUE%2B7xbC-2bA%40mail.gmail.com.

[Gabriel Callaghan]

I had a look at the permissions of the files used, and I get:
-rwxrwxrwx 1 jabes jabes 470 Feb 22 12:09 pushGames.sh
-rwxrwxrwx 1 jabes jabes 751 Feb 22 12:09 pushOneGame.sh

which should mean it all works?

I am currently looking into how to give the go account a way to run these. 

Marques, I did what you said and now I get this:

[go] Task: /bin/bash ./gocd/game-deploy/pushGames.shtook: 0.3s

Error happened while attempting to execute '/bin/bash ./gocd/game-deploy/pushGames.sh'.

Please make sure [/bin/bash ./gocd/game-deploy/pushGames.sh] can be executed on this agent.

[Debug Information] Environment variable PATH: /home/go:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Does this mean all my files have permissions set so I don't need to use chmod, but I need to poke around in the settings of GOCD more? I'm guessing this agent needs to be edited more? is the go user mentioned by Ram the same as the agent?

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAPKX9jZmhUQCaciGaD%3DB486CWsWM4Z-VvgNbCmOj-eEedhifNw%40mail.gmail.com.

[Sriram Narayanan]

On Mon, Feb 22, 2021 at 9:43 AM Gabriel Callaghan <gab...@rubyplaynetwork.com> wrote:

I had a look at the permissions of the files used, and I get:
-rwxrwxrwx 1 jabes jabes 470 Feb 22 12:09 pushGames.sh
-rwxrwxrwx 1 jabes jabes 751 Feb 22 12:09 pushOneGame.sh

which should mean it all works?

I am currently looking into how to give the go account a way to run these. 

Marques, I did what you said and now I get this:

[go] Task: /bin/bash ./gocd/game-deploy/pushGames.shtook: 0.3s

Error happened while attempting to execute '/bin/bash ./gocd/game-deploy/pushGames.sh'.

Please make sure [/bin/bash ./gocd/game-deploy/pushGames.sh] can be executed on this agent.

[Debug Information] Environment variable PATH: /home/go:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Does this mean all my files have permissions set so I don't need to use chmod, but I need to poke around in the settings of GOCD more? I'm guessing this agent needs to be edited more? is the go user mentioned by Ram the same as the agent?

Actually, Marques' suggestion to use bash and run the script is the best approach for the reasons he gave :) 

What you are facing right now is not a GoCD setting issue, but something specific to the invocation. 

Is the directory "gocd" within the current directory?

What do you see if you passed -x to bash? (/bin/bash -x ./gocd/game-deploy/pushGames.sh) ?

-- Ram

[Marques Lee]

I was referring to putting that within your script, not as a separate task.

I’ve also noticed you use /bin/sh (maybe your agent doesn’t have bash?).

You are on a Windows machine, right? Not sure how ntfs maps Linux file permissions, if at all. It could very well be the case that everything on your file system is considered 0777. However, I’m assuming the GoCD agent is running on Linux, which does need the permissions. So what you see locally probably does not reflect what the agent sees.

At any rate, I would make that script call part of your patent script that you showed us earlier (ie, the script with the curl calls) and not a separate task. It should get you farther.

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAKV4bkd4e2LMzr9Fab%3DLcWhp0hfVPQj%2Bzj%2BwLH-%2BdNWe4sOiTw%40mail.gmail.com.

[Marques Lee]

I believe it was line 13 of pushGames.sh. That’s where you should invoke the script with /bin/sh or /bin/bash explicitly.

[Marques Lee]

Gah! Sorry I might be confusing you more than helping. I misread what you last pasted (confused which script was being executed in the task).

Keep the task config as what you had before:

Command: /bin/sh

Args:

  ./gocd/game-deploy/pushGames.sh

Then on line 13 of pushGames.sh, invoke pushOneGame.sh like this:

/bin/sh ./gocd/game-deploy/pushOneGame.sh

You should not get the same permission denied error.

[Gabriel Callaghan]

Thanks Marques, that solved the problem! Really appreciate your help in all of this.

Also thanks Ram and Fenn for your input :) 

So next time I get a permission error, make sure to ensure that if im running a bash script that calls another bash script, to put /bin/sh in front of it.

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAPKX9ja0KhGf6_Vp2sxRJ%2BYFMcwYD1cAv0SpMmu38skxv%2BNGOQ%40mail.gmail.com.

[Marques Lee]

Glad I could help :)

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAKV4bkfyDbOAPqUEo%2BTWajXe6pWV%2Bouca-Czq2Tixq%2BgH%3D1GyQ%40mail.gmail.com.