shouldn't required resources also be at the pipeline level?

[Josh]

QUESTION:

Shouldn't we also be able to communicate a resource requirement at the pipeline level, and not just inside a single job?

I get that it definately needs to be at the job level since that's the smallest unit of work and some machines can't execute certain tasks.

But at the value-stream/pipeline/business level, you also want to be able to have some pipelines compiling on preferred resources, no?

is there a better way to accomplish this?

or perhaps this already is possible and i'm missing it.

i looked closely at the config since sometimes you can do something simple that is not possible inside the UI, but I'm not seeing it.

To restate use case:  We have some pipelines that are given higher preferences for agent/build resources.   Wanting to do a lot more of this, but it's tricky because resources can only be defined at the job level (in the UI).     Also we use a lot of templates, so having resources at job level means we end up having lots of alsomost identical templates that only vary by the resources used (which somewhat defeats the point of the templates and the value of gocd in this respect).

hoping there is a config hack or maybe i'm missinig something.

also if this could be done in a plugin, any color there would be helpful (and i would make sure it's open sourced if need be).

thx

ps i keep using other ci/cd products and gocd is still one of the all around bests.

[Chad Wilson]

Have you tried to use "environments" (or a mix of environments and resources) to achieve what you are trying to?

When scheduling jobs it's the combination of the resource and the environment that are matched to an agent, but the relevant environment is declared at the pipeline level like you refer to. Don't need to get hung up on the name so much. Yes, you can have "environment variables" attached to an environment and propagate those to all pipelines within it, but you don't have to use them like that.

Alternatively, to make the templates less duplicated and allow the resource to flow from the pipeline using the template, you could try using template parameters in the resources field? e.g #{job-resoure-requirement}? If there are only a small number of different resources used across the stages/jobs, you could use the parameters to "model" this I imagine.

-Chad

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/a9a4ba2c-b1c9-4202-9408-3e2566929b59n%40googlegroups.com.

[Ashwanth Kumar]

Use Parameters.

[Joshua Franta]

chad, thx for your reply.

re: using environments rather than resources... environments can't be defined at the pipeline level either though?

or i guess it's more correct to say that using environments is a bit of a side-car feature, in that we use interact w/environments through a different prisim/ui/config (no biggie) but also seems it's mutually exclusive to maximizing overall usage of agents.    for us if a given host can execute something (a pipeline, a job) it should.  and if it can't, it shouldn't.

trying to force a hard divider can be useful for prod/qa staging, but it seems to limit just being able to have pipelines declare their needs.

maybe i'm missing what you're saying but i don't think environments are functionally equivalent to resources?

we use template parameters extensively already.  

eg we even templatize further inside our own jobs by re-using scripts that interact with template parameters on most commonly used templates (eg our most popular template has maybe 10-15 pipelines).

however this is more of a job specific thing since it's at the job level.

if you're saying we could change every pipeline to read this at a pipeline level is a non trivial change to every job.

that's ok but i guess my overall question tho would be that if a given job decided it couldn't execute the pipeline parameters... it has no way to pass the job to another agent?

in such an example it would just fail the job, no?   again maybe i'm not following but this seems to not allow the business/value level to declare minimum needs

(environments seem like they are more about maximimal requirements, but i'm no expert)

You received this message because you are subscribed to a topic in the Google Groups "go-cd" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH8zGo6mu0ss0jCCyw0D7Hw4JOwEwfcfNu20yqo0aRRdWw%40mail.gmail.com.

[Joshua Franta]

thanks ashwanth- we already use parameter a ton.

i don't understand how a job could look at a parameter and force the job to a more capable agent.

i can definitely have a job fail based on a pipeline level parameter, but obviously trying to keep everything green here.

if you're sure it's possible, can you provide a bit more of an example?

You received this message because you are subscribed to a topic in the Google Groups "go-cd" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAD9m7CxMqUNkCqyx5D5P4HMb0ocMVQYx__VDz1AYN%2B5ukovB3w%40mail.gmail.com.

[Chad Wilson]

On Sat, Jul 22, 2023 at 8:21 PM Joshua Franta <jos...@pracplay.com> wrote:

re: using environments rather than resources... environments can't be defined at the pipeline level either though?

A pipeline is assigned to 0 or 1 environments (via the Admin  > Environments UI if not using pipelines-as-code) - thus it's at the pipeline level by definition. It defines a scheduling requirement for all jobs in that pipeline. Which seems what you asked for with "able to communicate a resource requirement at the pipeline level" right?

or i guess it's more correct to say that using environments is a bit of a side-car feature, in that we use interact w/environments through a different prisim/ui/config (no biggie) but also seems it's mutually exclusive to maximizing overall usage of agents.    for us if a given host can execute something (a pipeline, a job) it should.  and if it can't, it shouldn't.

trying to force a hard divider can be useful for prod/qa staging, but it seems to limit just being able to have pipelines declare their needs.

maybe i'm missing what you're saying but i don't think environments are functionally equivalent to resources?

I didn't imply they were functionally equivalent, but I did try to imply they were a different mechanism of defining a requirement on a job's scheduling, at the pipeline level. If a pipeline is assigned to an "environment", its jobs must be scheduled on agents that also declare they support that "environment". Similarly if a pipeline job declares a resource requirement, the agent must also have that resource declared for it to be assigned. This is a very similar, but different level of configuration of a scheduling requirement, no? https://docs.gocd.org/current/configuration/managing_environments.html

Anyway, perhaps I don't understand what you are trying to achieve. If you are currently trying to "prioritise" pipelines by using resources you can also "prioritise" pipelines by having pools of agents, say, dedicated to an environment you call "high-priority". As I said, "Don't need to get hung up on the name [environment]".

 

 

we use template parameters extensively already.  

eg we even templatize further inside our own jobs by re-using scripts that interact with template parameters on most commonly used templates (eg our most popular template has maybe 10-15 pipelines).

however this is more of a job specific thing since it's at the job level.

if you're saying we could change every pipeline to read this at a pipeline level is a non trivial change to every job.

You said you had many templates that varied only by the "resources" field for jobs. If that is the stated problem then parameters are a possible solution to remove duplication, no?

 

that's ok but i guess my overall question tho would be that if a given job decided it couldn't execute the pipeline parameters... it has no way to pass the job to another agent?

That's the same problem you have currently if the resource is typoed or wrong inside the template, no? If the resource requirement has no available agents, then it can't be scheduled.

 

in such an example it would just fail the job, no?   again maybe i'm not following but this seems to not allow the business/value level to declare minimum needs

(environments seem like they are more about maximimal requirements, but i'm no expert)

I'm not following what you're trying to say here, sorry.

Perhaps this would be easier if you gave a specific example of how you achieve "have some pipelines that are given higher preferences for agent/build resources" currently, rather than talking in abstract terms?

-Chad

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CABr%2BOtrG%2B0X3y4B6AWnN7N0K-OSpcKb4KdG-LbS8fCnMOR8Zdw%40mail.gmail.com.

[Ashwanth Kumar]

Taking another stab after Chad at answering the following question,

> We have some pipelines that are given higher preferences for agent/build resources.   Wanting to do a lot more of this, but it's tricky because resources can only be defined at the job level (in the UI).     Also we use a lot of templates, so having resources at job level means we end up having lots of alsomost identical templates that only vary by the resources used (which somewhat defeats the point of the templates and the value of gocd in this respect).

Since you are asking resource to be at a pipeline level, you can assign the resource for every job within a template to be #{pipline_resource} (a pipeline parameter) and set the value of pipeline_resource parameter at a pipeline level so all the jobs now have the same resource and the value is now essentially controlled at a pipeline level. This way you don't need duplicate templates varying by job resources anymore. This is what Chad meant when he said "You said you had many templates that varied only by the "resources" field for jobs. If that is the stated problem then parameters are a possible solution to remove duplication, no?"

Thanks,

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CABr%2BOto3xJnzTVup0eGk0zDiaocSrt42bBkRcfwdVKwxdit%3Dtw%40mail.gmail.com.

--

Ashwanth Kumar / ashwanthkumar.in

[Joshua Franta]

appreciate so many responses.  i think we're a little apart so i'll take the suggestion to give our example:

GROUPA = machines that have less beefy hardware

GROUPB = more expensive machines

we'd like to:

1. have a couple of pipelines only run on the more expensive machines 
   
2. have all other pipelines run in either group (next available agent)

perhaps this was not clear from my previous explanations, but a couple of people have suggested pipeline parameters.

EXAMPLE

a pipeline parameter is only going to be available to the job after the job has already been assigned an agent, right?

so if i have a pipeline called 'Priority' w/a parameter  called "group-id" and the pipeline has a 'Job' that is a shell script:

----

##!/bin/sh

agent_resource="$GO_AGENT_RESOURCE_VARIABLE"

if ! echo "#{group-id}" |grep -q "$agentr_esource"; then 

   echo "agent can't run #{group-id} pipelines"

   ## won't this will make my pipeline fail when I want it to simply try another agent?

    exit 1 

fi

----

or perhaps people saying this know of some environment variable that where we can request another agent?

obviously pipeline parameters themseles don't do anything, so i'm confused how i can affect assignment in a job that requires an agent before it runs.

this 2nd part is what i don't get above

appreciate any clarifications or suggestions thx

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH8d2amyXbcEhe8hOnbm6BmS-moT9u7Oo%3D4zzU%2BoXfnekQ%40mail.gmail.com.

[Sriram Narayanan]

You would have agents on the various machines, tag the agents with specific resource tags, and tag specific jobs with those resource tags. 

— Sriram

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CABr%2BOtr%3D-R3fKE0devgPVKgNbHGVnsn5E5OwxuP%3DMt1No_RNdg%40mail.gmail.com.

[Chad Wilson]

With that description, if you want to use environments rather than resources (and assuming you don't use environments for any other purpose), I would

1) Create 2 environments "fast" and "any"

2) Map agents to environments

agents on GROUPA = machines that have less beefy hardware

- declare environments "any" when registered

agents on GROUPB = more expensive machines

- declare both environments "fast" and "any" when registered

3) When configuring your pipelines

1. have a couple of pipelines only run on the more expensive machines <-- add these pipelines to "fast" environment
   
2. have all other pipelines run in either group (next available agent) <-- add these pipelines to "any" environment

This should give you roughly the semantics you say you want, but note it won't prioritise the GROUPB agents for use by the "couple of pipelines only run on the more expensive machines", it will just ensure they never run on the slower machines/agents. Something equivalent could also be done with resources.

There is no way to "try another agent" from inside the actual job's tasks. In this sense, the contents of tasks/scripts aren't relevant to scheduling. The GoCD resources and environments have to be known at schedule time. When you use pipeline parameters, they are realised at configuration time as when you create a pipeline from a template, it will force you to set the parameter values.

To clarify, when you talked earlier about "a resource requirement" are you actually referring to GoCD's concept of resources, or were you talking in a generic sense? The answers are assuming you are talking about GoCD resources but now I am more confused by your shell script. If you want to use resources (rather than environments) to affect scheduling, while still avoiding duplication of your templates, we are suggesting you use a parameter like this, not put it into some task content. You are setting the parameterized value into the field that determines the job's scheduling, not something that happens at execution time like a task. But again, if your goal is to control scheduling at pipeline level, for all jobs in a pipeline, you don't need to use resources, and can just use environments as in my earlier example above.

-Chad

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CABr%2BOtr%3D-R3fKE0devgPVKgNbHGVnsn5E5OwxuP%3DMt1No_RNdg%40mail.gmail.com.

[Joshua Franta]

chad thanks for your answer.

i think the main source of confusion is that I thought parameters could only be referred to in scripts!

I didn't know you could refer to them inside of other configuration properties!

Is this documented?   Regardless that's super useful, there's probably some other things that can be cleaned up knowing that.

I tried this on a pipeline w/out any template and it worked as described.   Just put the parameter reference in resource- UI accepts as long as parameter exists and works.

I still have a question about how this works in examples using templates.

If we didn't define the pipeline parameter by default, how would gocd interpret what I'm guessing would be a blank resource?

eg we have 

1. a pipeline template called FAST_OR_SLOW_PIPE 
2. every pipeline implementing this template defines a parameter called  PIPE_RESOURCE_PARAM

What happens if somebody only defines PIPE_RESOURCE_PARAM when the pipeline is FAST?

If it's left as empty for ANY-aka-SLOW resources, will gocd intepret this as a blank resource requirement and fail?

Or will it ignore blank resources?

It's not a big deal to define it anywhere, but I'm curious how it works when it's undefined since we didn't have this use case previously.   

If anybody knows would appreciate sharing.

For the time being I think this is a better solution than environments, which seems a bit more opinionated and perhaps limiting for other solutions.

However chad's demonstrate of the a non-traditional use of environments is helpful for thinking about other ways to use them.

thanks chad and everybody.    

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH8XmX4y11RT2PVXeUxQd8hvK0UBgbcx8ja-MWuOVqpfag%40mail.gmail.com.

[Sriram Narayanan]

On Mon, Jul 24, 2023 at 8:44 PM Joshua Franta <jos...@pracplay.com> wrote:

chad thanks for your answer.

i think the main source of confusion is that I thought parameters could only be referred to in scripts!

I didn't know you could refer to them inside of other configuration properties!

Is this documented?   Regardless that's super useful, there's probably some other things that can be cleaned up knowing that.

I tried this on a pipeline w/out any template and it worked as described.   Just put the parameter reference in resource- UI accepts as long as parameter exists and works.

I still have a question about how this works in examples using templates.

If we didn't define the pipeline parameter by default, how would gocd interpret what I'm guessing would be a blank resource?

eg we have 

1. a pipeline template called FAST_OR_SLOW_PIPE 
2. every pipeline implementing this template defines a parameter called  PIPE_RESOURCE_PARAM

What happens if somebody only defines PIPE_RESOURCE_PARAM when the pipeline is FAST?

If it's left as empty for ANY-aka-SLOW resources, will gocd intepret this as a blank resource requirement and fail?

Or will it ignore blank resources?

It's not a big deal to define it anywhere, but I'm curious how it works when it's undefined since we didn't have this use case previously.   

If anybody knows would appreciate sharing.

For the time being I think this is a better solution than environments, which seems a bit more opinionated and perhaps limiting for other solutions.

However chad's demonstrate of the a non-traditional use of environments is helpful for thinking about other ways to use them.

I suggest that you explore using Resources to definitely assign specfic agents or a collection of agents for specific Jobs. Within a Stage in an Pipeline, you can have one Job that runs on a FAST agent, another Job that runs on a regular agent, and a third that could run in either - all managed by the appropriate resource tags.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CABr%2BOtoymQdOg0poCtRYDRfxgX4UKfukvtvvpCOZicadaPKWyA%40mail.gmail.com.

[Chad Wilson]

On Mon, Jul 24, 2023 at 8:44 PM Joshua Franta <jos...@pracplay.com> wrote:

chad thanks for your answer.

i think the main source of confusion is that I thought parameters could only be referred to in scripts!

I didn't know you could refer to them inside of other configuration properties!

Is this documented?   Regardless that's super useful, there's probably some other things that can be cleaned up knowing that.

https://docs.gocd.org/current/configuration/admin_use_parameters_in_configuration.html#rules-around-usage-of-parameters

 

I tried this on a pipeline w/out any template and it worked as described.   Just put the parameter reference in resource- UI accepts as long as parameter exists and works.

I still have a question about how this works in examples using templates.

If we didn't define the pipeline parameter by default, how would gocd interpret what I'm guessing would be a blank resource?

eg we have 

1. a pipeline template called FAST_OR_SLOW_PIPE 
2. every pipeline implementing this template defines a parameter called  PIPE_RESOURCE_PARAM

What happens if somebody only defines PIPE_RESOURCE_PARAM when the pipeline is FAST?

If it's left as empty for ANY-aka-SLOW resources, will gocd intepret this as a blank resource requirement and fail?

Or will it ignore blank resources?

I'm not sure - perhaps just try it empirically? It could either fail or see it as blank i.e "no resource requirement" - I don't think there's a strong case for either behaviour being more correct.

-Chad

 

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CABr%2BOtoymQdOg0poCtRYDRfxgX4UKfukvtvvpCOZicadaPKWyA%40mail.gmail.com.

[Jason Smyth]

Hi Josh,

I think there may be some confusion here regarding GoCD terminology and common concepts.

> i think the main source of confusion is that I thought parameters could only be referred to in scripts!

> I didn't know you could refer to them inside of other configuration properties!

To the best of my knowledge, Parameters (GoCD concept) cannot be referenced in scripts. You can call a script that uses parameters (scripting concept), but as far as I know, GoCD Parameters are not persisted in the Agent's runtime environment unless they are somehow passed in via the Task definition. Are you sure you aren't thinking of Environment Variables (GoCD concept)? Environment Variables can be defined in a few different places in GoCD. As the name suggests, these values are persisted in the Agent's runtime environment when a Task is executed.

> I still have a question about how this works in examples using templates.

> If we didn't define the pipeline parameter by default, how would gocd interpret what I'm guessing would be a blank resource?

If a Template references a Parameter then every Pipeline that uses that Template _must_ define that Parameter. Depending on how the Parameter is used in the Template, leaving the Parameter Value blank may be valid.

In the case of using Parameters to define Resources, my testing shows that each Parameter must define a single, valid, Resource. That is, if you want to specify multiple Parameterized Resources, you must use multiple Parameters. You cannot, for example, provide a Parameter Value of "foo, bar" to make your Pipeline's Job depend on the "foo" and "bar" Resources. GoCD rejects the configuration as invalid if you try to save it. Similarly, GoCD rejects the configuration as invalid if a Parameter is used in the Resource field and you try to leave its Value blank.

Regarding your specific use case, you can solve it using either Environments or Resources. The right solution depends on your requirements and how you want to reason about your environment.

The way I understand it, in the context of this discussion you have 2 groups of Agents (Agents1 and Agents2) and 2 groups of Pipelines (PipelinesA and PipelinesB). The Pipelines in PipelinesA can run on any Agent, but the Pipelines in PipelinesB must run on the Agents in Agents2. We will ignore the fact that Pipelines can contain multiple Stages and multiple Jobs and assume either that all of the Pipelines contain a single Stage with a single Job, or that the scheduling requirements are the same for all Jobs in a given Pipeline. You have also talked about Pipeline priority.

Based on this, I assume your requirements are one of the following:

1. Agents should be used to the full extent possible; the workload in PipelinesB is heavier so those Pipelines must not run on Agents1, or

2. Pipelines in PipelinesB have a higher priority than those in PipelinesA; Agents in Agents2 should take Jobs from PipelinesA only if there are no pending Jobs for PipelinesB.

GoCD supports the first scenario. You can achieve this by assigning 2 Resources/Environments. Pipelines in PipelinesA get 1 Resource/Environment; Pipelines in PipelinesB get the other. Agents in Agents1 get the PipelinesA Resource/Environment; Agents in Agents2 get both.

GoCD does not support the concept of priority, so scenario 2 is not supported. The best you could accomplish would be to map each group of Pipelines to a single group of Agents.

Hope this helps. If I'm way off base it might help me better understand your situation if you would provide snippets of your actual configuration.

Cheers,

Jason

[Joshua Franta]

Jason, your knowledge here is off. Parameters can be used in scripts, see a previous email I this thread that shows how it works.

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/43cb7edd-5e0d-4b06-9dac-024e40509d43n%40googlegroups.com.

[Chad Wilson]

If you read Jason's message a bit more closely he is conveying that the script's runtime environment has no knowledge of the parameters - not that they can't be used at all.

They are just tokens that have already been 'realized' or replaced into the content by the time the script/task runs. So the scripting environment itself doesn't know that there were parameters used to generate the content to run/execute and you can't meta-program based on them inside the script's logic. (unlike environment variables)

I believe this is in reference to the earlier script-based example you gave which is a little confusing.

Anyway, seems you have a way forward here for your core requirement.

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CABr%2BOtp%3DquwcOTJv4jh8xKXf6JpOv5vHADqNkAwqaCS2yEZ%2BtA%40mail.gmail.com.

[Joshua Franta]

On Mon, Jul 24, 2023 at 10:44 AM Chad Wilson <ch...@thoughtworks.com> wrote:

On Mon, Jul 24, 2023 at 8:44 PM Joshua Franta <jos...@pracplay.com> wrote:

chad thanks for your answer.

i think the main source of confusion is that I thought parameters could only be referred to in scripts!

I didn't know you could refer to them inside of other configuration properties!

Is this documented?   Regardless that's super useful, there's probably some other things that can be cleaned up knowing that.

https://docs.gocd.org/current/configuration/admin_use_parameters_in_configuration.html#rules-around-usage-of-parameters

thanks and i even checked the docs for first version of gocd we used, feature was still there.

just missed it.  

 

If it's left as empty for ANY-aka-SLOW resources, will gocd intepret this as a blank resource requirement and fail?

Or will it ignore blank resources?

I'm not sure - perhaps just try it empirically? It could either fail or see it as blank i.e "no resource requirement" - I don't think there's a strong case for either behaviour being more correct.

-Chad

if somebody does it accidentally i will try to report back.

looking at the source, this bit here seems to suggest that 'NULL' resources ARE allowed.  a superficial glance suggests that 'empty' resources are ignored and GO_AGENT_RESOURCES isn't set when this is the case:

-----

// via: https://github.com/gocd/gocd/blob/f5859f59171fb95e1a40bf5911e4f6d3f2de43e9/server/src/main/java/com/thoughtworks/go/server/service/BuildAssignmentService.java#L309

                    if (CollectionUtils.isNotEmpty(agent.getResourceConfigs())) {
                        // Users relying on this env. var. can test for its existence rather than checking for an empty string
                        environmentVariableContext.setProperty(GO_AGENT_RESOURCES, agent.getResourceConfigs().getCommaSeparatedResourceNames(), false);
                    }

//...

BuildAssignment buildAssignment = BuildAssignment.create(job, pipeline.getBuildCause(), builders, pipeline.defaultWorkingFolder(), environmentVariableContext, requiredArtifactStores);

-----

chad thx again for your close reading of my answers.  not sure i would've discovered the misunderstanding so quickly otherwise!

really appreciate all the help from everybody.

-j

[Joshua Franta]

yes- i'm aware that parameters are replaced before the script is written into the agent and executed.

while it's correct that scripts don't know where the information inside them comes from, this is true for any script not just ones used by gocd.

perhaps these are more semantic level points.

eg i don't know what you mean about meta programming-  you can put a pipeline parameter into a variable and do whatever you like with it, same as env var.

but again that's little to do with gocd specifically.

regardless- yes got what i needed!

thx again to everybody who tried to help

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH9z%2Bkpdant8aRrDujZ-nag-mb6m_res0KLrPtYoef7Pqw%40mail.gmail.com.

[Chad Wilson]

Yeah, just semantics. 😅 Like you can't know the parameters "name" or anything like that in the script itself, or do things like "iterate on all parameter names" - you can only know their injected value, so if you want to make logic depend on a parameter's name, you have to assign it to a local var in the script first and work with the variable known to the script, e.g with GoCD parameter PIPE_RESOURCE_PARAM=fast and task/script content

#!/bin/sh
agent_resource="#{PIPE_RESOURCE_PARAM}"

At runtime, the shell runtime execution environment will see the below, with parameter name gone

#!/bin/sh
agent_resource="fast"

So you can't write logic that varies based on the parameters' names defined in GoCD (that's what I mean by meta-programming). But can still achieve most things you might like to with other workarounds 😬

By contrast, with env vars you can iterate on the entire env and see if anything has been defined at GoCD level, e.g with GoCD env var PIPE_RESOURCE_ENV_VAR=fast

#!/bin/sh
env # <--- will print PIPE_RESOURCE_ENV_VAR=fast (among other things)

So slightly different semantics due what the execution environment knows.

-Chad

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CABr%2BOtrvLfrhZzs%3DXTDKMFOEQLEiEbxcVtvUk3Lq%3DApFod-XQg%40mail.gmail.com.

[Joshua Franta]

semantics in a broad sense of arguing something not relevant to the thread.

everybody who responded said something meaningful to them, no doubt there.

I'm pretty sure it's not a good idea to introduce logic based on the name of a variable and not it's content.

(obv not the same as saying it's not possible or not ever useful).

even tho it's not relevant to the thread, i'd love to see your explanation of when this is a recommended suggestion in a CD context

(i'm guessing you have one)

thx chad!

 

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH_u8_45Qyeg%3DESmNUFbdHh-aWv%2BdznV5LZ%3D1JwTOLwG%3Dg%40mail.gmail.com.

[Jason Smyth]

Hi all,

 

I think my confusion comes from the word “script”.

 

Unless I am missing something, in vanilla GoCD a “script” is an external thing invoked by GoCD, not something embedded in GoCD. Since GoCD is responsible for the substitution of GoCD Parameters, they cannot be used inside an external script that is invoked by GoCD.

 

I can’t, for example, create a script called “/go/myscript.sh” with “echo #{Param}” as content and then have GoCD execute it with an EXEC Task that calls /go/myscript.sh. Regardless of the value of the Pipeline’s Parameter, this script would simply output “#{Param}”.

 

Similarly, the default EXEC Task does not parse environment variables. So while “echo #{Param}” works as expected, we can’t create an EXEC Task with “echo $EnvVar”. This outputs the literal string “$EnvVar”.

 

I suspect, therefore, that Josh is using a plugin (probably this one) that allows one to embed a script into GoCD so that parameters are expanded by GoCD before the script code is passed to the agent and the agent’s runtime environment is responsible for expanding environment variables referenced in the script. This is not a use case I have been exposed to before, but it would explain the gap between Josh’s understanding and my own.

 

I agree with Josh, though, that this is off-topic for the thread.

 

Back on-topic, I’m glad to hear that Josh got what he needed. As for the interaction between Parameters and Resources, I’m pretty sure I covered it in my earlier post so I won’t repeat myself here.

 

Cheers,

Jason

-Chad

 

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CABr%2BOtrhsksVXzMd-cE%3DsRb2FsTB7ryn%2BBOF76SEWs9xcpn58Q%40mail.gmail.com.