short-circuit eval in templated string works only sometimes?

[Scott Mcdermott]

Hello, why does short circuit only works for the first two cases:

- hosts: localhost
become: false
vars:
ivar: '{{hostvars[inventory_hostname].dne}}'
tasks:
- debug:
msg: "works: {{'foo' or dne}}"
- debug:
msg: "works: {{'bar' or hostvars[inventory_hostname].dne}}"
- debug:
msg: "fails: {{'baz' or ivar}}

The third debug will generate an error:

The task includes an option with an undefined variable.
The error was: {{hostvars[inventory_hostname].dne}}:
'ansible.vars.hostvars.HostVarsVars object' has no attribute 'dne'

Why is it even looking up in hostvars at all? The non-empty
string should evaluate True, so why is it reading the ORed
condition? The same thing happens with "if true else" and
ternary(true, ...) filter. It seems that it's properly
short-circuiting in the first two cases, because the variable
"dne" does not exist, and there is no error. Even more
confusing, the expansion of host vars is the exact same in cases
2 and 3, it's only going through an indirect variable in case 3.

This seems to happen regardless of the source of variables ie
extra vars, vars_files, etc.

Is this a bug, or what am I missing?

[Vladimir Botka]

Why don't you start debugging the variable?

- debug:
var: dne

See "Minimal working example"
https://en.wikipedia.org/wiki/Minimal_working_example


--
Vladimir Botka

[Vladimir Botka]

If the variable *dne* is not defined the playbook

shell> cat playbook.yml
- hosts: localhost

vars:
ivar: "{{ hostvars[inventory_hostname].dne }}"
tasks:
- debug:

var: ivar
- debug:
msg: "{{ 'foo' or dne }}"
- debug:
msg: "{{ 'foo' or ivar }}"

gives

shell> ansible-playbook playbook.yml
...
TASK [debug]
*************************************************
ok: [localhost] => ivar: VARIABLE IS NOT DEFINED!

TASK [debug]
*************************************************
ok: [localhost] => msg: foo

TASK [debug]
*************************************************
fatal: [localhost]: FAILED! => msg: |-
... The error was: 'dne' is undefined

In the second and third debug tasks, the expansion of the string "{{
... }}" is needed first. Then the *or* expression can be evaluated.
There is no problem in the second task because the variable *dne*
is neither expanded nor evaluated (the first element of the *or*
expression is True).

But, the third debug task fails because the expansion of the string
"{{ ... }}" fails before the *or* expression could be evaluated.
You'll need a default value if you want to use this kind of "lazy
evaluation" constructs, e.g.

shell> cat group_vars/all.yml
dne: False

--
Vladimir Botka

[Scott Mcdermott]

On Sun, May 16, 2021 at 12:23 AM Vladimir Botka wrote:
> third debug task fails because the expansion of the string
> "{{ ... }}" fails before the *or* expression could be evaluated.

Sure, but why is it expanded at all? You're right that we can
make my already-minimal test when variable Does Not Exist
(dne) even shorter without needing to use hostvars:

- hosts: localhost
become: false
vars:

ivar: '{{dne}}'
tasks:

- debug:
msg: "{{'foo' or ivar}}"

> You'll need a default value if you want to use this kind of

> "lazy evaluation" constructs ...

I had thought ansible variables were lazily evaluated already.
In docs/docsite/rst/reference_appendices/glossary.rst it says:

Lazy Evaluation
In general, Ansible evaluates any variables in playbook
content at the last possible second, which means
that if you define a data structure that data structure itself
can define variable values within it, and everything "just
works" as you would expect. This also means variable
strings can include other variables inside of those strings.

This seems to imply that things which aren't ever referenced,
will never be expanded. There is even an open issue with a
feature idea to force evaluation to be non-lazy:
https://github.com/ansible/ansible/issues/10374

I cannot use a default because this whole thing is relying
on whether it's defined or not later to take different actions.
the fuller example is:

- name: determine_subnet
when: >
project == project_bake
or hostvars[target].ipnet is defined
set_fact:
subnet: '{{
(project == project_bake) | ternary(
subnet_bake, determined_subnet
)}}'

the problem here is that determined_subnet is defined like this:

determined_subnet: '{{subnets[ip_cidrnet]}}'

and the chain for that is:

ip_host: '{{hostvars[ip_referent].ipnet}}'
ip_referent: '{{host | d(target | d(inventory_hostname))}}'
ip_network: "{{ip_host | ipaddr('network')}}"
ip_cidrnet: '{{ip_network}}/{{ip_prefix}}'

When ipnet (an inventory variable) is not defined,
determined_subnet is still being expanded, even though
project == project_bake, and should therefore never need
to evaluate the second arg to ternary(). For this case, ipnet
is not set.

I've made it work by splitting it into two set_facts with
cascading 'when' clauses, but I don't understand why it's
happening in the first place. The evaluation does not seem
to be lazy, or it would never need to evaluate an unused arg
right? And it's the same behavior using ternary, if/else, or
a disjunction ("or").

--
Scott

[Vladimir Botka]

On Sun, 16 May 2021 01:35:46 -0700
Scott Mcdermott <sc...@smemsh.net> wrote:

> I cannot use a default because this whole thing is relying
> on whether it's defined or not later to take different actions.
> the fuller example is:
>
> - name: determine_subnet
> when: >
> project == project_bake
> or hostvars[target].ipnet is defined
> set_fact:
> subnet: '{{
> (project == project_bake) | ternary(
> subnet_bake, determined_subnet
> )}}'

What value of *subnet* do you want to set when '(project !=
project_bake) and (determined_subnet is undefined)' ?

--
Vladimir Botka

[Vladimir Botka]

When the corresponding alternatives shall always be defined the
defaults can't hurt, e.g.

- set_fact:
subnet: '{{ (project == project_bake)|
ternary(subnet_bake|default(None),
determined_subnet|default(None) }}'

In this case test sanity first

- fail:
msg: Variable undefined
when: ((project == project_bake) and (subnet_bake is undefined)) or
((project != project_bake) and (determined_subnet is
undefined))

--
Vladimir Botka

[Scott Mcdermott]

On Sun, May 16, 2021 at 1:51 PM Vladimir Botka wrote:
> > I cannot use a default because this whole thing is relying
> > on whether it's defined or not later to take different actions.
> > the fuller example is:
> >
> > - name: determine_subnet
> > when: >
> > project == project_bake
> > or hostvars[target].ipnet is defined
> > set_fact:
> > subnet: '{{
> > (project == project_bake) | ternary(
> > subnet_bake, determined_subnet
> > )}}'
>
> What value of *subnet* do you want to set when '(project !=
> project_bake) and (determined_subnet is undefined)' ?

determined_subnet is always defined, except for project_bake.
project_bake nodes don't have ipnet set (or any inventory vars),
so the subnet url (for cloud api calls) can't be determined. So
it gets hardcoded for that project. Hence, the need for the
condition.

--
Scott

[Scott Mcdermott]

On Sun, May 16, 2021 at 2:26 PM Vladimir Botka wrote:
> > > I cannot use a default because this whole thing is relying
> > > on whether it's defined or not later to take different actions.
> > > the fuller example is:
> > >
> > > - name: determine_subnet
> > > when: >
> > > project == project_bake
> > > or hostvars[target].ipnet is defined
> > > set_fact:
> > > subnet: '{{
> > > (project == project_bake) | ternary(
> > > subnet_bake, determined_subnet
> > > )}}'
> >
> > What value of *subnet* do you want to set when '(project !=
> > project_bake) and (determined_subnet is undefined)' ?
>
> When the corresponding alternatives shall always be defined the
> defaults can't hurt, e.g.
>
> - set_fact:
> subnet: '{{ (project == project_bake)|
> ternary(subnet_bake|default(None),
> determined_subnet|default(None) }}'

Unfortunately that doesn't work, I tried it before. Because it's
so indirect, it needs a default at the level that gets referenced,
not the outer level of the indirection. It gets:

localhost failed | msg: The task includes an option with an
undefined variable. The error was: {{subnets[ip_cidrnet]}}:
{{ip_network}}/{{ip_prefix}}: {{ip_host | ipaddr('network')}}:
{{hostvars[ip_referent].ipnet}}:

'ansible.vars.hostvars.HostVarsVars object' has no attribute

'ipnet'

If only it would not try to dereference the thing, because it's
never actually consulted for a value, as it's the OR with a
true condition. That's the part I still don't get, why it's
happening at all.

Anyways, I have worked around it at this point by splitting
into two separate conditions with "when". I just wanted
to understand why. I still don't understand why, but that
happens sometimes... thanks.

[Martin Krizek]

This is effectively a side-effect of functionality in Ansible's
templating engine that allows nesting variables, like `ivar:
'{{hostvars[inventory_hostname].dne}}'` in your example. The second
operand of `or` is actually resolved somewhere within Jinja templating
machinery in all three cases. The difference is with the third one
where Ansible's extended templating functionality gets `ivar` and
tries to template it *because it is a template* (=this is what allows
nesting variables) which causes the failure.

There are issues reported asking to change this behavior, see
https://github.com/ansible/ansible/issues/58835 and
https://github.com/ansible/ansible/issues/56017 (or an interesting
example that was filed recently in
https://github.com/ansible/ansible/issues/74594).

The problem is that while changing this might result in expected
behavior in this case, it will break other scenarios. However the
above issues are open so feel free to give your input there.

Hopefully this clarifies the matter a bit.

Thanks,
Martin

[Vladimir Botka]

On Tue, 18 May 2021 14:54:07 +0200
Martin Krizek <mkr...@redhat.com> wrote:

> This is effectively a side-effect of functionality in Ansible's

> templating engine that allows nesting variables ...

FWIW, one more of this kind not mentioned yet, I think

- debug:
var: my_dict.a
vars:
my_dict:
a: test value
b: "{{ undefined_var }}"

gives

my_dict.a: VARIABLE IS NOT DEFINED!

--
Vladimir Botka