proposal: inline expressions in strings

[Ted Zlatanov]

I proposed inline math expressions in
https://dev.cfengine.com/issues/2111 but the end result was just the
eval() function. I kept hoping for more, so here's my proposal.

I think four things would greatly improve the usability of today's
CFEngine and can be harmlessly added to master (3.7) without damaging
backwards compatibility:

* the ability to call a function inside a string interpolation,
something like: `$((format("%d", 20.1)))` or `$((canonify("$(x)")))`
(note the double parenthesis, which today will not work).

* the ability to do math expressions inside a string interpolation,
something like `$(= 2 + 3 - $(total))` (note the = sign inside the
parenthesis, which today will not work). This is a shortcut for
`$((eval("2 + 3 - $(total)", math, infix)))` in the proposal above.
It would be nice.

* the ability to format data inside a string interpolation, something
like `$("%S %d", mycontainer, "$(total)")` (note the " sign inside the
parenthesis, which today will not work). This is a shortcut for
`$((format("%S %d", mycontainer, "$(total)")))` in the proposal above.
It would save 2-3 lines to generate a temporary variable for formatted
data.

* the ability to canonify strings inside a string interpolation,
something like `$(~total)` (note the ~ sign inside the parenthesis,
which today will not work). This is a shortcut for
`$((canonify("$(total)")))` in the proposal above. It's needed all the
time when you iterate through an array and want the canonified version
of the current key or value. For an impressive example of why this is
needed, look at the edit_line bundle `set_line_based` in files.cf.

The syntax may need adjusting, but does this seem like a reasonable
idea? I can try to implement it...

Ted

[Dimitrios Apostolou]

Hi Ted,

you know well that I like clear syntax, and while all this is useful,
it appears like perl to me! :-) Personally I would prefer to keep the
syntax crystal clear for anyone to read, i.e. non-idiomatic, and add
functionality via functions. Anything added inside what you mention as
"string interpolation", I'm against.

IMHO, one small step, very relevant to what you mention, that will
have *huge* impact on ease of use, is adding proper support for nested
function calls. Me not being an evaluator expert, am confused as to
what kind of function-call nesting is supported, because sometimes I
have no problem. See the following policy, and tell me straight if you
think I'm doing it wrong. :-)

bundle agent test
{
vars:
### THE FOLLOWING LINE DOES NOT WORK because of the nested function calls
# "result" string => nth(splitstring("/path/to/file", "/", 4), 1);

### THIS WORKS
"step1" slist => splitstring("/path/to/file", "/", 4);
"result" string => nth(step1, 1);

reports:
"$(result)";
}


Regards,
Dimitris

> --
> You received this message because you are subscribed to the Google Groups "dev-cfengine" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dev-cfengine...@googlegroups.com.
> To post to this group, send email to dev-cf...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/dev-cfengine/877fy6q1yj.fsf%40lifelogs.com.
> For more options, visit https://groups.google.com/d/optout.

[Nick Anderson]

Canonifying a string I would probably find immediately useful. I build canonified maps all the time and it's annoying. I am on with the syntax as well.

Math would he neat.

I find the function and formatting syntax hard to read, but don't have any real objections.

What are the pitfalls of doing it?


Sent from my mobile device.

[Ted Zlatanov]

On Thu, 4 Dec 2014 20:49:38 -0600 Nick Anderson <nick.a...@cfengine.com> wrote:

NA> Canonifying a string I would probably find immediately useful. I build canonified maps all the time and it's annoying. I am on with the syntax as well.
NA> Math would he neat.

NA> I find the function and formatting syntax hard to read, but don't have any real objections.

I'm open to suggestions there, but I don't think we want another sigil
besides `$()` to signal interpolation, so we're limited for the sake of
backwards compatibility.

NA> What are the pitfalls of doing it?

I guess the possibility of function calls anywhere you have strings, and
more syntax.

Ted

[Ted Zlatanov]

On Fri, 5 Dec 2014 03:45:22 +0100 Dimitrios Apostolou <ji...@cfengine.com> wrote:

DA> you know well that I like clear syntax, and while all this is useful,
DA> it appears like perl to me! :-)

I'd love to find a simpler syntax but backwards compatibility is an
issue. That's why I posted, to see what people think.

DA> Personally I would prefer to keep the syntax crystal clear for
DA> anyone to read, i.e. non-idiomatic, and add functionality via
DA> functions. Anything added inside what you mention as "string
DA> interpolation", I'm against.

OK, but we constantly run into things like this (from `set_line_based`):

# Be careful if the index string contains funny chars
"ci[$(i)]" string => canonify("$(i)");

# Escape the value (had a problem with special characters and regex's)
"ev[$(i)]" string => escape("$($(v)[$(i)])");

# Do we have more than one line commented out?
"comment_matches_$(ci[$(i)])"
int => countlinesmatching("^$(cp)($(i)$(bp).*|$(i))$",
$(edit.filename));

That's not fun, and functions don't help much to clear this up.

Another pain-pattern is the constant debugging pain of:

vars:
"container" data => ...something...
"container_s" string => format("%S", container);
reports:
"container has data $(container_s)";

So why did I propose `$((...))` and the rest of that syntax? Because I
am trying to keep backwards compatibility. Let's look at the
alternatives:

* Perl: "$x->[1]" interpolates to the second value in an array.
"@{[ function(data) ]}" calls a function inline. Neither will work because
of backwards compatibility (someone may have used those in policy and
they were OK before).

* Ruby: "#{varname}" interpolates to a variable, again won't work
because of backwards compatibility.

* Python has a nice way to do sprintf, `"%s %d" % "string here", 2`. Again,
backwards compatibility doesn't allow it.

* shell: `$(command x y z)` gives us the output of that command with
those arguments. Closest to what I'm suggesting.

* Lisp: uses sprintf, very little string interpolation.

* tcl: you don't wanna know.

* OOP methods on strings, e.g. `$(varname.canonify)` (heavily used in
Ruby, Java, etc.). It could work for function calls but the dot is
already used, so it would look kind of weird: `$(varname->canonify)`
and `$(varname->format(1, 2, 3))` where $(varname) is "%d %f %s".
This is probably my second favorite, but generally I don't see OOP
mixing well with CFEngine's syntax.

I hope that explains it better.

DA> IMHO, one small step, very relevant to what you mention, that will
DA> have *huge* impact on ease of use, is adding proper support for nested
DA> function calls. Me not being an evaluator expert, am confused as to
DA> what kind of function-call nesting is supported, because sometimes I
DA> have no problem. See the following policy, and tell me straight if you
DA> think I'm doing it wrong. :-)

DA> bundle agent test
DA> {
DA> vars:
DA> ### THE FOLLOWING LINE DOES NOT WORK because of the nested function calls
DA> # "result" string => nth(splitstring("/path/to/file", "/", 4), 1);

DA> ### THIS WORKS
DA> "step1" slist => splitstring("/path/to/file", "/", 4);
DA> "result" string => nth(step1, 1);

DA> reports:
DA> "$(result)";
DA> }

Generally you can nest scalars (expression, int, real, and string).
That's because internally they are all strings (although the parser
sometimes complains about mixing them). That's the area of my proposal,
so the problem you describe is not strongly related.

iranges and rranges are not returned by any function, I think. They are
currently underused.

Lists (slist, ilist, rlist) and data containers can't be nested
currently. It would be nice to have that ability, I agree.

Ted

[Bas van der Vlies]

Just my 2 cents. I do not like the $(()) or other shortcuts. For me it is hard to read. I rather got voor the OOP method.

> --
> You received this message because you are subscribed to the Google Groups "dev-cfengine" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dev-cfengine...@googlegroups.com.
> To post to this group, send email to dev-cf...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/dev-cfengine/87y4qmohda.fsf%40lifelogs.com.

> For more options, visit https://groups.google.com/d/optout.

---
Bas van der Vlies
| Operations, Support & Development | SURFsara | Science Park 140 | 1098 XG Amsterdam
| T +31 (0) 20 800 1300 | bas.van...@surfsara.nl | www.surfsara.nl |

[Neil Watson]

I like these changes and here is why. At first I believe that cf3's
syntax was written with the belief that it would be a simple
declarative language. The users have evolved beyond this simplistic
view. We are performing complex variable operations and the language
needs more features to help us. Policy is often longer and more
convoluted that it needs to be. Ted's inline canonification is a good
example of how this can be improved.

There is some opposition to the proposal, knocking Perl. Perl sigils allow
one to express meaning with less typing. In other languages is the
variable 'foo' a string, a list, an array, or a reference? In Perl this
is usually very clear because of the prefix sigil. Indeed Perl was
written by a linguist who wanted his language to have multiple ways to
express something, like human languages. Sorry, that was a little far
off topic.

We should stay away from anything resembling OO syntax. CFEngine's use
of class would only get more confusing if the syntax resembled OO.

--
Neil H Watson
Sr. Partner, Architecture and Infrastructure
CFEngine reporting: https://github.com/evolvethinking/delta_reporting
CFEngine policy: https://github.com/evolvethinking/evolve_cfengine_freelib
CFEngine and vim: https://github.com/neilhwatson/vim_cf3
CFEngine support: http://evolvethinking.com

[Ted Zlatanov]

On Fri, 5 Dec 2014 10:19:13 +0000 Bas van der Vlies <bas.van...@surfsara.nl> wrote:

BvdV> Just my 2 cents. I do not like the $(()) or other shortcuts. For me it is hard to read. I rather got voor the OOP method.

To be clear, the *ability* to do the four things I mentioned is much
more important than the specific syntax. Here they are, using OOP-style:

* the ability to call a function inside a string interpolation:
`$(x->canonify)` or `$("%d"->format(20.1))`. This is just calling a
partial application of a function.

* the ability to do math expressions inside a string interpolation

shoudl remain `$(= 2 + 3 - $(total))`. It has to be short and simple.

* the ability to format data inside a string interpolation: merged into
the above using OOP style `$("%d"->format(20.1))`.

* the ability to canonify strings inside a string interpolation:
`$(x->canonify)`. I still think `$(~x)` is a nice shortcut but the
explicit function call is more readable.

The above could be done with dots `.` instead of arrows, but I'd expect
it to be a bit trickier because the parser won't know if `x.canonify`
means "canonify($(x))" or "the variable 'canonify' in bundle 'x'".

I think the developer team and the language designers (especially Mark
Burgess, of course) should decide on the syntax, but IMO there is a
clear need in the policy language to do these things without going
through intermediate steps and variables. I hope we all agree on that.

Ted

[Bas van der Vlies]

Thanks for the examples. I agree with your goals. I wanted just to express my mixed feelings with perlish syntax. I already know to much of $!~ constructs languages. I think it makes the learning curve for new users higher and for readability it is not a step forward.

[Eystein Måløy Stenberg]

I tend to agree with this (in general).

It is better to write a few more lines in a clear well-defined language
than save a few characters while trying to deduce what this 5-character
expression actually means. Lowering the learning curve correlates to
having less to learn.

There are cases where perl is not the best thing to copy. :)

--

Eystein

[Ted Zlatanov]

On Fri, 05 Dec 2014 09:30:15 -0800 Eystein Måløy Stenberg <eystein.mal...@cfengine.com> wrote:

EMS> I tend to agree with this (in general).
EMS> It is better to write a few more lines in a clear well-defined language
EMS> than save a few characters while trying to deduce what this 5-character
EMS> expression actually means.

Can you comment specifically on whether the 4 abilities I proposed seem
reasonable, and in addition whether either syntax (bash-style or
OOP-style) looks OK? I can't tell what your position is, sorry.

That would give me and other contributors an idea whether it's worth our
time to produce prototypes of this functionality.

EMS> Lowering the learning curve correlates to having less to learn.

Sure, but the current syntax makes it really hard to do some things.
It's simple and consistent, but feels like doing 2 + 2 + 2 + 2 because
you are not allowed to say 2 * 4...

Ted

[Dimitrios Apostolou]

Hi Ted,

On Fri, Dec 5, 2014 at 5:22 AM, Ted Zlatanov <t...@lifelogs.com> wrote:
>
> # Be careful if the index string contains funny chars
> "ci[$(i)]" string => canonify("$(i)");
>
> # Escape the value (had a problem with special characters and regex's)
> "ev[$(i)]" string => escape("$($(v)[$(i)])");
>
> # Do we have more than one line commented out?
> "comment_matches_$(ci[$(i)])"
> int => countlinesmatching("^$(cp)($(i)$(bp).*|$(i))$",
> $(edit.filename));
>
> That's not fun, and functions don't help much to clear this up.

I agree that's not fun. How would your syntax change this? What is the
end result of all this, i.e. what is the code trying to achieve here?
I'll pretend I never saw this, so if you want to describe what the you
want as output, I'll give you my opinion on clear syntax to achieve
this. :-p

>
> Another pain-pattern is the constant debugging pain of:
>
> vars:
> "container" data => ...something...
> "container_s" string => format("%S", container);
> reports:
> "container has data $(container_s)";

This is an isolated point that may be resolved with minimal change.
But why doesn't the example above work like an slist? I.e. Iterating
over the container and printing "container has data container_entry"
for every entry in the container?

I believe I'm missing the real purpose that you have in mind. Because
the simple examples you gave have simple solutions. Some questions on
your initial use cases:

> the ability to do math expressions inside a string interpolation, something like `$(= 2 + 3 - $(total))`

Why not just call eval()? If you don't want intermediate variables,
call concat() as well. e.g.

concat("The result is ",
eval("2+4-$(total)"),
", anything more?");

> the ability to format data inside a string interpolation

Again, why not just call format() and assign to a string? Same
question for the canonify() syntax. You can always concat() as many
canonifications as you want.


Finally, in the languages you mentioned I know python's % operator is
being discouraged in favour of a function (actually method) call,
format().


Thanks,
Dimitris

[Ted Zlatanov]

On Sun, 7 Dec 2014 06:36:30 +0100 Dimitrios Apostolou <ji...@cfengine.com> wrote:

DA> On Fri, Dec 5, 2014 at 5:22 AM, Ted Zlatanov <t...@lifelogs.com> wrote:
>>
>> # Be careful if the index string contains funny chars
>> "ci[$(i)]" string => canonify("$(i)");
>>
>> # Escape the value (had a problem with special characters and regex's)
>> "ev[$(i)]" string => escape("$($(v)[$(i)])");
>>
>> # Do we have more than one line commented out?
>> "comment_matches_$(ci[$(i)])"
>> int => countlinesmatching("^$(cp)($(i)$(bp).*|$(i))$",
>> $(edit.filename));
>>
>> That's not fun, and functions don't help much to clear this up.

DA> I agree that's not fun. How would your syntax change this? What is the
DA> end result of all this, i.e. what is the code trying to achieve here?

We would avoid creating temporary arrays for canonified or escaped
strings (inline escaped could have something like $(\i) as the syntax).

This is extremely common, and the end result would be that the `ev` and
`ci` arrays in the policy would be unnecessary. So for instance the
promiser in the last line I showed would become

"comment_matches_$(~i)"

which I think is *much* more readable. That bundle has many
instances of `ci` that would be eliminated. I am pretty sure Diego's
book would become shorter by 5-10 pages if this was implemented :)

I experimentally looked in masterfiles to check. lib/3.6 has:

* bundles.cf: 1 variable "chost", 4 uses
* files.cf: 10 vars, 41 uses

update/update_processes.cf has the `cprocess` variable, used a dozen
times. The reader has to mentally keep track of two variables.

Another issue is that "classic" arrays don't take arbitrary strings as
keys, so `$(~i)` to canonify the variable i is more powerful. Finally,
"classic" arrays in CFEngine create actual variables for each entry,
which means you have to be aware of normal ordering and the possibility
of promise evaluation out of order.

I should point out that much of this proposal comes from multiple slides
in the CFEngine training courses I've taught, the discussions I've had
with perplexed students, and with "power" CFEngine users like Diego
Zamboni, Nick Anderson, and others. I just put it in one place.

>> Another pain-pattern is the constant debugging pain of:
>>
>> vars:
>> "container" data => ...something...
>> "container_s" string => format("%S", container);
>> reports:
>> "container has data $(container_s)";

DA> This is an isolated point that may be resolved with minimal change.

What change would you suggest?

DA> But why doesn't the example above work like an slist? I.e. Iterating
DA> over the container and printing "container has data container_entry"
DA> for every entry in the container?

When you're debugging, you'd see the data in one line instead of
multiples. This makes a big difference for readability, especially since
data containers can have nested complex structure which doesn't fit the
one-line-per-entry pattern. I think also JSON null items won't be
printed in normal iteration, but that's minor.

(BTW, did you know format("%S") can also be used on a slist and shows
cf_null correctly? It's also very useful for debugging vs. showing each
entry on a new line.)

>> the ability to do math expressions inside a string interpolation, something like `$(= 2 + 3 - $(total))`

DA> Why not just call eval()? If you don't want intermediate variables,
DA> call concat() as well. e.g.

DA> concat("The result is ",
DA> eval("2+4-$(total)"),
DA> ", anything more?");

Actually you would do:

concat("The result is ", eval("2+4-$(total)", math, infix), ", anything more?");

DA> Again, why not just call format() and assign to a string? Same
DA> question for the canonify() syntax. You can always concat() as many
DA> canonifications as you want.

Do you think that's better that `$(= 2+4-$(total))` or `$(~i)`? In many
cases you can't use function calls anyhow, or it requires temporary
variables. For instance when specifying class expressions. I feel the
syntax I proposed is clearly more convenient.

Ted

[Nick Anderson]

It is common that I want to use the key of an array in canonified form along with the value of that entry.

Sometimes that array is generated and I have troubles with the characters I want to use for keys. This gets really annoying to deal with.

Want a canonified list?
Generate a list, build a new array, generate a list:
https://gist.githubusercontent.com/nickanderson/5271023/raw/b86b207ec0c8b89513a5882ce08330a97dedd513/example_canonify_list.cf

Sent from my mobile device.

-----Original Message-----
From: Ted Zlatanov <t...@lifelogs.com>
To: dev-cf...@googlegroups.com

--
You received this message because you are subscribed to the Google Groups "dev-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-cfengine...@googlegroups.com.
To post to this group, send email to dev-cf...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dev-cfengine/87mw6zmyvw.fsf%40lifelogs.com.

[Ted Zlatanov]

On Sun, 7 Dec 2014 08:33:13 -0600 Nick Anderson <nick.a...@cfengine.com> wrote:

NA> It is common that I want to use the key of an array in canonified form along with the value of that entry.
NA> Sometimes that array is generated and I have troubles with the characters I want to use for keys. This gets really annoying to deal with.

NA> Want a canonified list?
NA> Generate a list, build a new array, generate a list:
NA> https://gist.githubusercontent.com/nickanderson/5271023/raw/b86b207ec0c8b89513a5882ce08330a97dedd513/example_canonify_list.cf

OK, so I implemented the following in the attached patch. Changed code:
roughly 70 lines. It really was easy to do the basics and this is
already useful for the math eval.

* canonification of scalars with $(~varname)

* escaping of scalars with $(\varname)

* math eval with $(= 2 + 2) (prints %lf)

* math integer eval with $(+ 2 + 2) (prints %ld)

I tried to do the canonification+escaping for lists but that code is
beyond me. I really doubt I can finish it on my own, so I hope the
developers help me pursue it. It would be great if there was just one
function for inner expansion, instead of two (ExpandScalar and the
really complicated ExpandAndMapIteratorsFromScalar). The code
duplication is painful, but the complexity of the latter function just
kills me.

What remains:

* formatting of variables and data like $("%S", mycontainer)

* calling functions inside an expansion

* settling on the specific syntax for all of this fun stuff

Code sample:

bundle agent test()
{
vars:
# build a list from the first index on the config array
"site_names" slist => { "a", "b", "c" };
"words" string => "a b c + d";

reports:
"canonified site: $(~site_names)";
"canonified bundle name: $(~this.bundle)";
"canonified words: $(~words)";
"escaped words: $(\words)";
"math expression: $(= 2 + 3 + pi)";
"math int expression: $(+ 2 + 3 + pi)";
"bad math expression: $(= x + y)";

}

Output:

R: canonified site: $(~site_names)
R: canonified bundle name: test
R: canonified words: a_b_c___d
R: escaped words: a\ b\ c\ \+\ d
R: math expression: 8.141593
R: math int expression: 8
info math error in $(= x + y): expression could not be parsed
R: bad math expression: $(= x + y)

[Dimitrios Apostolou]

On Sun, 7 Dec 2014, Ted Zlatanov wrote:

> On Sun, 7 Dec 2014 06:36:30 +0100 Dimitrios Apostolou <ji...@cfengine.com> wrote:
>
> DA> On Fri, Dec 5, 2014 at 5:22 AM, Ted Zlatanov <t...@lifelogs.com> wrote:
>>>
>>> # Be careful if the index string contains funny chars
>>> "ci[$(i)]" string => canonify("$(i)");
>>>
>>> # Escape the value (had a problem with special characters and regex's)
>>> "ev[$(i)]" string => escape("$($(v)[$(i)])");
>>>
>>> # Do we have more than one line commented out?
>>> "comment_matches_$(ci[$(i)])"
>>> int => countlinesmatching("^$(cp)($(i)$(bp).*|$(i))$",
>>> $(edit.filename));
>>>
>>> That's not fun, and functions don't help much to clear this up.
>
> DA> I agree that's not fun. How would your syntax change this? What is the
> DA> end result of all this, i.e. what is the code trying to achieve here?
>
> We would avoid creating temporary arrays for canonified or escaped
> strings (inline escaped could have something like $(\i) as the syntax).

I think I get it now. So wouldn't it be enough to be able to call
string functions in the promiser string?

concat("comment_matches_", canonify("$(i)")) int => countlinesmatching("...$(i)...", $(edit.filename))

Certainly not as terse as what you suggest, but wouldn't it solve the
problem of intermediate assignments?

>
> This is extremely common, and the end result would be that the `ev` and
> `ci` arrays in the policy would be unnecessary. So for instance the
> promiser in the last line I showed would become
>
> "comment_matches_$(~i)"
>
> which I think is *much* more readable. That bundle has many

> [...]

My opinion: No, because the tilde operator doing canonification is not
common among languages, I would have to learn it first. The more special
characters a language has, the less readable it is to outsiders.

>
> Another issue is that "classic" arrays don't take arbitrary strings as
> keys, so `$(~i)` to canonify the variable i is more powerful.

> [...]

Can you please elaborate on that? Do you mean that they reject
non-canonified characters as index? What if the arrays were fixed, and
they were proper dictionaries? IMHO this is a very reasonable feature
request. I'm not sure it's possible though, mostly because of ordering
issues...

>>> Another pain-pattern is the constant debugging pain of:
>>>
>>> vars:
>>> "container" data => ...something...
>>> "container_s" string => format("%S", container);
>>> reports:
>>> "container has data $(container_s)";
>
> DA> This is an isolated point that may be resolved with minimal change.
>
> What change would you suggest?

Maybe using the existing syntax of @(variable) would be an intuitive way
for printing all of variable (container or slist) in one line, without
iterating.

If you also need function calls for formating, canonifying etc then this
wouldn't be enough, you would need a separate variable.

> When you're debugging, you'd see the data in one line instead of
> multiples. This makes a big difference for readability, especially since
> data containers can have nested complex structure which doesn't fit the
> one-line-per-entry pattern. I think also JSON null items won't be
> printed in normal iteration, but that's minor.
>
> (BTW, did you know format("%S") can also be used on a slist and shows
> cf_null correctly? It's also very useful for debugging vs. showing each
> entry on a new line.)

Good info, thanks. Not doing iteration but accumulating the variable IMHO
would be better as part of the language instead of function parameters.

>>> the ability to do math expressions inside a string interpolation, something like `$(= 2 + 3 - $(total))`
>
> DA> Why not just call eval()? If you don't want intermediate variables,
> DA> call concat() as well. e.g.
>
> DA> concat("The result is ",
> DA> eval("2+4-$(total)"),
> DA> ", anything more?");
>
> Actually you would do:
>
> concat("The result is ", eval("2+4-$(total)", math, infix), ", anything more?");
>
> DA> Again, why not just call format() and assign to a string? Same
> DA> question for the canonify() syntax. You can always concat() as many
> DA> canonifications as you want.
>
> Do you think that's better that `$(= 2+4-$(total))` or `$(~i)`?

> [...]

Well if you factor out the ugly syntax of eval() (which can easily be
fixed by omitting the "math" and "infix" arguments and making them
defaults) then yes, I am clearly in favor of function calling.

But indeed function calling is restricting, since these cannot happen in
the promiser string. Maybe this is something worth to be fixed...


Dimitris


P.S. BTW even if I disagree with many of your points, I appreciate a lot
that you are bringing such discussions to this list. It feels good to
discuss development in the open. ;-)

[Mark at work]

Arrays used to accept arbitrary string keys, but I vaguely remember a discussion about changing that a year or two ago. I was against a change, but I don't know what the outcome was.

Having generic string keys has several important use cases.

On 8 Dec 2014, at 10:45, Dimitrios Apostolou <ji...@cfengine.com> wrote:

On Sun, 7 Dec 2014, Ted Zlatanov wrote:

 On Sun, 7 Dec 2014 06:36:30 +0100 Dimitrios Apostolou 

 DA> On Fri, Dec 5, 2014 at 5:22 AM, Ted Zlatanov 

[Ted Zlatanov]

On Mon, 8 Dec 2014 10:45:03 +0100 (CET) Dimitrios Apostolou <ji...@cfengine.com> wrote:

DA> I think I get it now. So wouldn't it be enough to be able to call
DA> string functions in the promiser string?

DA> concat("comment_matches_", canonify("$(i)")) int => countlinesmatching("...$(i)...", $(edit.filename))

DA> Certainly not as terse as what you suggest, but wouldn't it solve the
DA> problem of intermediate assignments?

I think that would be useful, yes. But, knowing the parser and evaluator
a little bit, I think it would actually be harder to implement than the
syntax I proposed.

IMO the language will get terribly verbose if you require function calls
for everything (Lisp suffers this problem). Almost every language,
including the super-verbose Java, has a way to concatenate strings
without calling a function, so you could say

"comment_matches_" + canonify($(i))

in the promiser. (Even C sort of does this, with the "x" "y" preprocessor
syntax that comes out to "xy". Handy in macro hacks.)

So with a first-class string concatenation operator everywhere, not just
in promisers, I think your alternative is very good and answers my need
for the ability to call functions anywhere you can use a string. My only
concern, as I said, is that the implementation will be hard.

>> "comment_matches_$(~i)"
>>
>> which I think is *much* more readable.

DA> My opinion: No, because the tilde operator doing canonification is not
DA> common among languages, I would have to learn it first. The more
DA> special characters a language has, the less readable it is to
DA> outsiders.

I understand the concern. Still, canonification is already special to
CFEngine. Every DSL and programming language (I think CFEngine sits in
the middle) has domain-specific shortcuts. So I think it makes sense to
give CFEngine users some they will use *all* the time. Guaranteed.
Those who want crystal clarity can use canonify(). CFEngine's own
masterfiles could use the syntax in dozens of places to reduce code and
complexity.

The same argument goes for $(\i) to escape the variable i.

I implemented both of these with a little bit of work for scalars and
really think they would make life better for CFEngine users.

>> Another issue is that "classic" arrays don't take arbitrary strings as
>> keys, so `$(~i)` to canonify the variable i is more powerful.
>> [...]

DA> Can you please elaborate on that? Do you mean that they reject
DA> non-canonified characters as index? What if the arrays were fixed, and
DA> they were proper dictionaries? IMHO this is a very reasonable feature
DA> request. I'm not sure it's possible though, mostly because of ordering
DA> issues...

MB> Arrays used to accept arbitrary string keys, but I vaguely remember a
MB> discussion about changing that a year or two ago. I was against a
MB> change, but I don't know what the outcome was.

MB> Having generic string keys has several important use cases.

Fortunately, data containers support generic string keys, which solves
many use cases. But the need here is for dynamically created arrays,
maparray()-style, with the keys and values transformed on the fly.
That's not easy for data containers because they are read-only so you
have to iterate. You can sort of hack it with mergedata() but it
doesn't work too well.

I wrote a PR for a mapjson() function that could do what maparray() did
but for data containers. A more general solution would be great, and if
the string concatenation syntax is implemented, it would be pretty easy
to write a data_maparray() or mapjson() function that looks like

"result" data => data_maparray("the key " + $(this.k), "the value " + $(this.v), myarray);

DA> Maybe using the existing syntax of @(variable) would be an intuitive
DA> way for printing all of variable (container or slist) in one line,
DA> without iterating.

Oh, you mean like:

reports:
"my container is @(container)";

That would work, but it may break backwards compatibility because the
string "@(mylist)" is used very often when you call bundles or
functions. I'm in favor of it, if it's possible.

DA> If you also need function calls for formating, canonifying etc then
DA> this wouldn't be enough, you would need a separate variable.

Yes, typically when you're debugging you want to see the actual contents
of the variable, so you don't need to transform them further. I think
that's the most important use case.

>> Do you think that's better that `$(= 2+4-$(total))` or `$(~i)`?

DA> Well if you factor out the ugly syntax of eval() (which can easily be
DA> fixed by omitting the "math" and "infix" arguments and making them
DA> defaults)

That's not a bad idea :)

DA> then yes, I am clearly in favor of function calling.

I think we should poll experienced CFEngine policy writers (maybe on
help-cfengine). I feel these very specific shortcuts would cut out a lot
of code, but I may be wrong about how much they are needed.

Thanks
Ted

[Ted Zlatanov]

On Mon, 08 Dec 2014 05:41:15 -0500 Ted Zlatanov <t...@lifelogs.com> wrote:

TZ> I think we should poll experienced CFEngine policy writers (maybe on
TZ> help-cfengine). I feel these very specific shortcuts would cut out a lot
TZ> of code, but I may be wrong about how much they are needed.

I asked there, we'll see what users say.

Mark Burgess contacted me directly and said he likes all the shortcut
inline proposals ($(~var), $(\var), and $(= 2 + 2)). That's awesome.

Ted

[Ted Zlatanov]

On Mon, 08 Dec 2014 05:41:15 -0500 Ted Zlatanov <t...@lifelogs.com> wrote:

TZ> So with a first-class string concatenation operator everywhere, not just
TZ> in promisers, I think your alternative is very good and answers my need
TZ> for the ability to call functions anywhere you can use a string. My only
TZ> concern, as I said, is that the implementation will be hard.

Thinking about this, I remembered that calling functions is much more
expensive than expanding variables, and there are some functions that
are cached, and so on. So I think at least for performance and code
simplicity, the inline shortcuts may be a good thing even if general
string concatenation with function calls is available everywhere.

Ted

[Kristian Amlie]

On 08/12/14 11:41, Ted Zlatanov wrote:
> On Mon, 8 Dec 2014 10:45:03 +0100 (CET) Dimitrios Apostolou <ji...@cfengine.com> wrote:
>
> DA> I think I get it now. So wouldn't it be enough to be able to call
> DA> string functions in the promiser string?
>
> DA> concat("comment_matches_", canonify("$(i)")) int => countlinesmatching("...$(i)...", $(edit.filename))
>
> DA> Certainly not as terse as what you suggest, but wouldn't it solve the
> DA> problem of intermediate assignments?
>
> I think that would be useful, yes. But, knowing the parser and evaluator
> a little bit, I think it would actually be harder to implement than the
> syntax I proposed.
>
> IMO the language will get terribly verbose if you require function calls
> for everything (Lisp suffers this problem). Almost every language,
> including the super-verbose Java, has a way to concatenate strings
> without calling a function, so you could say
>
> "comment_matches_" + canonify($(i))
>
> in the promiser. (Even C sort of does this, with the "x" "y" preprocessor
> syntax that comes out to "xy". Handy in macro hacks.)
>
> So with a first-class string concatenation operator everywhere, not just
> in promisers, I think your alternative is very good and answers my need
> for the ability to call functions anywhere you can use a string. My only
> concern, as I said, is that the implementation will be hard.

Even if the implementation is hard, I think this is the route to go. An
operator for doing string concatenation is much more common in
languages, and doesn't hinder readability for anyone, including
beginners. And it solves many related annoyances regarding concatenating
strings without several variables or overuse of concat().

--
Kristian

[Ted Zlatanov]

On Wed, 10 Dec 2014 09:18:12 +0100 Kristian Amlie <kristia...@cfengine.com> wrote:

KA> On 08/12/14 11:41, Ted Zlatanov wrote:
>> So with a first-class string concatenation operator everywhere, not just
>> in promisers, I think your alternative is very good and answers my need
>> for the ability to call functions anywhere you can use a string. My only
>> concern, as I said, is that the implementation will be hard.

KA> Even if the implementation is hard, I think this is the route to go. An
KA> operator for doing string concatenation is much more common in
KA> languages, and doesn't hinder readability for anyone, including
KA> beginners. And it solves many related annoyances regarding concatenating
KA> strings without several variables or overuse of concat().

OK. Do you want me to open a ticket? It does break backwards
compatibility, but IMO it will improve the language significantly.

The poll about the shortcuts on help-cfengine was mixed, with some
positive and some "eh" reactions. Most were concerned about readability.
Since Mark liked them and there was no strong sentiment against them,
I'd like to pursue them further. I also noted some performance benefits
with the shortcuts, because functions don't have to be called.

If I extend my patch so it works in all strings for the proposed
shortcuts:

1) $(~var) to canonify a variable (I can use anything but ~)

2) $(\var) to escape a variable

3) $((3+2)) to do math inline bash-style, formatting as integer if the
result is close enough to an integer (earlier I had $(= 2+2) and $(+
2+2) which is uglier and less standard

...is there a good chance it will get accepted? As I mentioned the
iteration expansion logic is nasty, so I don't want to tackle this if
it's going to sit in the PR queue for years. But if it has a good
chance, I'll try the code and write docs and acceptance tests as usual.

I'm asking here instead of a Redmine ticket because I want to preserve
the context of this thread.

Thanks
Ted

[Kristian Amlie]

On 10/12/14 12:15, Ted Zlatanov wrote:
> On Wed, 10 Dec 2014 09:18:12 +0100 Kristian Amlie <kristia...@cfengine.com> wrote:
>
> KA> On 08/12/14 11:41, Ted Zlatanov wrote:
>>> So with a first-class string concatenation operator everywhere, not just
>>> in promisers, I think your alternative is very good and answers my need
>>> for the ability to call functions anywhere you can use a string. My only
>>> concern, as I said, is that the implementation will be hard.
>
> KA> Even if the implementation is hard, I think this is the route to go. An
> KA> operator for doing string concatenation is much more common in
> KA> languages, and doesn't hinder readability for anyone, including
> KA> beginners. And it solves many related annoyances regarding concatenating
> KA> strings without several variables or overuse of concat().
>
> OK. Do you want me to open a ticket? It does break backwards
> compatibility, but IMO it will improve the language significantly.
>
> The poll about the shortcuts on help-cfengine was mixed, with some
> positive and some "eh" reactions. Most were concerned about readability.
> Since Mark liked them and there was no strong sentiment against them,
> I'd like to pursue them further. I also noted some performance benefits
> with the shortcuts, because functions don't have to be called.
>
> If I extend my patch so it works in all strings for the proposed
> shortcuts:
>
> 1) $(~var) to canonify a variable (I can use anything but ~)
>
> 2) $(\var) to escape a variable

I'm rather against these two. The meaning is not at all clear to
beginners, and it's one of those things that are pretty hard to search
for as well. I really don't see that "escape" and "canonify", both short
words, are such a big problem to type?

> 3) $((3+2)) to do math inline bash-style, formatting as integer if the
> result is close enough to an integer (earlier I had $(= 2+2) and $(+
> 2+2) which is uglier and less standard

I'm more unsure of this one. It's definitely more recognizable and hard
to mistake for anything else. But we need to tread carefully here. If we
also want to enable string concatenation with the '+' symbol, we may
have to be prepared for operator precedence trouble if you are also able
to do maths.

I do like the $((2+2)) syntax much better than $(= 2+2) though!

> ...is there a good chance it will get accepted? As I mentioned the
> iteration expansion logic is nasty, so I don't want to tackle this if
> it's going to sit in the PR queue for years. But if it has a good
> chance, I'll try the code and write docs and acceptance tests as usual.

I'm not impossible to budge on this subject, but given that two of the
main developers (counting Dimitrios and myself) are skeptical towards
this change, I don't think it will be accepted in its current form.

I don't consider this subject closed though. For me the most
questionable part is the ~ and \ operators. Perhaps we should focus
first on the maths syntax and especially the string concatenation, since
I feel there is more consensus there.

> I'm asking here instead of a Redmine ticket because I want to preserve
> the context of this thread.

I'd still use the mailing list at least until we have rudimentary
agreement. Once we start looking at implementation details we can create
a task.

--
Kristian

[Nick Anderson]

I would argue the canonify shortcut is the most useful.

Sent from my mobile device.

-----Original Message-----
From: Kristian Amlie <kristia...@cfengine.com>
To: dev-cf...@googlegroups.com
Sent: Wed, 10 Dec 2014 7:51 AM
Subject: Re: proposal: inline expressions in strings

--
You received this message because you are subscribed to the Google Groups "dev-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-cfengine...@googlegroups.com.
To post to this group, send email to dev-cf...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dev-cfengine/548841B6.7000104%40cfengine.com.

[Ted Zlatanov]

On Wed, 10 Dec 2014 13:51:02 +0100 Kristian Amlie <kristia...@cfengine.com> wrote:

KA> On 10/12/14 12:15, Ted Zlatanov wrote:
>> 1) $(~var) to canonify a variable (I can use anything but ~)
>>
>> 2) $(\var) to escape a variable

KA> I'm rather against these two. The meaning is not at all clear to
KA> beginners, and it's one of those things that are pretty hard to search
KA> for as well.

Right, that's understood. But it's not intended for beginners (although
you're right that they will need it pretty quickly with today's core).

KA> I really don't see that "escape" and "canonify", both short words,
KA> are such a big problem to type?

Until general string concatenation is available, in 3.10 or 4.x or
whenever, it's a big hassle to create intermediate variables (and
because of CFEngine's evaluation model, it's much more complex to do it
right). It's also nice to provide syntactic sugar for the most commonly
used things, and it maintains backwards compatibility. So it's
definitely not about saving keystrokes.

The shortcuts could be $(:escape:var) and $(:canonify:var) just as well.
Or $(escape(var)) and $(canonify(var)). Would that make them easier to
accept?

I'm glad you're unsure of the math shortcut. One step at a time :)

If we follow the shortcut pattern above, the math shortcut could be
$(:eval:2+2) or $(eval(2+2)).

KA> I'm not impossible to budge on this subject, but given that two of the
KA> main developers (counting Dimitrios and myself) are skeptical towards
KA> this change, I don't think it will be accepted in its current form.

KA> I don't consider this subject closed though. For me the most
KA> questionable part is the ~ and \ operators. Perhaps we should focus
KA> first on the maths syntax and especially the string concatenation, since
KA> I feel there is more consensus there.

I can propose the math syntax in a separate patch, but the current code
in expand.c is pretty nasty for iterated expansions. I can make it work
for scalar expansions quickly, I already have a patch I posted earlier.

I hope the alternate shortcuts are more convincing to you :)

Ted

[Kristian Amlie]

On 10/12/14 14:51, Ted Zlatanov wrote:
> On Wed, 10 Dec 2014 13:51:02 +0100 Kristian Amlie <kristia...@cfengine.com> wrote:
>
> KA> On 10/12/14 12:15, Ted Zlatanov wrote:
>>> 1) $(~var) to canonify a variable (I can use anything but ~)
>>>
>>> 2) $(\var) to escape a variable
>
> KA> I'm rather against these two. The meaning is not at all clear to
> KA> beginners, and it's one of those things that are pretty hard to search
> KA> for as well.
>
> Right, that's understood. But it's not intended for beginners (although
> you're right that they will need it pretty quickly with today's core).
>
> KA> I really don't see that "escape" and "canonify", both short words,
> KA> are such a big problem to type?
>
> Until general string concatenation is available, in 3.10 or 4.x or
> whenever, it's a big hassle to create intermediate variables (and
> because of CFEngine's evaluation model, it's much more complex to do it
> right). It's also nice to provide syntactic sugar for the most commonly
> used things, and it maintains backwards compatibility. So it's
> definitely not about saving keystrokes.
>
> The shortcuts could be $(:escape:var) and $(:canonify:var) just as well.
> Or $(escape(var)) and $(canonify(var)). Would that make them easier to
> accept?

I definitely like those better, yes. Much clearer what they do.

In general function expansion within strings would be a very useful
feature. I still think that proper string concatenation is the way to go
though, in the sense that you would provide your strings quoted, and
your function calls unquoted, with some gluing operator in between (or
maybe just whitespace, even). If we provide the expansion inside the
quotes I can see a lot of ambiguities arise: What if the argument to a
function takes a ')' for instance? How would you quote it, since you are
already inside quotes. Even Perl doesn't allow this.

So the result would be something like:

"newvar" string => "prefix_" + canonify($(var));

Do you think there would be a big difference in implementation between
the two approaches from the code perspective (I'm not an expert on the
evaluation part myself)?

> I'm glad you're unsure of the math shortcut. One step at a time :)
>
> If we follow the shortcut pattern above, the math shortcut could be
> $(:eval:2+2) or $(eval(2+2)).

This would also be covered by the above.

> KA> I'm not impossible to budge on this subject, but given that two of the
> KA> main developers (counting Dimitrios and myself) are skeptical towards
> KA> this change, I don't think it will be accepted in its current form.
>
> KA> I don't consider this subject closed though. For me the most
> KA> questionable part is the ~ and \ operators. Perhaps we should focus
> KA> first on the maths syntax and especially the string concatenation, since
> KA> I feel there is more consensus there.
>
> I can propose the math syntax in a separate patch, but the current code
> in expand.c is pretty nasty for iterated expansions. I can make it work
> for scalar expansions quickly, I already have a patch I posted earlier.

I don't believe the language should suffer for mistakes made in the
code. We should make the language the best that we can, even if it means
more work refactoring existing code.

--
Kristian

[Ted Zlatanov]

On Fri, 12 Dec 2014 11:03:52 +0100 Kristian Amlie <kristia...@cfengine.com> wrote:

KA> If we provide the expansion inside the quotes I can see a lot of
KA> ambiguities arise: What if the argument to a function takes a ')'
KA> for instance? How would you quote it, since you are already inside
KA> quotes. Even Perl doesn't allow this.

We already do this with JSON quoting: parsejson('{ "a": 1}')

Generally, you just add backslashes :)

KA> So the result would be something like:

KA> "newvar" string => "prefix_" + canonify($(var));

KA> Do you think there would be a big difference in implementation between
KA> the two approaches from the code perspective (I'm not an expert on the
KA> evaluation part myself)?

On the code side, the above is much more complex. The parser doesn't
handle function evaluation in the promiser, the class expression (which
could be a string), or basically anywhere but in the promise attributes.
Sometimes you can use scalar functions in body attributes, but it can be
tricky (jimis knows about the new cf-serverd code). To me this feels
like a 4.x kind of feature, it would be a big change.

My proposal can probably be done by me in a weekend and preserves
backward compatibility.

KA> I don't believe the language should suffer for mistakes made in the
KA> code. We should make the language the best that we can, even if it means
KA> more work refactoring existing code.

Sure. The problem is backward compatibility, as always.

Ted

[Kristian Amlie]

On 12/12/14 13:29, Ted Zlatanov wrote:
> On Fri, 12 Dec 2014 11:03:52 +0100 Kristian Amlie <kristia...@cfengine.com> wrote:
>
> KA> If we provide the expansion inside the quotes I can see a lot of
> KA> ambiguities arise: What if the argument to a function takes a ')'
> KA> for instance? How would you quote it, since you are already inside
> KA> quotes. Even Perl doesn't allow this.
>
> We already do this with JSON quoting: parsejson('{ "a": 1}')
>
> Generally, you just add backslashes :)
>
> KA> So the result would be something like:
>
> KA> "newvar" string => "prefix_" + canonify($(var));
>
> KA> Do you think there would be a big difference in implementation between
> KA> the two approaches from the code perspective (I'm not an expert on the
> KA> evaluation part myself)?
>
> On the code side, the above is much more complex. The parser doesn't
> handle function evaluation in the promiser, the class expression (which
> could be a string), or basically anywhere but in the promise attributes.
> Sometimes you can use scalar functions in body attributes, but it can be
> tricky (jimis knows about the new cf-serverd code). To me this feels
> like a 4.x kind of feature, it would be a big change.

Isn't the promise attributes where you need it, primarily? We could at
least start with that; it doesn't have to be perfect in the first
iteration, as long as the road ahead is clear.

> My proposal can probably be done by me in a weekend and preserves
> backward compatibility.

It's unfortunate that gluing quoted strings and unqouted function calls
together is more work, but I still believe it's the right way. The fact
that no other language has quoted function calls further alienates the
CFEngine policy language from common patterns, when we should work to
make it more recognizable.

> KA> I don't believe the language should suffer for mistakes made in the
> KA> code. We should make the language the best that we can, even if it means
> KA> more work refactoring existing code.
>
> Sure. The problem is backward compatibility, as always.

Would it be a problem with backwards compatibility? We are adding
syntax, not changing existing syntax.

--
Kristian

[Ted Zlatanov]

On Tue, 16 Dec 2014 09:25:18 +0100 Kristian Amlie <kristia...@cfengine.com> wrote:

KA> On 12/12/14 13:29, Ted Zlatanov wrote:
KA> "newvar" string => "prefix_" + canonify($(var));
KA> Do you think there would be a big difference in implementation between
KA> the two approaches from the code perspective (I'm not an expert on the
KA> evaluation part myself)?
>>
>> On the code side, the above is much more complex. The parser doesn't
>> handle function evaluation in the promiser, the class expression (which
>> could be a string), or basically anywhere but in the promise attributes.
>> Sometimes you can use scalar functions in body attributes, but it can be
>> tricky (jimis knows about the new cf-serverd code). To me this feels
>> like a 4.x kind of feature, it would be a big change.

KA> Isn't the promise attributes where you need it, primarily? We could at
KA> least start with that; it doesn't have to be perfect in the first
KA> iteration, as long as the road ahead is clear.

OK, as long as we don't say "it's good enough" as soon as that's done
and make the users jump through hoops forever after. It needs to be
consistent eventually.

>> My proposal can probably be done by me in a weekend and preserves
>> backward compatibility.

KA> It's unfortunate that gluing quoted strings and unqouted function calls
KA> together is more work, but I still believe it's the right way.

OK, I can't argue with that kind of conviction :)

KA> The fact that no other language has quoted function calls further
KA> alienates the CFEngine policy language from common patterns, when we
KA> should work to make it more recognizable.

I was talking about embedded canonicalization and escaping and math.
Those do exist in other languages to various degrees, from regular
expressions to bash-isms to Perl to Makefiles. They are convenient, but
seem to be losing favor as more programmers learn with the C++/Java
style of string and function call concatenation.

Anyhow, this direction is fine. Most people will find it familiar.
Please count me in favor, especially if it can realistically target 3.x.

>> Sure. The problem is backward compatibility, as always.

KA> Would it be a problem with backwards compatibility? We are adding
KA> syntax, not changing existing syntax.

You'd be changing the syntax of what's acceptable in a string context,
so the 3.6 parser will barf on something like

vars: "x" string => "1" + "2"; # or whatever the concat operator is

Compare with my proposal, which the 3.6 parser will ignore silently.

Ted

[Kristian Amlie]

On 16/12/14 15:28, Ted Zlatanov wrote:
> On Tue, 16 Dec 2014 09:25:18 +0100 Kristian Amlie <kristia...@cfengine.com> wrote:
>
> KA> On 12/12/14 13:29, Ted Zlatanov wrote:
> KA> "newvar" string => "prefix_" + canonify($(var));
> KA> Do you think there would be a big difference in implementation between
> KA> the two approaches from the code perspective (I'm not an expert on the
> KA> evaluation part myself)?
>>>
>>> On the code side, the above is much more complex. The parser doesn't
>>> handle function evaluation in the promiser, the class expression (which
>>> could be a string), or basically anywhere but in the promise attributes.
>>> Sometimes you can use scalar functions in body attributes, but it can be
>>> tricky (jimis knows about the new cf-serverd code). To me this feels
>>> like a 4.x kind of feature, it would be a big change.
>
> KA> Isn't the promise attributes where you need it, primarily? We could at
> KA> least start with that; it doesn't have to be perfect in the first
> KA> iteration, as long as the road ahead is clear.
>
> OK, as long as we don't say "it's good enough" as soon as that's done
> and make the users jump through hoops forever after. It needs to be
> consistent eventually.

Yes, of course!

> KA> The fact that no other language has quoted function calls further
> KA> alienates the CFEngine policy language from common patterns, when we
> KA> should work to make it more recognizable.
>
> I was talking about embedded canonicalization and escaping and math.
> Those do exist in other languages to various degrees, from regular
> expressions to bash-isms to Perl to Makefiles. They are convenient, but
> seem to be losing favor as more programmers learn with the C++/Java
> style of string and function call concatenation.
>
> Anyhow, this direction is fine. Most people will find it familiar.
> Please count me in favor, especially if it can realistically target 3.x.

Yes, that should be no problem I believe. Feature releases don't have to
be forwards compatible (3.6 agent running 3.7 policy), only backwards
compatible (3.7 agent running 3.6 policy).

>>> Sure. The problem is backward compatibility, as always.
>
> KA> Would it be a problem with backwards compatibility? We are adding
> KA> syntax, not changing existing syntax.
>
> You'd be changing the syntax of what's acceptable in a string context,
> so the 3.6 parser will barf on something like
>
> vars: "x" string => "1" + "2"; # or whatever the concat operator is
>
> Compare with my proposal, which the 3.6 parser will ignore silently.

Ignore silently, but still won't do the right thing, which I consider to
be worse. In any case, we have the masterfiles/lib/3.x mechanism to make
sure that incompatible policy doesn't leak into the wrong version of the
agent. If any users require it for their own policy, they can copy that
approach.

--
Kristian

[Ted Zlatanov]

On Wed, 17 Dec 2014 07:58:19 +0100 Kristian Amlie <kristia...@cfengine.com> wrote:

KA> Yes, that should be no problem I believe. Feature releases don't have to
KA> be forwards compatible (3.6 agent running 3.7 policy), only backwards
KA> compatible (3.7 agent running 3.6 policy).

I recall the goal was to preserve forward compatibility for 2 minor
releases, so e.g. 3.6 should be able to run 3.8 policy. But you're
right that versioned libdirs solve this problem :)

Ted

[Ted Zlatanov]

On Tue, 16 Dec 2014 09:28:20 -0500 Ted Zlatanov <t...@lifelogs.com> wrote:

[ We were talking about allowing function calls in places like class
expressions and promisers, instead of implementing special escape
sequences to canonify strings. ]

TZ> Anyhow, this direction is fine. Most people will find it familiar.
TZ> Please count me in favor, especially if it can realistically target 3.x.

It's been a while since this discussion and we're talking about 3.7 now.
Is there any interest in this functionality for 3.7 or 3.8? I'm happy to
invest time in implementing it and perhaps others will help too.

Ted

[Brian Bennett]

I would probably take advantage of it if it's there.

--
Brian Bennett
Looking for CFEngine training?
http://www.verticalsysadmin.com/

> --
> You received this message because you are subscribed to the Google Groups "dev-cfengine" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dev-cfengine...@googlegroups.com.
> To post to this group, send email to dev-cf...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/dev-cfengine/87pp873hh0.fsf%40lifelogs.com.

[Kristian Amlie]

Absolutely! Perhaps you can create a task with the "spec" first, based
on the discussions in this thread, so we know we are on the same page?

As long as it doesn't go beyond April it should be no problem to include
it in 3.7.

--
Kristian

[Ted Zlatanov]

As a first step towards allowing functions everywhere as we discussed,
I've proposed a solution for https://dev.cfengine.com/issues/2504 in
https://github.com/cfengine/core/pull/2150

To summarize, this is syntactic sugar to make

"$(myvariable)":: "mypromise"...;

act exactly like

any:: "mypromise" ifvarclass => "$(myvariable)" ...;

It's a first step, a small change, and lets us discover all the places
we need to touch in order to accomplish the greater task. More
importantly, if a C amateur like me can hack on it for 2 hours and
produce a working patch, that gives me hope the greater task is possible :)

The patch needs review and testing, so I'd appreciate your help.

Thanks
Ted

[Neil Watson]

On Wed, Mar 25, 2015 at 08:05:11AM -0400, Ted Zlatanov wrote:
>As a first step towards allowing functions everywhere as we discussed,
>I've proposed a solution for https://dev.cfengine.com/issues/2504 in
>https://github.com/cfengine/core/pull/2150
>
>To summarize, this is syntactic sugar to make
>
> "$(myvariable)":: "mypromise"...;
>
>act exactly like
>
> any:: "mypromise" ifvarclass => "$(myvariable)" ...;

Exciting. If it acts like ifvarclass then does the next promise have to
have its own class?

"$(myvariable)":: "mypromise"...;
any:: "mypromise" ...;

How are compound expressions handled?
"$(myvariables)".other_class:: "mypromise"...

--
Neil H Watson
Sr. Partner, Architecture and Infrastructure
CFEngine reporting: https://github.com/evolvethinking/delta_reporting
CFEngine policy: https://github.com/evolvethinking/evolve_cfengine_freelib
CFEngine and vim: https://github.com/neilhwatson/vim_cf3
CFEngine support: http://evolvethinking.com

[Ted Zlatanov]

On Wed, 25 Mar 2015 08:31:00 -0400 Neil Watson <cfen...@watson-wilson.ca> wrote:

NW> Exciting. If it acts like ifvarclass then does the next promise have to
NW> have its own class?

NW> "$(myvariable)":: "mypromise"...;
NW> any:: "mypromise" ...;

It's like regular class contexts, so no, it "sticks" until the end of
the current promise type or a new class context.

NW> How are compound expressions handled?
NW> "$(myvariables)".other_class:: "mypromise"...

It has to be a string, and is passed exactly to ifvarclass:

"$(myvariables).other_class":: "mypromise"...

Ted

[Kristian Amlie]

This looks quite exciting, and a much better solution than ifvarclass!

--
Kristian

[Mark Burgess]

This won't work in the current evaluator because classes are not part of
promise expansion, hence the reason for the ifvarclass workaround.

M

[Ted Zlatanov]

On Fri, 27 Mar 2015 10:54:50 +0100 Mark Burgess <mark.b...@cfengine.com> wrote:

MB> On 27/03/15 10:30, Kristian Amlie wrote:
>> On 25/03/15 13:54, Ted Zlatanov wrote:

>>> "$(myvariables).other_class":: "mypromise"...
>> This looks quite exciting, and a much better solution than ifvarclass!

MB> This won't work in the current evaluator because classes are not part
MB> of promise expansion, hence the reason for the ifvarclass workaround.

Hi Mark,

it seems to work in my testing. Can you give some counter-examples I can
use in the acceptance tests for
https://github.com/cfengine/core/pull/2150 please?

(I'm still using ifvarclass behind the scenes.)

Thanks
Ted

[Mark Burgess]

"$(myvariables).other_class":: "mypromise"...

>>> This looks quite exciting, and a much better solution than ifvarclass!
> MB> This won't work in the current evaluator because classes are not part
> MB> of promise expansion, hence the reason for the ifvarclass workaround.
>
> Hi Mark,
>
> it seems to work in my testing. Can you give some counter-examples I can
> use in the acceptance tests for
> https://github.com/cfengine/core/pull/2150 please?
>
> (I'm still using ifvarclass behind the scenes.)
>
> Thanks
> Ted

Well, I don't think class eval is inside the list expansion, but perhaps
that has changed. If it works, ignore my comment. Just didn't want
anyone to bite off more than they expected.

M

[Ted Zlatanov]

On Fri, 27 Mar 2015 16:02:45 +0100 Mark Burgess <mark.b...@cfengine.com> wrote:

MB> "$(myvariables).other_class":: "mypromise"...

This seems to work. I've added it to the acceptance test and updated
the PR.

#+begin_src cfengine3
bundle agent main
{
vars:
"mylist" slist => { "a", "b", "c" };
classes:
"$(mylist)" expression => "any";

reports:
"$(mylist).cfengine"::
"GOOD: iteration class $(mylist)";
}
#+end_src

Output:

R: GOOD: iteration class a
R: GOOD: iteration class b
R: GOOD: iteration class c

Is that what you meant?

MB> Well, I don't think class eval is inside the list expansion, but
MB> perhaps that has changed. If it works, ignore my comment. Just didn't
MB> want anyone to bite off more than they expected.

The most important thing is to ensure it doesn't break anything, so
counter-examples that can go into the acceptance test are the best way
to help me do that. Underneath, this is really a very simple
transformation, not actual new behavior.

Ted

[Mark Burgess]

Forget I said anything.

M

--
Founder and Advisor to CFEngine
Author of In Search of Certainty: The Science of our Information Infrastructure

[Nick Anderson]

What if the variable dereferences to am invalid class name. Shouldn't it be automatically canonified?

Sent from my mobile device.

-----Original Message-----

--
You received this message because you are subscribed to the Google Groups "dev-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-cfengine...@googlegroups.com.
To post to this group, send email to dev-cf...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dev-cfengine/87pp7uxy7t.fsf%40lifelogs.com.

[Ted Zlatanov]

On Fri, 27 Mar 2015 08:32:07 -0700 Nick Anderson <nick.a...@cfengine.com> wrote:

NA> What if the variable dereferences to am invalid class name. Shouldn't it be automatically canonified?

It acts exactly like ifvarclass. Internally it's appended to the
ifvarclass attribute.

Ted

[Nick Anderson]

Yes, I'm saying I think it ought to automatically canonify.

Sent from my mobile device.

-----Original Message-----
From: Ted Zlatanov <t...@lifelogs.com>
To: dev-cf...@googlegroups.com
Sent: Fri, 27 Mar 2015 8:51 AM
Subject: Re: proposal: inline expressions in strings

--
You received this message because you are subscribed to the Google Groups "dev-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-cfengine...@googlegroups.com.
To post to this group, send email to dev-cf...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dev-cfengine/87lhiixwte.fsf%40lifelogs.com.

[Neil Watson]

On Fri, Mar 27, 2015 at 11:51:57AM -0400, Ted Zlatanov wrote:
>It acts exactly like ifvarclass. Internally it's appended to the
>ifvarclass attribute.

It's exciting to think of ifvarclass going away, but will this quick fix
haunt us later? Will we put off a better fix because this is good
enough?

Possible tests:

What happens with ${var}:: is undefined or not defined *yet*?
What happens if ${var}:: is not a legal class string?
Can ${var} contain a class expression (class1.class2)?
Make a test for ${var} being an array or from a data container.

I was going to run Ted's change against EFL, but realized I'll have to
rewrite EFL to remove ifvarclass. More work than I can afford unless
this is going to be accepted into the core.

Sincerely,

[Ted Zlatanov]

On Fri, 27 Mar 2015 12:41:23 -0400 Neil Watson <cfen...@watson-wilson.ca> wrote:

NW> On Fri, Mar 27, 2015 at 11:51:57AM -0400, Ted Zlatanov wrote:
>> It acts exactly like ifvarclass. Internally it's appended to the
>> ifvarclass attribute.

NW> It's exciting to think of ifvarclass going away, but will this quick fix
NW> haunt us later? Will we put off a better fix because this is good
NW> enough?

ifvarclass is not going away!!!

This change is not a fix, it's syntactic sugar. It converts

"ANYTHING":: "promise" ifvarclass => "V";

to

any:: "promise" ifvarclass => "(ANYTHING).V";

That's it. I promise you it's really that simple and predictable.

NW> What happens with ${var}:: is undefined or not defined *yet*?

Same as ifvarclass, but note it's "${var}":: (must be in single or
double quotes).

NW> What happens if ${var}:: is not a legal class string?

Same as ifvarclass, but note it's "${var}":: (must be in single or
double quotes).

NW> Can ${var} contain a class expression (class1.class2)?

Same as ifvarclass.

NW> Make a test for ${var} being an array or from a data container.

Same as ifvarclass.

Ted

[Eystein Måløy Stenberg]

Hm, I think it needs to apply to a block and not just 1 promise like
ifvarclass, otherwise the behaviour would seem counter-intuitive.

What happens in the case below?

vars:
"myvar" => string "cfengine_3";

commands:

$(myvar)::

"/bin/echo abc";

"/bin echo def";

--

Eystein

[Ted Zlatanov]

On Fri, 27 Mar 2015 14:20:18 -0700 Eystein Måløy Stenberg <eystein.mal...@cfengine.com> wrote:

EMS> Hm, I think it needs to apply to a block and not just 1 promise like
EMS> ifvarclass, otherwise the behaviour would seem counter-intuitive.

I agree.

EMS> What happens in the case below?

vars:
"myvar" => string "cfengine_3";

commands:

$(myvar)::

"/bin/echo abc";

"/bin echo def";

The context "sticks". Internally, P.currentvarclasses is set until
either the current parsing section ends or a new context is set. In
your example, both commands would be run.

I added that case to the acceptance test.

Ted

[Bas van der Vlies]

I am just back from holidays and catching uo with all the new stuff ;-). Great work Ted

> --
> You received this message because you are subscribed to the Google Groups "dev-cfengine" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dev-cfengine...@googlegroups.com.
> To post to this group, send email to dev-cf...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/dev-cfengine/87619mxg9l.fsf%40lifelogs.com.

> For more options, visit https://groups.google.com/d/optout.

---
Bas van der Vlies
| Operations, Support & Development | SURFsara | Science Park 140 | 1098 XG Amsterdam
| T +31 (0) 20 800 1300 | bas.van...@surfsara.nl | www.surfsara.nl |

[Aleksey Tsalolikhin]

I agree.  This is sweet!  And thanks all for the support of Ted's work, your care to make sure this is done right, etc.

To view this discussion on the web visit https://groups.google.com/d/msgid/dev-cfengine/23EF40FA-B926-4539-9F8C-20FF51CDA544%40surfsara.nl.

[Ted Zlatanov]

On Wed, 25 Mar 2015 08:05:11 -0400 Ted Zlatanov <t...@lifelogs.com> wrote:

TZ> As a first step towards allowing functions everywhere as we discussed,
TZ> I've proposed a solution for https://dev.cfengine.com/issues/2504 in
TZ> https://github.com/cfengine/core/pull/2150

TZ> To summarize, this is syntactic sugar to make

TZ> "$(myvariable)":: "mypromise"...;

TZ> act exactly like

TZ> any:: "mypromise" ifvarclass => "$(myvariable)" ...;

Update: this was merged.

I think allowing functions everywhere is not something I can tackle with
my limited time and abilities (though I gave it a try), so I'll have to
ask the CFEngine developers or others to step in.

Furthermore, as I mentioned in another thread, I think allowing inline
expressions is still a good idea despite all the arguments in favor of
"functions everywhere." I gave this a lot of thought and think inline
expressions fit really well with the way CFEngine works and would
actually lower the learning curve and make policy writing more pleasant.

Ted

[Brian Bennett]

Does that mean that it will be in 3.7?

--
Brian Bennett
Looking for CFEngine training?
http://www.verticalsysadmin.com/

> --
> You received this message because you are subscribed to the Google Groups "dev-cfengine" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dev-cfengine...@googlegroups.com.
> To post to this group, send email to dev-cf...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/dev-cfengine/87pp4ek2q2.fsf%40lifelogs.com.

[Ted Zlatanov]

On Mon, 29 Jun 2015 13:31:57 -0700 Brian Bennett <brian....@verticalsysadmin.com> wrote:

>> On Jun 29, 2015, at 7:29 AM, Ted Zlatanov <t...@lifelogs.com> wrote:

TZ> To summarize, this is syntactic sugar to make
TZ> "$(myvariable)":: "mypromise"...;
TZ> act exactly like

TZ> any:: "mypromise" ifvarclass => "$(myvariable)" ...;
>>
>> Update: this was merged.
>>
>> I think allowing functions everywhere is not something I can tackle with
>> my limited time and abilities (though I gave it a try), so I'll have to
>> ask the CFEngine developers or others to step in.
>>
>> Furthermore, as I mentioned in another thread, I think allowing inline
>> expressions is still a good idea despite all the arguments in favor of
>> "functions everywhere." I gave this a lot of thought and think inline
>> expressions fit really well with the way CFEngine works and would
>> actually lower the learning curve and make policy writing more pleasant.

BB> Does that mean that it will be in 3.7?

I'm honestly not sure which of the three things you mean since you
top-posted :)

1) quoted contexts like "DEBUG|DEBUG_$(this.bundle)":: are in 3.7!

2) "functions everywhere" is not in 3.7 and I don't think I can do it
because it has a lot of complexity in the way it will interact with
promises and bundles.

3) inline expressions in strings is not in 3.7 but I can definitely do
it in a few hours, since it doesn't have any evaluation or parsing
implications.

Ted

[Aleksey Tsalolikhin]

On Mon, Jun 29, 2015 at 7:29 AM, Ted Zlatanov <t...@lifelogs.com> wrote:

On Wed, 25 Mar 2015 08:05:11 -0400 Ted Zlatanov <t...@lifelogs.com> wrote:

TZ> As a first step towards allowing functions everywhere as we discussed,
TZ> I've proposed a solution for https://dev.cfengine.com/issues/2504 in
TZ> https://github.com/cfengine/core/pull/2150

TZ> To summarize, this is syntactic sugar to make

TZ>     "$(myvariable)":: "mypromise"...;

TZ> act exactly like

TZ>     any:: "mypromise" ifvarclass => "$(myvariable)" ...;

Update: this was merged.

Nice!

 

I gave this a lot of thought and think inline
expressions fit really well with the way CFEngine works and would
actually lower the learning curve and make policy writing more pleasant. 

Awesome sauce. 

--

Need CFEngine training?  Email trai...@verticalsysadmin.com

[Dimitrios Apostolou]

On Mon, Jun 29, 2015 at 4:29 PM, Ted Zlatanov <t...@lifelogs.com> wrote:
> TZ> To summarize, this is syntactic sugar to make
>
> TZ> "$(myvariable)":: "mypromise"...;

+1 very nice. Thanks Ted for the contribution!

>
> TZ> act exactly like
>
> TZ> any:: "mypromise" ifvarclass => "$(myvariable)" ...;
>
> Update: this was merged.

Dimitris

[Dimitrios Apostolou]

Ted, as far as I understand you want the canonification special syntax
to avoid intermediate variables for "myvariable" in this new syntax:

On Tue, Jun 30, 2015 at 3:01 AM, Ted Zlatanov <t...@lifelogs.com> wrote:
> TZ> "$(myvariable)":: "mypromise"...;

What if canonification was automatic in that context? Does it even
make sense to not canonify? May it's already like that?


Thanks,
Dimitris

[Ted Zlatanov]

On Thu, 2 Jul 2015 21:11:10 +0200 Dimitrios Apostolou <ji...@cfengine.com> wrote:

DA> Ted, as far as I understand you want the canonification special syntax
DA> to avoid intermediate variables for "myvariable" in this new syntax:

DA> On Tue, Jun 30, 2015 at 3:01 AM, Ted Zlatanov <t...@lifelogs.com> wrote:
TZ> "$(myvariable)":: "mypromise"...;

That's just one of many use cases. The general need is when you want to
grab and canonify a variable's value but want to avoid the intermediate
work to generate it, especially if you need a reliable mapping. The
normal ordering can make that simple operation really tough.

DA> What if canonification was automatic in that context? Does it even
DA> make sense to not canonify? May it's already like that?

It can be a variable that has "DEBUG.inform_mode" in the value, so
auto-canonifying it would be wrong.

Ted

[Dimitrios Apostolou]

On Thu, Jul 2, 2015 at 10:46 PM, Ted Zlatanov <t...@lifelogs.com> wrote:
> It can be a variable that has "DEBUG.inform_mode" in the value, so
> auto-canonifying it would be wrong.

Oh, I didn't know that /class expressions/ were allowed in the
variable. I was thinking only single class names.


Dimitris

[Ted Zlatanov]

On Mon, 29 Jun 2015 10:29:09 -0400 Ted Zlatanov <t...@lifelogs.com> wrote:

TZ> I think allowing functions everywhere is not something I can tackle with
TZ> my limited time and abilities (though I gave it a try), so I'll have to
TZ> ask the CFEngine developers or others to step in.

TZ> Furthermore, as I mentioned in another thread, I think allowing inline
TZ> expressions is still a good idea despite all the arguments in favor of
TZ> "functions everywhere." I gave this a lot of thought and think inline
TZ> expressions fit really well with the way CFEngine works and would
TZ> actually lower the learning curve and make policy writing more pleasant.

I wrote a big patch over the weekend to do some of it but am just not
happy with it, so let me explain the situation. Currently the promiser
is a plain C string. This means that to allow expressions in the
promiser (the most useful part of "functions everywhere" other than the
new operator syntax) we need to convert the promiser to an Rval.

Executive summary: compared to the inline string expressions, which are
20-100 lines of code, the "functions everywhere" work is just massive.

Converting the promiser to an Rval affects ALL of CFEngine, from top to
bottom. My patch is about 500 changed lines, most of it replacing
pp->promiser with RvalToString(pp->promiser) (which will not work in
many places, I was just finding the trouble spots!).

CFEngine 3 does evaluation with Rvals, so I don't know another way
(besides writing a new evaluation engine) to implement this. The problem
is not with general evaluation, that can be done even during the parse.
It's with iteration and variables or classes that are not yet defined.

Another place where a Rval promiser is a problem is where the promiser
is used for promise locking. So changing it to an Rval means that the
locks need to use a serialized version of that Rval.

Another problem is that Log() can't take Rvals so in at least 200 places
we have to serialize pp->promiser.

There's an alternative approach: make the promiser a promise attribute
when it's an Rval, and try to patch it into the string whenever it's
evaluated from an expression to a value (which happens during evaluation
but before promises are verified). That will keep the promiser static
(so it looks like '"a"+"b"') until just before the verify code is called
(at that point, evaluation of the promise attributes is done). And at
that point, it will be replaced with another static string. I see lots
of potential bugs with the desynchronized state of the promise (if the
promiser string is not updated at all the right times).

The last alternative is a new syntax and a new evaluation engine, which
sounds scary but may turn out to be the best solution technically and
long-term (see also the "CFEngine 4" discussions).

You can look at my promiser-as-rval patch in
https://github.com/tzz/core/tree/feature/promiser-rval but it's not
going to work at all, it crashes right away.

Thanks
Ted

[Ted Zlatanov]

On Tue, 14 Jul 2015 10:06:26 -0400 Ted Zlatanov <t...@lifelogs.com> wrote:

TZ> There's an alternative approach: make the promiser a promise attribute
TZ> when it's an Rval, and try to patch it into the string whenever it's
TZ> evaluated from an expression to a value (which happens during evaluation
TZ> but before promises are verified). That will keep the promiser static
TZ> (so it looks like '"a"+"b"') until just before the verify code is called
TZ> (at that point, evaluation of the promise attributes is done). And at
TZ> that point, it will be replaced with another static string. I see lots
TZ> of potential bugs with the desynchronized state of the promise (if the
TZ> promiser string is not updated at all the right times).

I am still concerned I've missed some cases, but this is working for me
now. The PR is in https://github.com/cfengine/core/pull/2295 and could
use a review. It was a lot of work.

Ted