In search of a CFEngine language specification (for 4.x)

[Mike Weilgart]

Hello,

I want to take a moment to talk about CFEngine adoption far, far forward into the future.

Promise Theory is an extremely brilliant mathematical model on which to base configuration management.  CFEngine was the original configuration management software, and it's noteworthy that all its competitors have copied the notion of convergence and contain some aspects of a declarative language.  However, CFEngine 3 is much closer to a clean implementation of Promise Theory than any other software existing today, and has this as an "edge" over its rivals.

------

After working for over a year with CFEngine in an enterprise-scale environment, training technical personnel in the writing of CFEngine policy, driving executive adoption of CFEngine and its enterprise reporting UI, and comparing CFEngine to its competitors as well as to its underlying Promise Theory—the obstacles I see which must be dealt with to ensure CFEngine's long term survival are as follows:

1. The learning curve.  More explicitly, the lack of easily understandable and easily searchable documentation.  Effort must be explicitly exerted to make CFEngine easy to get started with for newcomers.  As a community, we need to actually put attention on "number of users of CFEngine" as a statistic which must be increased.

This is the factor which trumps everything else, as if there are users, there will be new developers, there will be improvements, there will be adoption.

(This is sometimes referred to as "Worse is Better," which is really not true.  The valid point underlying that statement is that regardless of how good your product is, unless it is used it will not survive and thrive.  It is possible to be excellent and be widely used.)

2. CFEngine's adherence to Promise Theory.

The primary advantage of CFEngine over its competitors is that it is firmly rooted in a precise mathematical model which closely mirrors the reality of configuration management.

Configuration management products which are not based on any coherent theory may be widely used at the present, but their viability long term is closely related to how accurately they are based on a workable approach (a workable theory) for the problem domain to which they are addressed.

The greatest threat to CFEngine's future survival (spanning multiple decades) is its potential violation of its fundamental purpose, which is to apply Promise Theory to computer configuration management.  More bluntly, if CFEngine is to stand the test of time, it must adhere much more closely to Promise Theory at every level of its design and implementation than it currently does.

This relates to the first point above, as reading the current CFEngine documentation will not give a newcomer any insight into Promise Theory.

I believe that Promise Theory is an extremely accurate model and a workable mathematics, and that a product or language which cleanly implements that mathematical model will be, and remain, successful.

------

This brings me to the purpose of this email.

CFEngine may be regarded conceptually as consisting of three distinct parts:

1. The underlying theory and mathematical model (Promise Theory);

2. A conception of how that mathematical model may be applied to the subject of configuration management;

3. An implementation of the ideas in (2) as a software product.

Part 2 may appear unfamiliar.  That's because it doesn't truly exist currently as a distinct entity.  Part 2 should consist of a language specification for the CFEngine policy language.

Right now there is only one interpreter for CFEngine policies.  As such, there is no clear differentiation between the language itself (as a means of representing concepts, ideas, intentions, configuration states) and the particular parser for that language known as CFEngine Core.  In other words, there is no distinction between the language and the implementation.

It is particularly important for a declarative language to have a clear-cut specification, as the concept embodied in CFEngine is that you should not concern yourself with the procedural, sequential steps by which CFEngine will achieve the end state you specify—you should define the end state and CFEngine will make it so.  (I wrote about this elsewhere recently, though perhaps not so clearly.)

Anyone who has worked with CFEngine 3 policies at scale for any length of time knows that this ideal has not been achieved in CFEngine 3.  There are innumerable ways in which one must tweak and adjust the sequence of promises, specify procedures, and work around implementation flaws to achieve the desired end state.

For example, if one has mutual promise dependencies (of different promise types) within a bundle which exceed the arbitrary three-pass limitation in CFEngine Core, the final promises in the dependency chain will never be evaluated.  This is both documented behavior and has possible workarounds; that's not the point.  The point is that it is an implementation detail that a policy writer must take into account, or the results will be totally unexpected.  Thus it violates the principle of simply implementing Promise Theory.

-------

To implement Promise Theory directly as a configuration management language will not require drastic changes to the CFEngine policy language.

However, establishing a language spec will change the expected behavior of all existing policies written for CFEngine 3, if only because they are so closely tied to implementation details as to leave no "wiggle room" to establish a spec that is backward compatible and future compatible.

More bluntly, existing policies are so inextricably intertwined with implementation details that most of the current execution results should be undefined behavior in a well-written spec.

I made a brief start at writing such a language spec, which may illustrate better what I am expounding.

-------

(In actual fact, there should probably be a fourth ingredient beyond those enumerated above: In between the underlying theory and a CFEngine language specification [in between (1) and (2) in the earlier list] should come a directive establishing at a high level what features a language should include (and should not include) in order to truly reflect and implement Promise Theory.  This would be a document for Promise Theory serving the same role as The Third Manifesto serves for the relational database model.  But that's not quite so pertinent for the CFEngine Dev's mailing list.)

-------

What is your response, as a CFEngine developer, to the ideas laid out above?  Shall we work toward a more precise language specification for CFEngine 4.x?

Thanks for your time and attention.

Best,

--Mike Weilgart

Vertical Sysadmin, Inc.

[P. Christeas]

At Sunday, 27 November 2016 9:33 am. Mike Weilgart wrote:
> I want to take a moment to talk about CFEngine adoption *far, far forward*
> into the future.
> ...
> What is *your* response, as a CFEngine developer, to the ideas laid out

> above? Shall we work toward a more precise language specification for
> CFEngine 4.x?

I just couldn't agree more.

For a start, I can presume that current CFEngine3.x promise file syntax may be
more than enough to cover all usages and underlying theory. Let me provoke
that there is no need to introduce any new syntactic elements.

But, as you said, problem is that many of the existing promise types
(implementations, that is) have arbitrary quirks in the way they are evaluated
and executed. I had resorted several times into reading the source code to
understand why some promises had been behaving so.

So, indeed, there must be a core, clear and universal approach to all
promises, their dependencies etc.

I would dare to add, also, that extending promise types within the monolithic
binary may not work long-term, making CFEngine's code hard to maintain and
adapt to all usage scenarios (think the needs of embedded devices vs. data-
center clusters). Given the above formal language specification, parts of
CFEngine could be split out to /modules/, as in dynamically-linked libraries.

That said, I'm also (IMHO, always) against this trend of abstracting the
promise declaration at some JSON configuration file. Using a different
technology for each layer of abstraction makes the system less comprehensible,
the learning curve steeper. Instead, let's investigate why the CFEngine
language/syntax can't (or can it? ) be used, itself, to abstract promises/
bundles into higher levels..

[Mike Weilgart]

> For a start, I can presume that current CFEngine3.x promise file syntax may be 
> more than enough to cover all usages and underlying theory. Let me provoke 
> that there is no need to introduce any new syntactic elements.

This is almost entirely true.  There is one syntactic change I believe may be appropriate (and only one).  That would be to resolve the non-symmetricity between the "depends_on" attribute and the "promisees" list, either by adding a dedicated syntax for listing "dependent" promises, or by removing the dedicated syntax for "promisees."

As hardly anyone uses the "promisees" list and there is a dedicated syntax for list attributes already, I would favor removing the "promisees" syntax entirely.  (If it is desired it could be defined as a "promisees" attribute, which would increase syntax regularity.)  I will write as though "promisees" lists don't exist, as I favor removing that dedicated syntax in CFEngine 4.x.

Also, let's be clear that by syntax we mean only those elements described in the very first two code snippets in the Language Concepts documentation.  These elements are so simple that they may be formally defined within a two page document.  From a syntactic standpoint, with no concern for semantics, there are only two alphanumeric (non-symbolic) keywords: "bundle" and "body."

Here is a rough draft of a formal syntax specification:

------------------

A policy, informally, is a collection of bundles.  More precisely, a policy consists of zero or more bundles and zero or more bodies.

A bundle is a collection of zero or more promises.  More precisely, a bundle consists of the "bundle" keyword, a bundle type, a bundle name, an open curly brace, zero or more promises, and a close curly brace.

A promise consists of a promise type followed by a colon, an optionally specified context, a promiser, zero or more comma-separated attributes, and a terminating semicolon.  The promise type (along with its following colon) may be omitted from any promise other than the first promise in a bundle.  (Semantically: An omitted promise type shall be inferred from the closest preceding promise in the bundle for which a promise type is specified.  A promise with an omitted context but a specified promise type shall be inferred to have the context "any::".  A promise with both an omitted context and an omitted promise type shall have its context inferred from the immediately preceding promise in the bundle.)

A body is a collection of attributes.  More precisely, a body consists of the "body" keyword, a body type, a body name, an open curly brace, one or more semicolon-terminated attributes (with optionally-specified contexts) and a close curly brace.  Any attribute within a body may be preceded by a context.  (Semantically: If a context is specified within a body, it applies to all attributes forward from that context until either another context is specified or the end of the body is reached.  Attributes for which no context is specified shall be inferred to have the context "any::".)

An attribute is a key-value pair.  More precisely, an attribute consists of an attribute name, the characters "=>", and a value.

A value can be either a scalar value or a list value.

A scalar value is either an integer, a real number, or a string.

A list value is either an integer list, a real number list, or a string list.  Any list value consists of an open curly brace, zero or more comma-separated scalar values which are all of the same type, and a closing curly brace.

A context consists of a class expression followed by the characters "::".

Body semantics:

If an attribute has an attribute name which is a defined body type, the value of the attribute is expected to be a string containing a body name referring to a body which is defined somewhere in the policy.

------------------

In this draft syntax I only defined some italicized terms, notably not including "string" nor "class expression."  I believe these are much more complicated to define precisely in a first draft, particularly considering the quoting and whitespace rules.

I do think typing should be strengthened (strong typing) in CFEngine 4 compared to CFEngine 3.7.

--------

One syntactic change I considered but ultimately rejected is the request for anonymous in-line bodies in promises (as described in CFE-2196, the help-cfengine post "Anonymous bodies - language syntax suggestion," and CFE-2064.)  I wrote the post, and yet I reject this idea.  Why?

Consider the purpose of promise bodies: they are for knowledge management.

All attributes of, let us say, a "copy_from" body are actually attributes of a "files" promise.  It would not violate the simple, regular, predictable syntax outlined above if a "files" promise were allowed to directly include those attributes rather than referencing a "copy_from" body.  Why not allow this?

Likewise, why hardcode the allowable body types in the parser?  Why not simply define ALL attribute names related to a promise type as belonging to that promise type, and allow ANY arbitrary collection of them to be specified as a body type?

This could done (within the grammar above) with a special "bundle type."  Call it a "body_def" and you would get the following:

#NOT REAL CODE

bundle body_def copy_from {

  attributes:

    "source";

    "servers";

    "collapse_destination_dir"

      default_string => "false";

    "compare"

      default_string => "???",

      comment => "There is no way to accomplish default behavior explicitly.";

    "copy_backup"

      default_string => "true";

    "encrypt"

      default_string => "false";

    "check_root"

      default_string => "false";

    "copylink_patterns";

    "copy_size"

      default_irange => "0,inf",

      comment => "I have no idea what type this is supposed to be.";

    "findertype";

    "linkcopy_patterns";

    "link_type"

      default_string => "symlink";

    "force_update"

      default_string => "false";

    "force_ipv4"

      default_string => "false";

    "portnumber";

    "preserve"

      default_string => "false";

    "protocol_version"

      default_string => "classic",

      comment => "Can also be an int???";

    "purge"

      default_string => "false";

    "stealth"

      default_string => "false";

    "timeout"

      default_int => "default_timeout",

      comment => "Yes, I know that's not an int.";

    "trustkey"

      default => "false",

      comment => "I forgot '_string' because it wouldn't be intuitive.

                  I like 'default' better.";

    "type_check" default => "true";

    "verify" default => "false";

}

----------------------

Writing this out was an interesting exercise that highlighted a few more syntax questions:

1. In CFEngine 3, each attribute name is tied to a specific type.  In some CFEngine 4 cases I can conceive (such as the "default" attribute of an "attributes" promise in a "body_def" bundle, in the made up example above), it would be useful not to tie these together so tightly.  Should attribute names be tied to specific data types by the parser?

2. Booleans are inconsistently handled in CFEngine 3.  For class expressions, "any" is true and "!any" is false, and there is evidently a dedicated internal type for classes which is distinct from strings.

Menu items for attributes which may be "true" or "false" are nicely human readable, but if we are to expand the interoperability of the language to allow new promise types to be added more easily it could make sense to allow booleans to be an actual type.

For instance, why should the following syntax be necessary in CFEngine 3?

body copy_from conditionally_secure {

  be_secure::

    encrypt => "true";

  !be_secure::

    encrypt => "false";

}

  

If booleans were fully supported as a type, related to class expressions, you could simplify this.  Incorporating my notions of inline attributes instead of bodies:

  files:

    "/path/to/some/file"

      source => "/path/on/hub/to/file",

      servers => "$(sys.policy_hub)",

      encrypt => "be_secure";

3. Should there be some sort of "range" data type??  It seems that "copy_size" could just as well be two separate attributes, which wouldn't violate the key-value pair concept and would preserve regularity.

4. Very importantly, CFEngine 3 contains multiple promises packed into a single "promise."  This is viewable in promise outcomes, wherein multiple outcomes can result from a "single" promise.  (Nick Anderson calls them "compound promises.")  This is probably the single biggest discrepancy between Promise Theory per se and the CFEngine 3 implementation.

How should this be coordinated with the language semantics and syntax?

(Note that the "create" attribute of a files promise does not strictly align to Promise Theory at all.  This could be made the default behavior, and the current behavior could be accomplished with "if => fileexists("$(this.promiser)")".  This would solve accidental empty file creation as just a side effect.)

---------

> So, indeed, there must be a core, clear and universal approach to all 

> promises, their dependencies etc. 

> 

> I would dare to add, also, that extending promise types within the monolithic 

> binary may not work long-term, making CFEngine's code hard to maintain and 

> adapt to all usage scenarios (think the needs of embedded devices vs. data- 

> center clusters). Given the above formal language specification, parts of 

> CFEngine could be split out to /modules/, as in dynamically-linked libraries. 

I completely agree with both points here.  There will always be new needs requiring new additional promise types as computer applications in the world continue to grow exponentially.

For example, Promise Theory is more than adequate to model cloud orchestration, yet CFEngine offers no orchestration features.  You might have a "machines:" or "vms:" promise type, with attributes such as 'os => "redhat 6"', 'subnet => "10.10.10.0/24"'—all sorts of things.  These would require updates to the language specification but these should be semantic changes rather than syntactic.

(Note: I actually wrote the text from here down before I wrote any other part of this post.)

Drawing on the pure mathematical model of Promise Theory, it should be possible to define a standard by which any promises (and promisees and promisers) can be expressed—and if that notation is compatible wherever possible to the CFEngine 3 policy language, the connection can be quite straight forward.  What I'm talking about here would probably be classified as a formal grammar.  C has an official formal grammar, though not available for free.  (It's also summarized in K&R.)  POSIX shell has a formal grammar.

Now whether or not CFEngine gets a formal grammar (which it should, eventually), it should certainly have a language specification.  They're not the same thing.  In actual fact, because of the (intended) regularity of CFEngine, I suspect that a formal grammar for the CFEngine 4 policy language would actually be simpler than the full language specification!

The language specification would include the defined behavior of all the promise types and attributes, whereas the grammar would define simply how elements are structured.  If the regularity between bundles and bodies could be captured in the formal grammar, any manner of promise could be written in the language, and people could learn the language without necessarily learning the particular bundle types, promise types and attributes available in a particular parser for the "Promises Language."

I love your suggestions regarding splitting out parts of CFEngine into modules.

[Neil Watson]

I do not want to see yet another CM syntax and product unless that
syntax is designed and offered as an open standard, not ruled or
controlled by any for-profit corp, that any CM product can use. The
selfish silos of open source CM have hurt us all.

Many orgs are not happy with their first choice of CM tool, but the cost
of rewriting all their policy to a new one keeps them where they are.

--
Neil H Watson @neil_h_watson
CFEngine reporting: https://github.com/neilhwatson/delta_reporting
CFEngine policy: https://github.com/neilhwatson/evolve_cfengine_freelib
CFEngine and vim: https://github.com/neilhwatson/vim_cf3

[Ted Zlatanov]

On Tue, 29 Nov 2016 09:10:27 -0500 Neil Watson <cfen...@watson-wilson.ca> wrote:

NW> I do not want to see yet another CM syntax and product unless that
NW> syntax is designed and offered as an open standard, not ruled or
NW> controlled by any for-profit corp, that any CM product can use. The
NW> selfish silos of open source CM have hurt us all.

NW> Many orgs are not happy with their first choice of CM tool, but the cost
NW> of rewriting all their policy to a new one keeps them where they are.

I agree with both Mike and Neil. But Neil's point applies to our entire
industry.

An open standard for CM tools will grow the market by creating an open
community that can share expertise, modules, and tools. The history of
many other industries shows this.

Ted

[joaqu...@gmail.com]

I hope I am not too late to add my two pence.  I am inspired by your enthusiasm, passion, and insight, and hope to contribute thoughtful discourse on this.

In creating a successful software product, capability VS viability VS usability/desirability should be considered [1].  With out a balance, the greatest technology in the world can fail.

A mathematical model alone is not going to make a product a success, unless it can predict the market itself.

Thus the capability (implementation of promise theory) must be measured against viability (use cases and meeting business needs) and both of these are measured against desirability/usability (do users derive pleasure or frustration in using a product).

In conversations at convention, such as RedHat Summit 2016, CFEngine is treated like a dirty word.  Honestly, I heard boos and hisses, and not one person commented positively.  I felt that was unfair how people reacted, and I am not sure exactly their experience, but I can imagine is this has to do with desirability/usability factor, where CFEngine is often associated with complexity to use and get up to speed (lead time) vs the competition of other solutions; Puppet DSL or HashCorp or YAML (Salt, Ansible, Hiera) are far easier to master.  CFEngine DSL is not all that intuitive, especially to a new comer.

So the question is, do people derive pleasure or frustration with CFEngine?  Usability testing can elicit this, by having set of use cases, and measuring how quickly a user can accomplish the task with tool X, tool Y, or tool C.  From there, you can adjust the product to close the gap, and design an experience where users will want to use the product.

The actual use cases will represent the business/customer/market requirements.  A large part of the market are moving away from convergence style configuration (snowflake servers[2]) to congruence style configuration[3].  This means that immutable production (phoenix servers[4]) with containers is a big game changer, and with a push to microservice architecture [5] or other cluster solutions, containers make sense.  The cost savings (ROI, CBA) are enormous for immutable production patterns, and thus this is a game changer, as much as digital film vs. chemical film.  As a metaphor, CF Engine is still addressing chemical film (runtime state convergence) as the market shifts to digital film (deploy time config, immutable stateless containers).

On top of this shift to immutable production with containers, there is rising popularity to apply convergence to managing not just systems themselves (or networking equipment) but also applying it to resources exposed through RESTful APIs.  With this, you can not only configure CI tasks and builds with Jenkins or TeamCity, and integration with code repos (GitHub, GitLab) and chat ops (Slack, Jabber), you can configure the entire infrastructure like AWS, GCE, Azure, and others.  This style of IaC (Infrastructure as Code), especially when combined with ImP (Immutable Production) has tremendous value add, both in savings and capability (flexibility, short lead time, cloning data centers), and increases innovation[6].  

But alas for both ImP or IaC, CFEngine does not have an answer.  This is viability.

As an analogy, I feel like CF Engine is the dream car, but assembly is required, but some parts, like transmission you may have to build it yourself.  Once assembled, there's two stick shifts.  This is instead of fully assembled with automatic shifting.

Other solutions offer the fully assembled car with not only automatic shifting, but all kinds of amenities on top of it, which allows shorter lead time to craft solutions that for engineering best practices and patterns using microservices and 12-factor applications [7].

For examples, Ansible is intuitive for deploy, orchestration, Docker, and Cloud Provisioning (GCE, AWS, etc.). Terraform focuses only on RESTful resources, and has full robust DSL for AWS, Azure, GCE, Jenkins, GitHub and more. The crossroads of tools from HashiCorp, Ansible, and others is truly amazing.  On container orchestration, schedulers, and service discovery, there's a whole new set of patterns, and solutions, but one that stands out is Habitat, which does this using promise theory.  Docker has InfraKit that on the surface might do similar things, and there are numerous custom solutions out there using a mix of Consul (async agent) and Nginx (routing, health check).

CF Engine could do all of this, but it would be a lot of work to expose such configurable resources, get state from them, and achieve idempotence.  It would take 2 years R&D (~ $500K+) work to build the same tooling using CF Engine in order to catch up to what the current market is at with other tools out there.  Even if that could be achieve, so that product is more viable, there's still a question on desirability/usability.  As for capability, I hope I implicitly addresses that, there's not doubt that CF Engine has that, but again, is that enough.

Thank you for reading,

Joaquin Menchaca

Notes/Imagination:
[1] Capability, Viability, Desirability is Larry Keeley’s three qualities for high tech products presented in Alan Cooper’s classic book: The Inmates Are Running the Asylum: Why High Tech Products Drive Us Crazy and How to Restore the Sanity (2004)
[2] Snowflake server mentioned Gene Kim’s The Visible Ops Handbook: Implementing ITIL in 4 Practical and Auditable Steps
[3] Divergence, Convergence, Congruence is from Why Order Matters: Turing Equivalence in Systems Administration is from article in USENIX publication by Stephen Gordon Traugott (2002)
[4] Phoenix Server by Martin Fowler - http://martinfowler.com/bliki/PhoenixServer.html
[5] Microservices by Martin Fowler - http://www.martinfowler.com/articles/microservices.html
[6] Imagine, using CFEngine to master entire pipeline (without calling the shell): build, functional/systems test, deploy pre-prod, integration/load testing, promote to production, with feedback in slack, email, github badges, metrics, deploying entire infrastructure (routers, switches, vlans, firewalls, disks, systems, DNS, and applications) testing it, then destroying it.
[7] 12-Factor Application https://12factor.net/

[Mike Weilgart]

JI> A mathematical model alone is not going to make a product a success,

JI> unless it can predict the market itself.

Indeed.  Factually I am more interested in the increase of application and use of Promise Theory in the entire configuration management industry, than I am in the specific success of the CFEngine product itself.  I do, however, believe the two are more closely related than you appear to think.

NW> I do not want to see yet another CM syntax and product unless that
NW> syntax is designed and offered as an open standard, not ruled or
NW> controlled by any for-profit corp, that any CM product can use. The
NW> selfish silos of open source CM have hurt us all.

NW> Many orgs are not happy with their first choice of CM tool, but the cost
NW> of rewriting all their policy to a new one keeps them where they are. 

TZ> I agree with both Mike and Neil. But Neil's point applies to our entire
TZ> industry.

TZ> An open standard for CM tools will grow the market by creating an open
TZ> community that can share expertise, modules, and tools. The history of
TZ> many other industries shows this.

I couldn't agree more.  This relates to my point above in response to Joaquin.

If a language is:

1. well designed,

2. from an open standard,

3. is simple to learn, and

4. has a means of expression for the actual use cases to which it is intended to be applied,

...it has a greater chance of being widely adopted than a closed language which changes based on the whims of the few developers of the sole parser for that language.

I would like to see such an open CM language that is based on Promise Theory.  I think that could potentially do as much for the long-term relevance of configuration management as Relational Theory has done for databases.

I would also like to avoid the departures from the underlying theory which were built in to SQL.  SQL is close enough to the workable underlying theory that it has been a remakable success in terms of adoption, but it is not a relational language.  There are a great many factual reasons why that is, and a great many historical reasons why it came about.  (If you're interested, review the works of C. J. Date.)  In my opinion, the non-relationality of SQL is one of the reasons for the "success" of NoSQL databases.  By violating Relational Theory in important ways, SQL makes certain practical use cases very difficult, or impossible, which are very easily handled by pure Relational Theory—thus forcing users into other systems.  (I place "success" in quotes because I consider this success to be relatively ephemeral; I believe the popular NoSQL databases will none of them still be in use in 100 years, while Relational Theory will be.)

As CFEngine policy language was explicitly based on Promise Theory, it seems a logical starting point for envisioning a CM language which could handle any aspect of Promise Theory with accuracy.  (You may call me an idealist if you wish.)

The point is not that Promise Theory must be regarded as holy.  The point is that if holes are found in the practical application of the theory, where it does not seem to apply, perhaps the theory could even be extended.  But first would come a reality check: Does the current implementation even apply Promise Theory?  Are we actually testing the applicability of the theory itself, or are we omitting parts of the theory that would handle the practical problems we are running into?

-------------

A comment on CFEngine 3 syntax:

In considering CFEngine 3's "compound promises" today, I realized something: There are no promise types in the CFEngine 3 policy language.

"What about 'files' and 'users' and 'packages,' then?  What are all those things, if they're not promise types?"

They are PROMISER types.

A "single files promise" can actually consist of multiple promises, as noted earlier in this thread.  However, what does that really mean?  It means that the same promiser (the file) has made multiple promises.

In a Promise Theory Language, this should be distinguished better.

Taking a look at an example of CFEngine syntax to represent promises for something other than server configuration, such as HTML—what do you see as the "promiser" and what do you see as a "promise"?

html:

 main::

  "h1"

    font_size => "12px",

        color => "red";

  "p"

    text_align => "justify";

I see two promisers—"h1" and "p"—and three promises.  What we call "attributes," I believe, were actually intended each to be distinct promises in the original CFEngine 3 syntax conception.

(Note: This example was written by Mark Burgess!)

This syntax seems to me extremely intuitive.  The devil, as they say, is in the [implementation] details.

Best,
—Mike Weilgart
Vertical Sysadmin, Inc.

--
You received this message because you are subscribed to the Google Groups "dev-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-cfengine...@googlegroups.com.
To post to this group, send email to dev-cf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dev-cfengine/08bd3ab5-4657-49d8-970b-3a8460ec4d73%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[joaqu...@gmail.com]

In my early days of Apple, I remember talking about things like "good implementation of a bad idea" and "bad implementation of a good idea".  In this case, Promise Theory is the good idea.

What do you think about Promise Theory outside the scope of just pure CM?

Examples

• Container Choreography
• Habitat - uses promise theory in bash (maybe not most intuitive, but with keep it simple w/ bash variables approach)
  
• https://techcrunch.com/2016/06/14/chefs-new-habitat-project-wants-to-make-applications-infrastructure-independent/
• https://www.habitat.sh/docs/continuous-deployment-overview/
• InfraKit (https://github.com/docker/infrakit/) - seems that aspects are Promise Theory.
  
• Service Discovery
• Consul (https://www.consul.io/) - with consul agent, can implement Promise Theory w/ dyanmic async distributed config, though how that config happens is up to implementer.
  
• JavaScript:
• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
• https://davidwalsh.name/promises
• https://www.toptal.com/javascript/javascript-promises

It would be interesting to see a DSL parser in Go and Rust as open source as these languages are becoming popular for distributed computing trends in cloud computing, especially with microservices.  If I had the skills of compiler designer, I may be bold to implement it.

Could HashiCorp HCL (https://github.com/hashicorp/hcl) be used to represent promises?  They use it for their Terraform product (https://www.terraform.io/), which is a CM but strictly for RESTful API, like AWS or GCE.

[Ted Zlatanov]

On Tue, 29 Nov 2016 22:33:51 -0800 (PST) joaqu...@gmail.com wrote:

ji> In conversations at convention, such as RedHat Summit 2016, CFEngine is
ji> treated like a dirty word.

It's good to note here that RedHat owns Ansible since summer 2016. But
generally I agree.

ji> CFEngine is often
ji> associated with complexity to use and get up to speed (lead time) vs
ji> the competition of other solutions; Puppet DSL or HashCorp or YAML
ji> (Salt, Ansible, Hiera) are far easier to master. CFEngine DSL is not
ji> all that intuitive, especially to a new comer.

This is one of the reasons a common CM language would be valuable. There
is no way to write a popular CM tool currently without also inventing a
new vendor ecosystem, since there is no sharing of techniques between
the vendors. Maybe I want to use CFEngine with a Puppet module? "No,
that's just not how it's done."

ji> As an analogy, I feel like CF Engine is the dream car, but assembly is
ji> required, but some parts, like transmission you may have to build it
ji> yourself. Once assembled, there's two stick shifts. This is instead of
ji> fully assembled with automatic shifting.

To continue the car analogy, currently driving every CM car requires
retraining drivers, special wheels, and separate lanes for every vendor.
That's clearly a ridiculous situation, but it's typical of young
industries such as ours.

Currently it's really hard to write a single CM management policy to do
everything you need, except for trivial cases. So people end up with 40%
tool X and 50% tool Y and 10% done by hand or in golden images. This has
technical costs such as rewriting policy, supporting multiple modules,
and testing multiple tools on multiple platforms. But the technical
costs are a small part of the overall costs.

This knowledge fragmentation results in misunderstandings, complexity,
conflicts, political battles in the organizations, and security
problems. These costs are partly why customers are moving away from CM
tools in general.

Ted

[lesley....@gmail.com]

Hello all;

I apologize for breaking into this conversation but this is one of the few conversations I found discussing CM tools in the context of Promise Theory.

I have just been looking into various CM tools (currently use Puppet) and started reading about CFEngine and Promise Theory. I after reading a few articles and watching a few videos I'm still not sold or not fully understanding the implication of Promise Theory. Here is the text from a post I made on one article. Please clear up any misunderstandings or incorrect statements that I may have made:

The idea of each actor or node being autonomous seems somewhat farfetched in reality or an abstraction at best.

The theory is explained such that these actors make their promises and coordinate amongst themselves. It is differentiated from obligation in that the actors supposedly can't be 'told' what to do. However, even though such systems are written in a declarative style these are still effectively  'instructions' and not 'requests'. The actor really doesn't have a choice of whether to execute them or not. Unless, of course, the particular node doesn't have a relevant operation to support the request.

Furthermore, it seems to me, that the distinction between imperative and declarative is fuzzy. Yes, we write our configuration in declarative language but ultimately the agent on the node implements our declarative requests using predefined imperative routines. The actors have no true intelligence other than being written specifically for a particular platform and executing routines that are relevant to those platforms. The declarative style is, therefore, simply an abstraction that prevents us from having to write instructions toward a specific architecture.

Probably the use of agent-based solutions lends itself to better scaling but I feel that this trying to fit the operation of it all into the fancy formalized box with specialized dialect just over-complicates things for the average administrator.

[joaqu...@gmail.com]

On Declarative vs Imperative:
=====================

Everything is imperative ultimately because it runs as machine language, which are byte opcodes running step-by-step imperatively. Declarative is well suited to change config as it abstracts complexity system specific implementation, also it clearly communicates desired behavior to a broader audience. That is the concept.

All popular change configurations use declarative with exception of Chef, which uses a pseudo-DSL in Ruby. I like declaritive as it creates more readible/portable code, where pure ruby for example, invites complexity, as the change config becomes optional at this point, e.g. file resource from change config API vs file API in ruby.

As for CFEngine, while the concept is sound, the implementation may be another story. In otherwords, maybe it is not the most intuitive implementation of declarative.

In practice (reality), a little of both is needed. Puppet added a lot of imperative pieces to their DSL making more complex and specialized, defeating original design of puppet IMHO. They may have had no choice. Chef maybe has and advantage in this regard as they started with imperative, and promote community style guides and best practice to encourage readibility and portability.

Obligation vs Promise
================
This is a broad topic, that applies to other systems beyond change config, such as service discovery or JavaScript API mechanism. Typically it is implemented with an agent on the client.

For, traditional change config, where change is applied at runtime, agent system is optimal to have client meet its promise. The agent can converge to a desired state, and thus have a limited self-healing capability.

For change applied at build or deploy time, such as immutable production with Docker, complex centralized change config become less necessary. Push configuration system (obligation) become more optimal, e.g. Ansible.

In immutable production scenarios with Docker, obligation is very popular, either rolling your own custom one with Ansible or other tool, or using an orchestrator like Mesos/Marathon (DC/OS), Kubernetes, or Swarm. However, there is a point where promise theory may become more optimal, likely around 10,000+ system. Habitat is one solution that uses promise theory for containers; they call it choreography vs orchestration (obligation).

So, promise is optimal for runtime config changes, obligation is optimal for < 10K for immutable prod (build time or deploy time config changes) , promise optimal for > 10K+ w. Immutable.

There has been the real world self-healing and auto-scaling archectures using custom or orchestration (obligation) system like Kubernetes. So this is not far fetched. Other tools like service discovery (zookeeper, etcd, consul) along may play a role to coordinate live active state. This would then be a mix or promise and obligation.

True self-healing cannot happen easily with current batch of centralized change config tools, as they may not have the current source of truth, for it is stored or queried from a intemediary source.

How all of this turns out in practice, is in a state of flux, only community interests and marketplace demands will determine how this plays out.