New major release: CFEngine 3.7.0 LTS

[Dimitrios Apostolou]

Dear CFEngine Community,

we are proud to announce today a major new release: CFEngine 3.7.0 LTS
(see [1] about our new LTS release schedule that guarantees a lifetime
of 3 years).

[1] http://cfengine.com/company/blog-detail/releases-after-3-7-0/

CFEngine 3.7.0 brings a number of features and improvements that have
been in development for more than a year. New packages promise, YAML
support, TLS encrypted network traffic as default, new switch
--trust-server=no for easily bootstrapping over untrusted networks,
new "@if minimum_version(x.x)" syntax for backwards compatible
policy, configurable bandwidth limiting, many new
functions, and much more! As usual a vast number of bug fixes have been
applied as well.

A big "thank you" goes to the numerous volunteers that contributed new
features, bugfixes, and have tested the Beta, this release wouldn't have
been possible without you!

Read more details on the features of the release in the blog post:

[2] http://cfengine.com/company/blog-detail/cfengine-3-7-0-released-new-package-promise-and-change-reporting/

Please take special care regarding some backwards-incompatible
changes. In particular:

* Arbitrary arguments to cfruncommand (using "cf-runagent -o") are
not acceptable any more. (Redmine #6978)

* Bootstrapping 3.7 agents to 3.5 or earlier hubs no longer works
by default, as TLS is being used to secure the session

* 3.4 is no longer supported in masterfiles.

* Masterfiles source tarball now installs in the usual UNIX way, for
example to install in "/path/to/masterfiles", run:

./configure --prefix=/path/to && make install

* If you are installing from our RPM or APT repositories, make sure you
import our new GPG key from [3]. Detailed instructions are at [4].

[3] https://cfengine.com/pub/gpg.key
[4] http://cfengine.com/product/community/cfengine-linux-distros/


You can download the packages from [5], and as always verify the
correctness of you download by comparing the SHA1 hashes.

[5] http://cfengine.com/product/community/

c9d6b5c1c890b6dbf875cd18b2805cf93848ba78 cfengine-3.7.0.tar.gz
45b009e132b475f71f1dab60928afbbff87b0c14 cfengine-masterfiles-3.7.0.tar.gz
d643a26cec139f28f61e041efdbd9b9cca9d6f72 cfengine-community_3.7.0-1_amd64.deb
34d320b748c58e2a3d3cfd58a194bb91f2856631 cfengine-community_3.7.0-1_i386.deb
855efe6451e8dfa72030ef6e10edb64797927f46 cfengine-community-3.7.0-1.x86_64.rpm
58442b23cb55793c03302fbf99f38bbaeb533936 cfengine-community-3.7.0-1.el6.x86_64.rpm
ea3c4624935a8daaccc08a281707e9f956162d25 cfengine-community-3.7.0-1.i386.rpm


For completeness, I'm appending the list of new features. Even more
details about changes and bugfixes you can find in our changelog:

[6] https://github.com/cfengine/core/blob/3.7.0/ChangeLog


3.7.0:
New features:
- New package promise implementation.
The syntax is much simpler, to try it out, check out the syntax:
packages:
"mypackage"
policy => "absent/present",

# Optional, default taken from common control
package_module => apt_get,

# Optional, will only match exact version. May be
# "latest".
version => "32.0",

# Optional.
architecture => "x86_64";

- Full systemd support for all relevant platforms
- New classes to determine whether certain features are enabled:
* feature_yaml
* feature_xml
For the official CFEngine packages, these are always enabled, but
packages from other sources may be built without the support.
- New readdata() support for generic data input (CSV, YAML, JSON, or auto)
- YAML support: new readyaml() function and in readdata()
- CSV support: new readcsv() function and in readdata()
- New string_mustache() function
- New data_regextract() function
- eval() can now be called with "class" as the "mode" argument, which
will cause it to return true ("any") if the calculated result is
non-zero, and false ("!any") if it is zero.
- New list_ifelse() function
- New mapjson() function as well as JSON support in maparray().
- filestat() function now supports "xattr" argument for extended
attributes.
- "ifvarclass" now has "if" as an alias, and "unless" as an inverse
alias.
- Ability to expand JSON variables directory in Mustache templates:
Prefix the name with '%' for multiline expansion, '$' for compact
expansion.
- Ability to expand the iteration *key* in Mustache templates with @
- Canonical JSON output: JSON output has reliably sorted keys so the
same data structure will produce the same JSON every time.
- New "@if minimum_version(x.x)" syntax in order to hide future language
improvements from versions that don't understand them.
- compile time option (--with-statedir) to
override the default state/ directory path.
- Fix error messages/ handling in process signalling which no longer
allowed any signals to fail silently
- Also enable shortcut keyword for cf-serverd classic protocol, eg to
simplify the bootstrap process for clients that have different
sys.masterdir settings (Redmine #3697)
- methods promises now accepts the bundle name in the promiser string,
as long as it doesn't have any parameters.
- In a services promise, if the service_method bundle is not specified,
it defaults to the promiser string (canonified) with "service_" as a
prefix. The bundle must be in the same namespace as the promise.
- inline JSON in policy files: surrounding with parsejson() is now
optional *when creating a new data container*.
- New data_expand() function to interpolate variables in a data container.
- Add configurable network bandwidth limit for all outgoing
connections ("bwlimit" attribute in "body common control") . To
enforce it in both directions, make sure the attribute is set on both
sides of the connection.
- Secure bootstrap has been facilitated by use of
"cf-agent --boostrap HUB_ADDRESS --trust-server=no"
- Implement new TLS-relevant options (Redmine #6883):
- body common control: tls_min_version
- body server control: allowtlsversion
- body common control: tls_ciphers
- body server control: allowciphers (preexisting)



As always, your feedback is much appreciated.

Enjoy!
Dimitrios Apostolou
CFEngine AS

[Aleksey Tsalolikhin]

Congratulations and well done!  Keep up the good work.

--

Need CFEngine training?  Email trai...@verticalsysadmin.com

[Brian Bennett]

It looks like cfengine-masterfiles-3.7.0.tar.gz contents are wrong. The previous version, masterfiles-3.6.5.tar.gz was a complete set of masterfiles that could be dropped into place.

The current cfengine-masterfiles-3.7.0.tar.gz doesn't contain the full set of masterfiles (lib, services and templates are missing). It doesn't even have autoconf.sh, so it's not even a "buildable" distribution.

I would actually be willing to skip 3.8 entirely and wait an extra six months so you guys could get your release engineering fixed. I'm sorry to say, but it's really embarrassing. Your release process is just as important as the actual code distribution.

--
Brian Bennett
Looking for CFEngine training?
http://www.verticalsysadmin.com/

[Ted Zlatanov]

On Sat, 18 Jul 2015 09:09:31 -0700 Brian Bennett <brian....@verticalsysadmin.com> wrote:

BB> It looks like cfengine-masterfiles-3.7.0.tar.gz contents are wrong.
BB> The previous version, masterfiles-3.6.5.tar.gz was a complete set of
BB> masterfiles that could be dropped into place.

BB> The current cfengine-masterfiles-3.7.0.tar.gz doesn't contain the full
BB> set of masterfiles (lib, services and templates are missing). It
BB> doesn't even have autoconf.sh, so it's not even a "buildable"
BB> distribution.

Maybe it's enough to use
https://github.com/cfengine/masterfiles/archive/3.7.0.zip which reflects
the 3.7.0 tag.

BB> I would actually be willing to skip 3.8 entirely and wait an extra six
BB> months so you guys could get your release engineering fixed. I'm sorry
BB> to say, but it's really embarrassing. Your release process is just as
BB> important as the actual code distribution.

I don't see it that way. The binary packages for 3.6 and 3.7 have been
fine, and that's what matters to the vast majority. I think with their
currently constrained resources, the CFEngine team is doing great work.

Ted

[Brian Bennett]

The world is not just RedHat and Debian. RedHat and Debian themselves also need to repackage from source. It matters. How hard is it to tar up some usable files?

And can you explain to me exactly why this is somehow my fault or problem? What exactly didn't I do right that got me into this situation? Why it isn't on the CfEngine release team to fix?

I get that they have limited resources. But so do I, which is why I don't have time to go chasing around amateur mistakes. And this is an amateur mistake.

-- 

Brian Bennett

Looking for CFEngine training?

http://www.verticalsysadmin.com/

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
To post to this group, send email to help-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/help-cfengine.
For more options, visit https://groups.google.com/d/optout.

[Ted Zlatanov]

On Sat, 18 Jul 2015 11:56:59 -0700 Brian Bennett <brian....@verticalsysadmin.com> top-posted:

>> On Jul 18, 2015, at 11:05 AM, Ted Zlatanov <t...@lifelogs.com> wrote:
>>
>> On Sat, 18 Jul 2015 09:09:31 -0700 Brian Bennett <brian....@verticalsysadmin.com> wrote:

BB> I would actually be willing to skip 3.8 entirely and wait an extra six
BB> months so you guys could get your release engineering fixed. I'm sorry
BB> to say, but it's really embarrassing. Your release process is just as
BB> important as the actual code distribution.
>>
>> I don't see it that way. The binary packages for 3.6 and 3.7 have been
>> fine, and that's what matters to the vast majority. I think with their
>> currently constrained resources, the CFEngine team is doing great work.

BB> The world is not just RedHat and Debian. RedHat and Debian
BB> themselves also need to repackage from source. It matters.

I'm not saying it doesn't matter. The vast majority of users, however,
don't care, so this is much less embarassing than a broken binary
package. My disagreement is, therefore, with the strength of your
statement rather than with the truth of it.

BB> And can you explain to me exactly why this is somehow my fault or
BB> problem? What exactly didn't I do right that got me into this
BB> situation? Why it isn't on the CfEngine release team to fix?

I can't explain statements I didn't make.

BB> I get that they have limited resources. But so do I, which is why I
BB> don't have time to go chasing around amateur mistakes. And this is
BB> an amateur mistake.

I'd love to know what you consider a professional mistake.

Ted

[Dimitrios Apostolou]

Hi Brian,

On Sat, Jul 18, 2015 at 6:09 PM, Brian Bennett
<brian....@verticalsysadmin.com> wrote:
> It looks like cfengine-masterfiles-3.7.0.tar.gz contents are wrong. The previous version, masterfiles-3.6.5.tar.gz was a complete set of masterfiles that could be dropped into place.
>
> The current cfengine-masterfiles-3.7.0.tar.gz doesn't contain the full set of masterfiles (lib, services and templates are missing). It doesn't even have autoconf.sh, so it's not even a "buildable" distribution.
>
> I would actually be willing to skip 3.8 entirely and wait an extra six months so you guys could get your release engineering fixed. I'm sorry to say, but it's really embarrassing. Your release process is just as important as the actual code distribution.

we've been trying to improve and automate the release process more and
more with each release. Thus, the latest masterfiles tarball is a
traditional source tarball, generated automatically using "make dist"
like every most other linux . As I wrote in the release email:

* Masterfiles source tarball now installs in the usual UNIX way, for
example to install in "/path/to/masterfiles", run:

./configure --prefix=/path/to && make install

Please let me know if you have any further trouble.

Dimitris

[Nick Anderson]

On 07/20/2015 09:27 AM, Dimitrios Apostolou wrote:
> ./configure --prefix=/path/to && make install

Most commonly:

./configure --prefix=/var/cfengine && make install

I did not see that lib was missing after running this.

[Brian Bennett]

*sigh*. Yes, I've already tried this. It *also* doesn't work.

$ ./configure --prefix=/var/cfengine
configure: error: cannot run /bin/sh ./config.sub

--
Brian Bennett
Looking for CFEngine training?
http://www.verticalsysadmin.com/

[Dimitrios Apostolou]

On Mon, Jul 20, 2015 at 6:06 PM, Brian Bennett
<brian....@verticalsysadmin.com> wrote:
> *sigh*. Yes, I've already tried this. It *also* doesn't work.
>
> $ ./configure --prefix=/var/cfengine
> configure: error: cannot run /bin/sh ./config.sub

Strange, I had no problems with that. What it your OS? Maybe you can
send me config.log? Is the cfengine-3.7.0 source tarball having the
same problems in your system?


Dimitris

[Brian Bennett]

cfengine-3.7.0 worked fine.

See https://gist.github.com/bahamat/f2c61cee2f682ce5b84e for logs.

--
Brian Bennett
Looking for CFEngine training?
http://www.verticalsysadmin.com/

[Dimitrios Apostolou]

On Mon, Jul 20, 2015 at 8:05 PM, Brian Bennett
<brian....@verticalsysadmin.com> wrote:
> cfengine-3.7.0 worked fine.
>
> See https://gist.github.com/bahamat/f2c61cee2f682ce5b84e for logs.

Thanks, but the very same tarball gives different directory contents
on my system, attached is the listing from "tar -tzf". What about the
SHA1 hash of the tarball and the size?

$ sha1sum cfengine-masterfiles-3.7.0.tar.gz
45b009e132b475f71f1dab60928afbbff87b0c14 cfengine-masterfiles-3.7.0.tar.gz

$ wc -c cfengine-masterfiles-3.7.0.tar.gz
441467 cfengine-masterfiles-3.7.0.tar.gz


Dimitris

[Brian Bennett]

For those of you playing along at home, I have the same byte count, same sha1 sum.

This caused me to start digging a big deeper which ultimately led to a discussion on IRC and some internal investigation at Joyent. As it turns out, the behavior I'm seeing is caused by a bug in GNU tar which was used to package the file on Debian 4 and revealed by Sun tar on illumos. With file paths over 100 characters (which this archive contains), the version of GNU tar in Debian 4 creates a tarball that has a record where the name field is empty (first character is NUL), but the prefix field is populated. This is causing endtape() to believe we have reached the end of the archive, and extraction finishes. Joyent will be updating Sun tar shipped with illumos to account for this and continue if possible. However, this won't cover Oracle Solaris and the fix is 1-3 weeks away on SmartOS before getting upstreamed to illumos.

I've filed https://dev.cfengine.com/issues/7407 about this issue. It's probably best to replace the cfengine-masterfiles source tarballs for both 3.6.6 and 3.7.0 with files created with a newer version of GNU tar.

So I'd like to apologize to everyone. This is not in any way an amateur mistake. This is definitely a professional quality mistake ;-)

--
Brian Bennett
Looking for CFEngine training?
http://www.verticalsysadmin.com/

> <cfengine-masterfiles-3.7.0.list.txt>

[Dimitrios Apostolou]

On Fri, Jul 17, 2015 at 9:44 PM, Dimitrios Apostolou <ji...@cfengine.com> wrote:
> c9d6b5c1c890b6dbf875cd18b2805cf93848ba78 cfengine-3.7.0.tar.gz
> 45b009e132b475f71f1dab60928afbbff87b0c14 cfengine-masterfiles-3.7.0.tar.gz

NOTE: For anyone having trouble unpacking the source tarballs - most
likely Solaris users - we have uploaded almost identical tarballs, the
only difference is that they are packed with newer version of GNU tar.

https://cfengine-package-repos.s3.amazonaws.com/tarballs/cfengine-3.7.0-2.tar.gz
https://cfengine-package-repos.s3.amazonaws.com/tarballs/cfengine-masterfiles-3.7.0-2.tar.gz

e69828a962cad396cc504e9a419edec98c8d16cb cfengine-3.7.0-2.tar.gz
8fdb7317d57f0f39407aaac0b27986d314c1cf0b cfengine-masterfiles-3.7.0-2.tar.gz


Regards,
Dimitris

[Neil Watson]

On Mon, Jul 20, 2015 at 04:38:33PM -0700, Brian Bennett wrote:
>
>So I'd like to apologize to everyone. This is not in any way an amateur
>mistake. This is definitely a professional quality mistake ;-)

I think your rant still has validity. Those packages were published
without testing. A less patient user or a new user may have given up and
gone to another product.

--
Neil H Watson
Sr. Partner, Architecture and Infrastructure
CFEngine reporting: https://github.com/evolvethinking/delta_reporting
CFEngine policy: https://github.com/evolvethinking/evolve_cfengine_freelib
CFEngine and vim: https://github.com/neilhwatson/vim_cf3
CFEngine support: http://evolvethinking.com

[Dimitrios Apostolou]

On Tue, Jul 21, 2015 at 7:05 PM, Neil Watson <cfen...@watson-wilson.ca> wrote:
> On Mon, Jul 20, 2015 at 04:38:33PM -0700, Brian Bennett wrote:
>>
>>
>> So I'd like to apologize to everyone. This is not in any way an amateur
>> mistake. This is definitely a professional quality mistake ;-)
>
>
> I think your rant still has validity. Those packages were published
> without testing. A less patient user or a new user may have given up and
> gone to another product.

Everything was manually tested on various platforms, including the
tarballs. Granted, it seems we missed unpacking the source tarball on
Solaris, using native tar.


Dimitris

[Ted Zlatanov]

"The nice thing about [tar] standards is that you have so many to choose
from." --Andrew Tar-nenbaum

Ted

[Joaquin Menchaca]

I guess this can be added to regression tests. :)