OpenWrt CFEngine 3.5.3 packages

[csdi...@gmail.com]

Some background:

I've been experimenting with CFEngine (3.5.3) on constrained embedded devices: x86, ARM, and MIPS; flash storage; between 8mb and 32mb RAM. Deployed on both physical devices and as kvm/qemu guests.
 
My physical tinker target of choice has been a MIPS-based TL-WR1043ND.

I've created a Makefile for the OpenWrt build system that will create a CFEngine packge for Backfire, Attitude Adjustement, and current; in general, any of these OpenWrt versions and any of their build targets that, for now, have a tokyocabinet package available.

The goal is to submit this work upstream and make available|maintain a CFEngine 3.5.3 package for OpenWrt.


Now the questions:

I've read the tiny cfengine thread and understand wear-leveling|tear, or lack thereof, and the general role of qdbm|tokyocabinet|lmdb in CFEngine but severely lack details...

1) how absolutely dependent is cfengine on one of these embedded databases?
    eg. if I was willing to touch the core, adjust the build system to account for openwrt, ifdef/stub-out the embedded db portions, etc... is there a possibility to decouple cfengine from these dependicies? If achievable, the impact and general level of effort?

2) what are the ramifications of placing these db+lock files on a tmpfs|ramfs? aka when they disappear on reboot or potentially run out of space, how does that affect cfengine and what are some types of information that are lost?
  

I'd be willing to have a stab at making the locations of the state and log files configurable... thankflly the OpenWrt syslogd is implemented as a in-ram ringbuffer so no need to worry about that.

Any info or suggestions would be appreciated.
 

[Ted Zlatanov]

On Wed, 15 Jan 2014 13:07:45 -0800 (PST) csdi...@gmail.com wrote:

c> *Now the questions:*I've read the tiny
c> cfengine<https://groups.google.com/forum/#!topic/help-cfengine/ixnyfXd3Z4c>thread
c> and understand wear-leveling|tear, or lack thereof, and the general
c> role of qdbm|tokyocabinet|lmdb in CFEngine but severely lack details...

c> 1) how absolutely dependent is cfengine on one of these embedded databases?
c> eg. if I was willing to touch the core, adjust the build system to
c> account for openwrt, ifdef/stub-out the embedded db portions, etc... is
c> there a possibility to decouple cfengine from these dependicies? If
c> achievable, the impact and general level of effort?

I believe the CFEngine core is completely dependent on some kind of
embedded database and LMDB will be the only choice for 3.6.0, but I
don't know that area well. Please make a new ticket and also look at
https://cfengine.com/dev/issues/3828 where the ability to switch backend
databases is mentioned.

c> 2) what are the ramifications of placing these db+lock files on a
c> tmpfs|ramfs? aka when they disappear on reboot or potentially run out of
c> space, how does that affect cfengine and what are some types of information
c> that are lost?

You lose promise locks and persistent classes, among others. I don't
think you lose the bootstrap information.

c> I'd be willing to have a stab at making the locations of the state and log
c> files configurable... thankflly the OpenWrt syslogd is implemented as a
c> in-ram ringbuffer so no need to worry about that.

I recall at least some of these are already configurable. Take a look
at https://github.com/cfengine/core/pull/1293 for starters.

Ted

[csdi...@gmail.com]

Thanks for the follow up Ted.

On Thursday, January 16, 2014 5:02:45 AM UTC-6, Ted Zlatanov wrote:

On Wed, 15 Jan 2014 13:07:45 -0800 (PST) csdi...@gmail.com wrote:

t> I believe the CFEngine core is completely dependent on some kind of
t> embedded database and LMDB will be the only choice for 3.6.0, but I
t> don't know that area well.  Please make a new ticket and also look at
t> https://cfengine.com/dev/issues/3828 where the ability to switch backend
t> databases is mentioned.

 
Not really all that surprised but too bad it's impossible to detangle core from an embedded database. Great to hear that it'll only be LMDB going forward though.

Thanks for the reference btw.

t> You lose promise locks and persistent classes, among others.  I don't

t> think you lose the bootstrap information.

Ok cool. I'll have to test this... losing bootstrap wouldn't be ideal but can be worked around.

 

I recall at least some of these are already configurable.  Take a look
at https://github.com/cfengine/core/pull/1293 for starters.

I've seen the pull Bas issued but I don't recall seeing a way to achieve everything I require with the current compile time options; I might have missed something though.

In a nutshell the following need to always be stored somewhere in tmpfs (/tmp and /var on openwrt)... I'll figure out a way to rotate/trash so as not to fill up ram:

*.log, *.lock, ./state/ and any embedded dbs, ./outputs/, ./reports/, and anything that gets written frequently.

The rest you'd normally expect to find in /var/cfengine/ will be in /etc/cfengine/.

The aim is to provide a CFEngine experience without the end-user having worry about tearing down their flash... eg out-of-box binaries should cause all writes to be committed only to ram (within what's possible).

I'll be up to the user to completely ignore the supplied warnings and proceed to write and run some absurd policy to kill their device :)


-Chris

[Ted Zlatanov]

On Fri, 17 Jan 2014 14:44:26 -0800 (PST) csdi...@gmail.com wrote:

I don't think you'll lose the bootstrap, sorry if I was unclear. You'll
lose the other things like promise locks.

c> I've seen the pull Bas issued but I don't recall seeing a way to achieve
c> everything I require with the current compile time options; I might have
c> missed something though.

It covers some but not all. Please request the things not covered;
setting the WORKDIR should be sufficient in general.

c> In a nutshell the following need to always be stored somewhere in tmpfs
c> (/tmp and /var on openwrt)... I'll figure out a way to rotate/trash so as
c> not to fill up ram:

c> *.log, *.lock, ./state/ and any embedded dbs, ./outputs/, ./reports/, and
c> anything that gets written frequently.

c> The rest you'd normally expect to find in /var/cfengine/ will be in
c> /etc/cfengine/.

That's really vague, but I think you mean the bindir, inputdir, and
masterdir? Or something else?

c> The aim is to provide a CFEngine experience without the end-user having
c> worry about tearing down their flash... eg out-of-box binaries should cause
c> all writes to be committed only to ram (within what's possible).

Yeah, of course. But could you use links (symbolic or not)?

I think this direction is particularly appropriate for CFEngine, the
code is ideal for constrained environments. I have a Raspberry Pi
sitting on my desk waiting for some love :)

c> I'll be up to the user to completely ignore the supplied warnings and
c> proceed to write and run some absurd policy to kill their device :)

Heh. CFEngine is not very supportive of that :)

Ted

[csdi...@gmail.com]

On Monday, January 20, 2014 9:17:43 AM UTC-6, Ted Zlatanov wrote:

On Fri, 17 Jan 2014 14:44:26 -0800 (PST) csdi...@gmail.com wrote:

t> I don't think you'll lose the bootstrap, sorry if I was unclear.  You'll
t> lose the other things like promise locks.

 

Ok cool. Let me know if you can think of anything else that would be lost.

Once everything is ready for prime time I'll drop a final post with a write-up of my experience, caveats, etc. Hopefully a link to an upstream package too! If not, upstream I'll paste the openwrt package makefile here... or maybe provide something more comprehensive in core/contrib/.

 

t> It [Bas' PR] covers some but not all.  Please request the things not covered;
t> setting the WORKDIR should be sufficient in general.

 

I've been playing with WORKDIR and the various compile time options but have yet to cherry-pick the changes Bas made. Going to bring in his changes and play around some more... still think I need a compile time option for the *.{log, embed db, lock} files though. If that is the case, I'll file the necessary requests in redmine.
 

t> That's really vague, but I think you mean the bindir, inputdir, and
t> masterdir?  Or something else?  

 

Sorry I was a bit rushed when I wrote that. You're above is correct:

bindir, inputdir, masterdir in /etc/cfengine/
cd /usr/bin; for i in /etc/cfengine/cf-*; do ln -s $i; done
*.{log,lock}, statedir, outputsdir, reportsdir, files frequently writen or volatile in /var/cfengine/ (tmpfs)

I'm probably forgetting something minor again but you get the idea.

t> Yeah, of course.  But could you use links (symbolic or not)?

Yessir. Symlinks for the binaries... perhaps some others if I can't get what I need--or implement--with the current compile time options.
 

t> I think this direction is particularly appropriate for CFEngine, the
t> code is ideal for constrained environments.  I have a Raspberry Pi
t> sitting on my desk waiting for some love :)

 

Me too! If you don't want to spend the cycles {native,cross}-compiling for ARM see here: https://github.com/cdituri/cfengine-raspberry-pi

There's a tarball in the toplevel: cfe has been working great on the pi.

I couldn't even fathom what the experience would be like with chef or puppet on the pi (salt, fabric, bcfg2, insert(product_x)... used them all)--let alone something like the TL-WR1043ND which, let's face it, would be impossible with the others. I'd be impressed if someone could prove me wrong on that last part though :)

t> Heh.  CFEngine is not very supportive of that :)

Lol :)

- Chris