External cache for fact values

[Hailee Kenney]

We are implementing caching in Facter, and are looking for feedback on how we've decided to do so. As further described in this ticket, we've deciding against an internal cache because it's really hard to get right in a concurrent environment. 

Our supported interface will be a simple yaml file on disk using only key/value pairs with out-of-band updates.

This will allow us to decouple the cache from the implementation of updating the cache, which will allow us to potentially transition to a daemonized version of Facter which can do piecemeal expiration of individual facts rather than replacement of the entire cache. The update method today might be cron, but it allows us some flexibility it how we do the updates.

This method has been proven effective in MCollective. 

We know we have some slow fact resolutions, this is not meant to bypass those, we'll still need to investigate and solve those issues.

--
Hailee Kenney

Development Intern 

Puppet Labs, Inc. 

hai...@puppetlabs.com

[Trevor Vaughan]

Just thinking out loud here but how about a daemonized process
(facterd?) that allows for fine grained retrieval of facts and solves
your concurrency issues with an internal mechanism?

Should facterd be unavailable, each process should be able to fall
back to a) gathering all facts or b) reading the cache file and
updating those items that have elapsed.

This way you end up with something like (for example):

os_version => '4 months'
uptime => 'no cache'
ps => 'persist'
etc...

The issue with some of these is that, while they won't change often,
when they change you *really* want to know. Like puppet_version and
ruby_version and possibly ipaddress.

A prod to the daemon with a USR1 (or whatever) should force a re-cache
of items selected as such.

Having a daemon process could also allow for asynchronous fact
collection via SNMP/REST/Whatever....

Those are my initial thoughts anyway.

Trevor

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To post to this group, send email to puppe...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-dev+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-dev?hl=en.



--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

[Jeff Weiss]

Trevor,

We've thought about a daemonized facterd, and it's in our vision. This is the first step along that path.  With Facter as both an executable and a library, it turns out to be really hard to be able to guarantee that a value will definitely be returned within a certain amount of time *and* if it takes longer than that reliably spawn something that will resolve the fact and store it in a cache. By pulling out the cache to a file, it defines a specific interface for those that need fast access to facts even though they may be slightly out-dated. With the file as the interface, we have the opportunity to create and improve a facterd without breaking people who are using the yaml cache.

So, short story even longer, we agree that you thoughts are roughly in-line with where we are going, but we need to take the first step, and this seems like a reasonable first step that will allow us some freedom to experiment without breaking users of the cache.

--Jeff

[R.I.Pienaar]

----- Original Message -----
> From: "Hailee Kenney" <hai...@puppetlabs.com>
> To: puppe...@googlegroups.com
> Sent: Wednesday, May 23, 2012 11:31:55 PM
> Subject: [Puppet-dev] External cache for fact values
>
>
> We are implementing caching in Facter, and are looking for feedback
> on how we've decided to do so. As further described in this ticket ,
> we've deciding against an internal cache because it's really hard to
> get right in a concurrent environment.
>
>
> Our supported interface will be a simple yaml file on disk using only
> key/value pairs with out-of-band updates.
>
>
> This will allow us to decouple the cache from the implementation of
> updating the cache, which will allow us to potentially transition to
> a daemonized version of Facter which can do piecemeal expiration of
> individual facts rather than replacement of the entire cache. The
> update method today might be cron, but it allows us some flexibility
> it how we do the updates.

can you give some more detail on how the cache will be used? If a fact
is found on disk via the rb file and there's nothing in the cache will
it then simply run the slow way? and update the cache?

sounds like there would be various chicken and egg situations with arranging
for pluginsync to have happened before attempts to build the cache so I
am looking to hear some more details to determine if that might be an issue.

There is a definite need for varying ttl times - totalmemory can have a long
ttl while something like EC2 facts might have a lower TTL

>
>
> This method has been proven effective in MCollective.

I wouldnt go so far as saying that :) generating fact caches via cronjob
or puppet writing out yaml files has been a recurring headache for people

[Chris Price]

On Wed, May 23, 2012 at 4:01 PM, Jeff Weiss <jeff....@puppetlabs.com> wrote:

<snip> 

With Facter as both an executable and a library, it turns out to be really hard to be able to guarantee that a value will definitely be returned within a certain amount of time *and* if it takes longer than that reliably spawn something that will resolve the fact and store it in a cache

<snip>

Jeff--could you elaborate on this a bit more?  If Facter is being used as both an executable and as a library, and that is leading to complications with the caching implementation, would refactoring to provide a cleaner separation between CLI and library usage alleviate any of the problems?

[Jeff Weiss]

On Wed, May 23, 2012 at 4:02 PM, R.I.Pienaar <r...@devco.net> wrote:

can you give some more detail on how the cache will be used? If a fact
is found on disk via the rb file and there's nothing in the cache will
it then simply run the slow way? and update the cache?

 Facter itself will not use the cache. If you have an application that needs facts and needs them quickly, you may read the yaml file on disk. An entirely separate process will update the cache. At this first step, the update process is planned to be a cron job, with hope of an actual facter daemon later.

 

sounds like there would be various chicken and egg situations with arranging
for pluginsync to have happened before attempts to build the cache so I
am looking to hear some more details to determine if that might be an issue.

At this time, we are not proposing Puppet use the external cache from the disk.

 

There is a definite need for varying ttl times - totalmemory can have a long
ttl while something like EC2 facts might have a lower TTL

Agreed. This first step has an effective TTL of the the period of the cron job. We're saying this is our ideal state, only the first step toward it.

 

> This method has been proven effective in MCollective.

I wouldnt go so far as saying that :) generating fact caches via cronjob
or puppet writing out yaml files has been a recurring headache for people

Tell me more, please.

 

[Jeff Weiss]

The inter-process reader/writer problem is only one portion.  If that's all it were, then, yes, we probably could refactor and make internal caching easier. However, we also have clients of Facter that have explicit performance expectations. For the sake of discussion let's say that performance expectation is that facts must return in less than 2 seconds. We could certainly institute a policy that after, say, 1.5 seconds if the facts haven't resolved then we'll read the cache and return.  This leads to tricky edge cases:

1) What happens when the resolution time is always > than policy time?

  a) how do you ever get the value?

  b) do you have spawn something that asynchronously updates the cache?

  c) who does that run as?

  d) does it actually have permission to write to the cache?

  e) this is fine for an executable, but is this proper behaviour for a library?  

2) Who's responsible for updating the cache?

 a) if you resolve inside policy time, do you write to the cache?

 b) what if you're a library and your process user doesn't have write permission?

These are all things that are better handled by a Facter daemon. We can specify the cached yaml file access API now and have a Facter daemon easily adhere to them while interactively being able to handle the above cases in a much more reliable manner.

[R.I.Pienaar]

sorry, It's not possible to reply sanely to your html email.

Thanks for clarifying, I think the word cache might be a confusing
choice of name for this feature in that case.

[Ken Barber]

>> can you give some more detail on how the cache will be used? If a fact
>> is found on disk via the rb file and there's nothing in the cache will
>> it then simply run the slow way? and update the cache?
>>
>  Facter itself will not use the cache. If you have an application that needs
> facts and needs them quickly, you may read the yaml file on disk. An
> entirely separate process will update the cache. At this first step, the
> update process is planned to be a cron job, with hope of an actual facter
> daemon later.

Err ... really? So we're not changing what we do at all today then?
People have been rolling facilities like this for quite some time
already (ie. with mcollective and yaml plugin), it doesn't seem like
we're adding much value at all. And from memory PE already does this.
I mean a cron job with:

facter -y > /var/lib/facter/cache.yaml

Is already do-able. So what I'm failing to understand is ... what are
the changes in facter we are proposing today then?

>> sounds like there would be various chicken and egg situations with
>> arranging
>> for pluginsync to have happened before attempts to build the cache so I
>> am looking to hear some more details to determine if that might be an
>> issue.
>
> At this time, we are not proposing Puppet use the external cache from the
> disk.

So nothing changes at all then? The problem we are setting out to
solve is more or less - not solved?

>> I wouldnt go so far as saying that :) generating fact caches via cronjob
>> or puppet writing out yaml files has been a recurring headache for people
>>
> Tell me more, please.

Atomic writes to the cache files for one.

ken.

[R.I.Pienaar]

----- Original Message -----
> From: "Ken Barber" <k...@puppetlabs.com>
> To: puppe...@googlegroups.com
> Sent: Friday, May 25, 2012 11:38:35 AM
> Subject: Re: [Puppet-dev] External cache for fact values
>

> >> can you give some more detail on how the cache will be used? If a
> >> fact
> >> is found on disk via the rb file and there's nothing in the cache
> >> will
> >> it then simply run the slow way? and update the cache?
> >>
> >  Facter itself will not use the cache. If you have an application
> >  that needs
> > facts and needs them quickly, you may read the yaml file on disk.
> > An
> > entirely separate process will update the cache. At this first
> > step, the
> > update process is planned to be a cron job, with hope of an actual
> > facter
> > daemon later.
>
> Err ... really? So we're not changing what we do at all today then?
> People have been rolling facilities like this for quite some time
> already (ie. with mcollective and yaml plugin), it doesn't seem like
> we're adding much value at all. And from memory PE already does this.
> I mean a cron job with:
>
> facter -y > /var/lib/facter/cache.yaml
>
> Is already do-able. So what I'm failing to understand is ... what are
> the changes in facter we are proposing today then?

I am also very confused, it seems this is a non feature and not needed
if it's not going to provide any actual caching to facter

[Chris Price]

On Thu, May 24, 2012 at 11:27 PM, Jeff Weiss <jeff....@puppetlabs.com> wrote:

 Facter itself will not use the cache. If you have an application that needs facts and needs them quickly, you may read the yaml file on disk. 

On Fri, May 25, 2012 at 3:38 AM, Ken Barber <k...@puppetlabs.com> wrote: 

And from memory PE already does this.

 

I mean a cron job with:

facter -y > /var/lib/facter/cache.yaml

Based on past experiences I am always a little unnerved by the idea of using a file on disk as "API".  It is revealing an implementation detail of your program that makes it much harder to make changes in the future without breaking backwards compatibility.  That concern, combined with Ken's  cron job snippet, gives me the inclination to say that people can fairly achieve what you are describing without us formally committing to that as an "API"... and that seems like a win to me.

 

[Chris Price]

s/fairly/fairly easily/

[Ken Barber]

>> I mean a cron job with:
>>
>> facter -y > /var/lib/facter/cache.yaml
>
>
> Based on past experiences I am always a little unnerved by the idea of using
> a file on disk as "API".  It is revealing an implementation detail of your
> program that makes it much harder to make changes in the future without
> breaking backwards compatibility.  That concern, combined with Ken's  cron
> job snippet, gives me the inclination to say that people can fairly achieve
> what you are describing without us formally committing to that as an
> "API"... and that seems like a win to me.

... just to be clear the code example I provided is an
over-simplification and doesn't take into account any atomic-write
issues. I think we had problems with partial writes of this cache file
in PE, that were solved in the cron job later on.

ken.