Generating a file from a set of fragments on the puppetmaster

[Paul Gear]

Hi,

I'm trying to create a squid url_regex ACL source file for various
different sites. Each site needs a slightly different configuration, so
my plan was to create the a bunch of files on the server, then drag them
down and concatenate them into a single file on the client.

I found http://reductivelabs.com/trac/puppet/wiki/CompleteConfiguration
and i've been trying to understand its approach to constructing a file
out of fragments. Am i right in thinking that i need to distribute an
entire directory from the server, then use concatenated_file to combine
those files into one file on the puppet agent?

I'd rather not distribute the entire directory from the server, since it
contains custom content for each node. Is there a way i can do this
with templates that include other files? (Or templates that are plain
text rather than .erb?) I'd really like to find a technique that
doesn't require separately copying the file fragments to the client also...

I've had 3 or 4 tries at getting the right approach and am still no
closer to a working solution. Attached is my non-working attempt at a
class to do this - what am i doing wrong? (Comments about what's not
working can be found after the @@@ comments.) I thought this was a
fairly simple problem, but i've been banging my head against it all day
without any success.

Paul

[David Schmitt]

My approach in the CompleteConfiguration repos is to use the
concatenated_file_part define to put file parts onto the client and then
locally concatenated them all on the client with the concatenated_file
define which does some shell magic internally.

You seem to try to use a server side function ( file() ) to create a
similar effect. You just have to be aware that functions always run on
the server.

Regards, DavidS

[Paul Gear]

David Schmitt wrote:
> ...

>> I'd rather not distribute the entire directory from the server, since it
>> contains custom content for each node. Is there a way i can do this
>> with templates that include other files? (Or templates that are plain
>> text rather than .erb?) I'd really like to find a technique that
>> doesn't require separately copying the file fragments to the client also...
>>

>> ...

>
> My approach in the CompleteConfiguration repos is to use the
> concatenated_file_part define to put file parts onto the client and then
> locally concatenated them all on the client with the concatenated_file
> define which does some shell magic internally.
>
> You seem to try to use a server side function ( file() ) to create a
> similar effect. You just have to be aware that functions always run on
> the server.

I knew i must be overcomplicating it somehow. :-) So if i give the
file() function the full pathname of the directory on the server, i
should be able to achieve what i'm wanting without needing to distribute
a local copy of the files?

Are there any drawbacks of doing this? (Such as having to pull down the
content every time to check its MD5 sum, perhaps?)

Paul

[David Schmitt]

Paul Gear wrote:
> David Schmitt wrote:
>> ...
>>> I'd rather not distribute the entire directory from the server, since it
>>> contains custom content for each node. Is there a way i can do this
>>> with templates that include other files? (Or templates that are plain
>>> text rather than .erb?) I'd really like to find a technique that
>>> doesn't require separately copying the file fragments to the client also...
>>>
>>> ...
>> My approach in the CompleteConfiguration repos is to use the
>> concatenated_file_part define to put file parts onto the client and then
>> locally concatenated them all on the client with the concatenated_file
>> define which does some shell magic internally.
>>
>> You seem to try to use a server side function ( file() ) to create a
>> similar effect. You just have to be aware that functions always run on
>> the server.
>
> I knew i must be overcomplicating it somehow. :-) So if i give the
> file() function the full pathname of the directory on the server, i
> should be able to achieve what i'm wanting without needing to distribute
> a local copy of the files?

From http://reductivelabs.com/trac/puppet/wiki/FunctionReference#file :
> Return the contents of a file. Multiple files can be passed, and the first file that exists will be read in.

You probably should use template() (which concatenates its parameters;
be aware of ERB expansion though) or create a local concat_file()
function that works as intended or use generate() to call a server-side
script.

> Are there any drawbacks of doing this? (Such as having to pull down the
> content every time to check its MD5 sum, perhaps?)

Exactly. content=> is compiled into the configuration which is sent to
the client.

Also, it is hard to add to these files from other clients, like with
export/collect: you need a component on the puppetmaster and a third
puppetd run.

Regards, DavidS

[korymatu]

My understanding is that there are a series of files that exist on the
server. Each contains information that is needed by various nodes. All
nodes need some of the files; but not all nodes need all of the files.

Based on what I think you are trying to get done, I would probably use
generate...

For example:

class squid {
# Generate should return the results of the command...
$file_content = generate
("cat","header","$hostname.file1","$hostname.file2","isp.
$isp","footer")

file { "/path/to/squid_file":
content => $file_content,
}
}

You could set the ISP variable per node, or maybe create a new fact in
facter; depends on your setup really. To keep from constantly pushing
the configuration, you could look at an initialization file which can
be done a couple ways. The easiest would be to push a file that has
the date in it or something of that nature. Subscribe to that file so
that whenever it is updated, the squid file is also updated. So when
you make changes to the config files, you just update the date in your
init file.

Anyway, just a thought...

On Jul 3, 3:54 am, David Schmitt <da...@dasz.at> wrote:
> Paul Gear wrote:
> > David Schmitt wrote:
> >> ...
> >>> I'd rather not distribute the entire directory from the server, since it
> >>> contains custom content for each node.  Is there a way i can do this
> >>> with templates that include other files?  (Or templates that are plain
> >>> text rather than .erb?)  I'd really like to find a technique that
> >>> doesn't require separately copying the file fragments to the client also...
>
> >>> ...
> >> My approach in the CompleteConfiguration repos is to use the
> >> concatenated_file_part define to put file parts onto the client and then
> >> locally concatenated them all on the client with the concatenated_file
> >> define which does some shell magic internally.
>
> >> You seem to try to use a server side function ( file() ) to create a
> >> similar effect. You just have to be  aware that functions always run on
> >> the server.
>
> > I knew i must be overcomplicating it somehow.  :-)  So if i give the
> > file() function the full pathname of the directory on the server, i
> > should be able to achieve what i'm wanting without needing to distribute
> > a local copy of the files?
>

>  Fromhttp://reductivelabs.com/trac/puppet/wiki/FunctionReference#file:

>
> > Return the contents of a file. Multiple files can be passed, and the first file that exists will be read in.
>
> You probably should use template() (which concatenates its parameters;
> be aware of ERB expansion though) or create a local concat_file()
> function that works as intended or use generate() to call a server-side
> script.
>
> > Are there any drawbacks of doing this?  (Such as having to pull down the
> > content every time to check its MD5 sum, perhaps?)
>
> Exactly. content=> is compiled into the configuration which is sent to
> the client.
>
> Also, it is hard to add to these files from other clients, like with
> export/collect: you need a component on the puppetmaster and a third
> puppetd run.
>

> Regards, DavidS- Hide quoted text -
>
> - Show quoted text -

[Paul Gear]

Paul Gear wrote:
> Hi,
>
> I'm trying to create a squid url_regex ACL source file for various
> different sites. Each site needs a slightly different configuration, so
> my plan was to create the a bunch of files on the server, then drag them
> down and concatenate them into a single file on the client.
>
> I found http://reductivelabs.com/trac/puppet/wiki/CompleteConfiguration
> and i've been trying to understand its approach to constructing a file
> out of fragments. Am i right in thinking that i need to distribute an
> entire directory from the server, then use concatenated_file to combine
> those files into one file on the puppet agent?
>
> I'd rather not distribute the entire directory from the server, since it
> contains custom content for each node. Is there a way i can do this
> with templates that include other files? (Or templates that are plain
> text rather than .erb?) I'd really like to find a technique that
> doesn't require separately copying the file fragments to the client also...

> ...

Thanks for the suggestions on this. I ended up going with a templated
solution that uses a bit of Ruby to intelligently grab files in on the
server. I've documented my efforts here:

http://reductivelabs.com/trac/puppet/wiki/Recipes/SquidConfiguration

Any suggestions for improvement gratefully accepted.

Paul

[Paul Gear]

Paul Gear wrote:
> ...
>
> Thanks for the suggestions on this. I ended up going with a templated
> solution that uses a bit of Ruby to intelligently grab files in on the
> server. I've documented my efforts here:
>
> http://reductivelabs.com/trac/puppet/wiki/Recipes/SquidConfiguration
>
> Any suggestions for improvement gratefully accepted.

Hi folks,

Any feedback on whether i could do this better? I'd definitely like to
get that check_file method out of the top of every file and put it in a
common place.

Regards,
Paul

[Burkholder, Peter]

Puppeteers,

It's not just Paul who'd like an answer, if anyone has one to proffer.
He's been kind enough to document his work on the wiki. If it's not
exemplary of best practice, then do us all a favor and provide a better
solution.

Thanks,

Peter