Re: [Puppet Users] Puppet and installing packages from source

[Matthew Burgess]

On Wed, Jan 2, 2013 at 9:32 AM, <magi...@gmail.com> wrote:
> Hello Everyone,
>
> I'm looking for a way to install packages from source via puppet, I was able
> to locate maybe 5 posts on this subject which is very strange, as I would
> expect people to still use ./configure with their own customized options.
> For example, I need to compile Curl with c-ares support, for that I need to
> set an option for ./configure but can't find a way to do it.

I'll get to your actual errors in a moment, but first, I have to ask 'why'?

Compiling everything from source on every host just seems really
inefficient. It also will require a compiler and other development
tools on all servers, which isn't generally recommended in a
production environment.

What is preventing you from compiling your special flavour of Curl on
a development box, and packaging it up using your operating system's
native package management tools (.deb, .rpm, etc.)? Then you can use
puppet to manage the package, rather than as a glorified script
execution mechanism.

> Debug: Exec[configure](provider=posix): Executing './configure'
> Debug: Executing './configure'
> Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: 1:
> ./configure: expr: not found
> Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: 1:
> ./configure: expr: not found
> Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: 95:
> ./configure: as_fn_error: not found
> Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: line 83:
> expr: command not found
> Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: line 90:
> expr: command not found
> Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: line 95:
> as_fn_error: command not found
> Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: line 465:
> sed: command not found
> Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: line 464:
> expr: command not found
> Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: line 465:
> sed: command not found
> Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: line 479:
> sed: command not found
> Notice: /Stage[main]/Web_dev/Exec[configure]/returns: : error: cannot create
> .lineno; rerun with a POSIX shell
> Error: ./configure returned 1 instead of one of [0]
> Error: /Stage[main]/Web_dev/Exec[configure]/returns: change from notrun to 0
> failed: ./configure returned 1 instead of one of [0]
>
> Anyone did this or knows whats wrong ?

Yes, the environment that puppet runs 'exec' statements under is
restricted. You need to set the 'path' parameter to include all the
paths in which those 'missing' executables can be found. See
http://docs.puppetlabs.com/references/3.0.latest/type.html#exec.

Regards,

Matt.

[Throwe, Jesse]

One thing I will say about that ( at least with RPM ) is that packaging binaries and libraries in out of place locations can become an uphill battle.  I've yet to find a clean way of getting RPM to let me have the OS version of something, say curl or perl in /usr, and my own internal one in /usr/local without it becoming a tangled mess.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[magi...@gmail.com]

Thanks for the reply Matt and Jesse,

I accept the reasons (Matt) mentioned for not using source files.

I guess the 1st reason I didn't go with rolling up my own .deb fie is that I lack the knowledge to do so, and I'm under a tight schedule for this project so I thought to stick with what I know.

The 2nd reason is that I could not believe that Puppet does not support working with source files with a built in mechanism (provider), digging through documentation I realized it does not provide any means to handle source.

I guess I wanted to emulate my current installation method which mixes both apt and compiling packages.

I fixed the configure part with adding the path to the binaries mentioned in the error (/bin, /usr/bin).

Thanks for all the help,

Magic.

[Jochen Schalanda]

Hi,

On 02.01.2013 16:56, magi...@gmail.com wrote:
> I accept the reasons (Matt) mentioned for not using source files.
> I guess the 1st reason I didn't go with rolling up my own .deb fie is
> that I lack the knowledge to do so, and I'm under a tight schedule for
> this project so I thought to stick with what I know.

You might want to take a look at fpm [1] and fpm-cookery [2], two
programs which make it sufficiently easy to create distribution packages
(think RPM or DEB) from source tarballs.

> The 2nd reason is that I could not believe that Puppet does not support
> working with source files with a built in mechanism (provider), digging
> through documentation I realized it does not provide any means to handle
> source.

The reason for the lack of a "source" provider is that there simply
isn't even remotely a standardized way for building software from
source. Which build system would you support? GNU Autotools? CMake?
SCons? Rake? Maven? Some obscure proprietary build system?

Also when building software from source you usually need to take care of
having the correct (build and runtime) dependencies installed which is a
big pain on its own. Plus usually you don't want to run a complete build
environment on production systems anyway.

You'll be better off using your operating system's package manager.


[1]: https://github.com/jordansissel/fpm
[2]: https://github.com/bernd/fpm-cookery


Best regards,

Jochen

[jcbollinger]

On Wednesday, January 2, 2013 3:32:55 AM UTC-6, magi...@gmail.com wrote:

Hello Everyone,

I'm looking for a way to install packages from source via puppet, I was able to locate maybe 5 posts on this subject which is very strange, as I would expect people to still use ./configure with their own customized options.

Why would you expect that?  Responsible sysadmins use packages wherever possible.  The alternative is madness.

Moreover, if you continue with the puppet-controlled build approach, you will find that you have to go to a fair amount of effort to ensure that the program is built and installed only at need.  Then, eventually you may find that it's even trickier to fit updates into the picture.  And you may discover that you need to explicitly manage dependencies -- hopefully before something breaks unexpectedly.  Relying on your package manager makes all this a lot easier to deal with.


John

[jcbollinger]

On Wednesday, January 2, 2013 8:01:36 AM UTC-6, Jesse Throwe wrote:

One thing I will say about that ( at least with RPM ) is that packaging binaries and libraries in out of place locations can become an uphill battle.  I've yet to find a clean way of getting RPM to let me have the OS version of something, say curl or perl in /usr, and my own internal one in /usr/local without it becoming a tangled mess.

Out of curiosity, what's messy about that?  As long as you name the custom packages differently from the standard ones, and you don't have any file conflicts, yum/rpm should have no objection to installing them and thereafter managing them.  Indeed, that's how RedHat packages "compatibility" and preview software, and I've done the same myself.


John

[magi...@gmail.com]

I understand the points you mentioned, but as long as software is still being distributed as source packages I think it should be supported at some level by Puppet.

That being said, I'm off to learn some packaging techniques.

Thanks for all the help.

Magic.

[R.I.Pienaar]

----- Original Message -----
> From: "magic rh" <magi...@gmail.com>
> To: puppet...@googlegroups.com
> Sent: Thursday, January 3, 2013 10:52:55 AM
> Subject: [Puppet Users] Re: Puppet and installing packages from source
>
> I understand the points you mentioned, but as long as software is still
> being distributed as source packages I think it should be supported at some
> level by Puppet.

I think the problem here is that you think they are "source packages" they
are not packages at all, they are source. Sources used as input to creating
packaged means of deploying code tailored to suit the needs of your distribution

Just as you cannot consider bread and flour to be the same thing or say flour
has the same properties as bread or offer the same advantages and usage models
you cannot consider source a form of package.

> That being said, I'm off to learn some packaging techniques.
>
> Thanks for all the help.
> Magic.
>
> On Wednesday, January 2, 2013 7:30:01 PM UTC+2, jcbollinger wrote:
> >
> >
> >
> > On Wednesday, January 2, 2013 3:32:55 AM UTC-6, magi...@gmail.com wrote:
> >>
> >> Hello Everyone,
> >>
> >> I'm looking for a way to install packages from source via puppet, I was
> >> able to locate maybe 5 posts on this subject which is very strange, as I
> >> would expect people to still use ./configure with their own customized
> >> options.
> >>
> >
> >
> > Why would you expect that? Responsible sysadmins use packages wherever
> > possible. The alternative is madness.
> >
> > Moreover, if you continue with the puppet-controlled build approach, you
> > will find that you have to go to a fair amount of effort to ensure that the
> > program is built and installed only at need. Then, eventually you may find
> > that it's even trickier to fit updates into the picture. And you may
> > discover that you need to explicitly manage dependencies -- hopefully
> > before something breaks unexpectedly. Relying on your package manager
> > makes all this a lot easier to deal with.
> >
> >
> > John
> >
> >
>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/8TpS7e3N-aIJ.

> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>

[magi...@gmail.com]

I understand exactly what source packages do, if it makes it easier for you I'll call them tar.gz files.

Also, I don't find your comparison valid, as the result of package installation be it yum or ./configure && make && make install is the same: software being installed.

My meaning is that compiling from source will always be with us, and it's the prominent way of releasing new software.

Thanks,

Magic.

On Wednesday, January 2, 2013 11:32:55 AM UTC+2, magi...@gmail.com wrote:

Hello Everyone,

I'm looking for a way to install packages from source via puppet, I was able to locate maybe 5 posts on this subject which is very strange, as I would expect people to still use ./configure with their own customized options.

For example, I need to compile Curl with c-ares support, for that I need to set an option for ./configure but can't find a way to do it.

Here's the manifest, it works up to the point of "configure" which fails:

file { "/usr/local/src": ensure => directory }

file { "/usr/local/src/c-ares-1.9.1.tar.gz":

source => "puppet:///modules/web_dev/c-ares-1.9.1.tar.gz",

alias => "ares-source-tgz",

before => Exec["untar-ares-source"]

}

exec { "tar xzf /usr/local/src/c-ares-1.9.1.tar.gz":

cwd => "/usr/local/src",

path => "/bin",

creates => "/usr/local/src/c-ares-1.9.1",

alias => "untar-ares-source",

subscribe => File["ares-source-tgz"]

}

exec { "configure":

command => "./configure",

cwd => "/usr/local/src/c-ares-1.9.1",

path => "/usr/local/src/c-ares-1.9.1",

require => Exec[untar-ares-source],

creates => "/usr/local/src/c-ares-1.9.1/config.h",

}

The ERROR:

Debug: Exec[configure](provider=posix): Executing './configure'

Debug: Executing './configure'

Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: 1: ./configure: expr: not found

Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: 1: ./configure: expr: not found

Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: 95: ./configure: as_fn_error: not found

Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: line 83: expr: command not found

Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: line 90: expr: command not found

Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: line 95: as_fn_error: command not found

Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: line 465: sed: command not found

Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: line 464: expr: command not found

Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: line 465: sed: command not found

Notice: /Stage[main]/Web_dev/Exec[configure]/returns: ./configure: line 479: sed: command not found

Notice: /Stage[main]/Web_dev/Exec[configure]/returns: : error: cannot create .lineno; rerun with a POSIX shell

Error: ./configure returned 1 instead of one of [0]

Error: /Stage[main]/Web_dev/Exec[configure]/returns: change from notrun to 0 failed: ./configure returned 1 instead of one of [0]

Anyone did this or knows whats wrong ?

Thanks,

Magic.

[R.I.Pienaar]

----- Original Message -----
> From: "magic rh" <magi...@gmail.com>
> To: puppet...@googlegroups.com
> Sent: Friday, January 4, 2013 9:39:53 AM
> Subject: [Puppet Users] Re: Puppet and installing packages from source
>

> I understand exactly what source packages do, if it makes it easier for you
> I'll call them tar.gz files.
> Also, I don't find your comparison valid, as the result of package
> installation be it yum or ./configure && make && make install is the same:
> software being installed.

it's not really though is it? the make install route drops a bunch of files
down, just tossing it out there with no regard for what comes before or after

the yum package apart from putting files on disks also does to name just a few:

- manages the serving, locating and transferring of the package onto your OS
for you, this is important because built into this process is things like
check summing, mirroring including fastest mirror determination, local
caches and so forth. You get some of this of course with the puppet file
server like the checksums but Puppet isnt really great at it.

- manage dependencies: makes sure you have all the pre-determined libraries
you need for the software to function. the ./configure method will adjust
the software based on the requirements met - you might find that on one
machine your ./configure results in a tool minus a key feature while on
the others it has it due to a simple ordering problem with when you deliver
devel libraries

- manage conflicts: if a OS update in the future comes with a file that might
conflict with your ./configure build you'll be none the wiser, it'll just be
overridden. Packages won't do this, it'll register a conflict and refuse to
install thus surfacing this behavior change and potential problem in a mature
way

- manage file lists: every file the package installs are registered and tracked
and you can later compare them for changes. or when you upgrade to the next
version orphaned files are detected and removed, config files are not blindly
overwritten but backed up and possibly not even replaced. you can later safely
uninstall it based on this database

- handles things like integration with the OS init, rc, pam, logging and other
subsystems

- registers the new capability for other packages to be able to rely on it

- provide many CLI tools to query, validate and generally manage the software you
have on your system in a single consistent way that means you never need to
track - as a user or as an automation system - things like different configure
flags depending on the type of software. It's all standardised and unified into
a single coherent system.

All in a way that combines with how your OS vendor chose to design their OS so you
know that by working with their approach you'll end up with a better functioning
whole in the end

Puppet is designed to use the facilities provided by your package manager, for example
the package resource has the ability to install, uninstall and upgrade a package
and it needs the ability to query if a package is installed. By using these
facilities it provides a idempotent means of managing the full life cycle of your
software in a mature way. Because your tgz files do not do any of the above things
puppet is simply not capable of providing a solid management experience of software
delivered in such a manner - the key information sources and standardised management
just does not exist for tgz files as they do for true packages.

The configure method does the absolute minimal, it just puts files down in some
non deterministic location. Worse it requires you to know and manage a wide range
of different configure flags and have appropriate error handling for every possible
scenario wedged awkwardly into some exec statements. This is not package management,
they are not packages. This is just wholesale creation of technical debt.

> My meaning is that compiling from source will always be with us, and it's
> the prominent way of releasing new software.

We'll always be compiling software as you say, it's how you manage the compilation
and delivery of the software that changes. fpm is no harder to use than the tar command
and while it does not provide all of these benefits it goes a long way to improving
what happens to your software once its compiled.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/puppet-users/-/zLcAvPzXtPYJ.

[magi...@gmail.com]

Thanks Hans, great answer.

I'll definitely get into customized packaging, demystify their magics.

Regards,

Magic.

On Friday, January 4, 2013 3:35:20 PM UTC+2, Hans van der Made wrote:

On Friday, January 4, 2013 10:39:53 AM UTC+1, magi...@gmail.com wrote:

I understand exactly what source packages do, if it makes it easier for you I'll call them tar.gz files.

Also, I don't find your comparison valid, as the result of package installation be it yum or ./configure && make && make install is the same: software being installed.

My meaning is that compiling from source will always be with us, and it's the prominent way of releasing new software.

When all the required features of a source build & install environment are met, like uninstalling, upgrading, listing, you end up with a new type of package provider. As you don't want a split between OS-provided software (libraries your source file depends on), it's easier to use the provider your OS comes with.

The requirements for building source are often many. You'll need a compiler and a toolchain, at least. Many don't want all of these tools installed on their (virtual) production servers, because they consume space, require updates, and because they facilitate exploits. Less software means less work and less risk.

Building software on a separate system and deploying the resulting binaries and documentation to production systems is one step up from compiling on your production systems.

Many of us used to compile from source earlier in our careers, so don't asume we don't know about anything but ready made packages ;-)

Now for a quick hack to address your problem:

Why don't you try to do a ./configure --prefix=/usr/local --specialopts && make && make install on a clean system and create a tar file of /usr/local/ afterwards?  Missing dependencies can be met by installing OS packages on your test system. The installation on your target system can be a hack like this:

# dependencies
package { [ "dep1", "dep2", "dep3", ]: ensure => installed, }

file { "/var/tmp/curl-custom-version.tar.gz":
  ensure => file,
  source => "puppet:///module/curl-custom-version.tar.gz",
  notify => Exec["unpackcurl"],
}

exec { "unpackcurl":
  command => "tar -C /usr/local -xzf /var/tmp/curl-custom-version.tar.gz",
  creates => "/usr/local/bin/curl",
  path => "/usr/sbin:/usr/bin:/sbin:/bin",
  logoutput => true,
  refreshonly => false,
}

If possible, use packages, but if you can't meet your deadline this way, try the above (and discover the wonderful world of packaging anyway at a later time).

Best,

Hans

[Jakov Sosic]

On 01/04/2013 02:35 PM, Hans van der Made wrote:

> Many of us used to compile from source earlier in our careers, so don't
> asume we don't know about anything but ready made packages ;-)

Many of us still do that today, but compile to build packages and not to
shuffle files around production root filesytem ;)



--
Jakov Sosic
www.srce.unizg.hr