Re: Proposed hook(s) for automated install/config, & looking to hire someone to implement them

[Don Jackson]

I'd like to update this thread with a summary (and pointers) to the
work that has been accomplished to date to provide some hooks and tools
in order to fully automate the installation and configuration of
OpenBSD.

My original proposal for two hooks is below, (and also here: http://archives.neohapsis.com/archives/openbsd/2008-07/0190.html
)

I worked with Nick Bender to get these hooks implemented, Nick's code/
diffs are documented in the following two emails:

http://archives.neohapsis.com/archives/openbsd/2008-09/0942.html

http://archives.neohapsis.com/archives/openbsd/2008-09/1662.html

and at Nick's web site:

http://nbender.com/install.netboot/install.html

Nick completely implemented the functionality I requested. After
using it for a while, I realized that installation via pxeboot
is not always the most convenient way to initiate the re-install of an
existing OpenBSD system, the issue is that on many machines,
one must escape into the BIOS during the boot process to set pxeboot
as the first boot target. This itself is a non-automatable process,
and furthermore, once the machine has finished installing, and
reboots, one must race to abort the reboot, escape into the BIOS again,
and remove pxeboot as the first boot target, or a (infinite?) series
of repeated re-installations will result.

The workaround I proposed is yet another modification/variant of the
bsd.rd installer, which behaves as if it had been pxebooted (ie, fires
up a network interface
and requests an IP address via DHCP), even when it has not been
pxebooted. With this, one can go to a running OpenBSD machine,
change /etc/boot.conf to run this variant of bsd.rd when booting, then
reboot the machine. Nick graciously implemented this new feature for
me, and it works like a charm.

Originally, I proposed that pxeboot request a series of boot.conf
files, one of which would have the MAC address of the client machine
appended to the boot.conf pathname
(eg /etc/boot.conf.MACADDRESS). Nick implemented it, and it works fine.

I also thought of another way to accomplish the same goal, which would
be more flexible, general,and not require changes to OpenBSD's
pxeboot, which is to modify tftpd to support a scripting interface, so
tftp requests could be processed by a user defined script. I chose to
closely mimic the interface between tftpd and the script on the CGI
specification.
I proposed this functionality here:

http://archives.neohapsis.com/archives/openbsd/2008-12/0652.html

And the resulting implementation was announced (and is available) here:

http://archives.neohapsis.com/archives/openbsd/2009-02/0023.html

One possible implementation option for an automated installation/
configuration system is to use a database to store a machine's
configuration.
I have had great success using Verdad,

https://sourceforge.net/projects/verdad/

And I announced an OpenBSD port here:

http://archives.neohapsis.com/archives/openbsd/2008-11/1674.html

With the above hooks and tools, I have accomplished my goal of having
an automated installation and configuration facility that supports
OpenBSD.

I hope that over time the hooks and functionality enhancements to
bsd.rd will be accepted into the official release.
It would be awesome to have this functionality there all the time, if
people want to use it (existing behavior would of course be the default)

The tftpd-cgi server is now available as a port, perhaps in time will
be committed, and possibly the resulting package will become standard
someday.

Acknowledgements:

Nick Bender implemented the modified versions of bsd.rd, automated/
automatable versions of the OpenBSD install shell scripts, and made
the changes I proposed to pxeboot.

Jason Creighton implemented the cgi interface to tftpd and the OpenBSD
port for the resulting tftpd-cgi

Jeff R Allen updated Verdad, and created the OpenBSD port

The idea to modify bsd.rd to act as if it had been pxebooted when it
had not was inspired by Mike Erdely's yaifo:
http://erdelynet.com/?s=yaifo
http://sourceforge.net/projects/yaifo/

I was originally inspired and motivated to further develop an
automated OpenBSD installation system after reading
about Brendan Strejcek's posting here:

http://people.cs.uchicago.edu/~brendan/howtos/openbsd_install/

Brendan's post was very helpful to me in my early efforts.

And Richard Newman is a constant source of help and useful advice,
thank you!

Regards,

Don Jackson

On Jul 3, 2008, at 1:17 PM, Don Jackson wrote:

> Hello,
>
> I am looking to add a couple of hooks into OpenBSD to enable automated
> install and configuration.
> The existing install.site/siteXY.tgz are great, and I use that hook
> extensively, but I would like to automate
> getting my machines to the point where install.site can be run.
>
> My overall goal is that various attributes of a machine should be
> specified in a configuration management system, and the key identifier
> of
> a machine is its mac address, which should be used to look up
> everything about the machine.
>
> I've given this some thought, and here are the minimum two hooks in
> OpenBSD that I think are required:
>
> 1) pxeboot should request a series of specific "boot.conf" files:
> pxeboot should be modified to first request boot.conf.MAC-ADDRESS
> (the mac address of the machine), if that file is found via tftp, it
> is used.
> If not, pxeboot should then request
> boot.conf.OpenBSD-Major.Minor-machine (eg boot.conf.OpenBSD-4.3-amd64
> ), if found, use that, if not
> pxeboot should finally request boot.conf (as it does today). The
> advantage of this is that one can specify very specific information
> about
> which version of bsd.rd should be used to install the machine,
> and this can be detailed down to the individual machine via the mac
> address.
> If you don't need that much granularity, the
> boot.conf.OpenBSD-Major.Minor-machine allows proper installation for a
> broader class of machines
> and releases than is supported today. BTW, ISC-DHCP already
> provides the ability to specify a unique pxeboot file for a given mac
> address, so
> that part is already complete, the snag is that all OpenBSD
> versions of pxeboot request the same etc/boot.conf file!!!
>
> 2) bsd.rd should be able to request (over the network) the install
> script to be used.
> Currently, bsd.rd executes .profile
> (src/distrib/miniroot/dot.profile) to ask the user what to do
> (install, upgrade, sh).
> Before that, I'd like dot.profile to attempt to download an
> install/upgrade script from the network, if it finds such a script, it
> automatically
> would use that, and begin executing it. I am not sure exactly how
> to specify the name of this script, ideally via a config line in the
> boot.conf file
> that pxeboot obtains, but if that is not easy to do, I am open to
> other ideas. We could hard code this URL, but ideally we'd allow some
> granularity
> like in the boot.conf case above. (e.g.
> http://openbsd-install/mac-address/custom-install.sh,
> http://openbsd-install/4.3/amd64/custom-install.sh )
>
> That's it. With these two hooks, it would be relatively
> straightforward to pxeboot a new machine, and the configuration
> management system
> would interact with the local infrastructure (DHCP, tftp,
> configuration web server, etc) to bring a machine up, on the network,
> install the OS, and
> presumably finally executing the install/upgrade.site script at the
> end.
>
> I'm not a hard core systems programmer, so I'm looking to hire someone
> to develop the above.
> Ideally we could implement it so that current behavior is preserved (I
> believe my proposal above falls back to existing behavior), and
> hopefully we could get these additions accepted for some future
> OpenBSD release.
>
> If you might be interested in implementing the proposal above, please
> contact me.
>
> If you have a better idea of how to implement what I have proposed
> above, I'd love to hear your ideas also.
>
> Best regards,
>
> Don

[Mike Erdely]

On Sun, Feb 01, 2009 at 11:21:36AM -0800, Don Jackson wrote:
> Acknowledgements:

>
> The idea to modify bsd.rd to act as if it had been pxebooted when it had
> not was inspired by Mike Erdely's yaifo:
> http://erdelynet.com/?s=yaifo
> http://sourceforge.net/projects/yaifo/

In the spirit of giving credit where it's due, YAIFO was created by
Waldemar Brodkorb.

-ME

[Brendan Strejcek]

It's cool that people are still making use of my little tutorial.

Thanks for the note.

Best,
Brendan

> installation/configuration system is to use a database to store a machine's

> configuration.
> I have had great success using Verdad,
> https://sourceforge.net/projects/verdad/
> And I announced an OpenBSD port here:
> http://archives.neohapsis.com/archives/openbsd/2008-11/1674.html
> With the above hooks and tools, I have accomplished my goal of having an
> automated installation and configuration facility that supports OpenBSD.
> I hope that over time the hooks and functionality enhancements to bsd.rd
> will be accepted into the official release.
> It would be awesome to have this functionality there all the time, if people
> want to use it (existing behavior would of course be the default)
> The tftpd-cgi server is now available as a port, perhaps in time will
> be committed, and possibly the resulting package will become standard
> someday.
> Acknowledgements:
> Nick Bender implemented the modified versions of bsd.rd,

> automated/automatable versions of the OpenBSD install shell scripts, and

> made the changes I proposed to pxeboot.
> Jason Creighton implemented the cgi interface to tftpd and the OpenBSD port
> for the resulting tftpd-cgi
> Jeff R Allen updated Verdad, and created the OpenBSD port

> The idea to modify bsd.rd to act as if it had been pxebooted when it had not
> was inspired by Mike Erdely's yaifo:
> http://erdelynet.com/?s=yaifo
> http://sourceforge.net/projects/yaifo/