Options for shorter change turn around time

[J. Mixer]

All,

My appliance is pretty big and takes about 30 minutes to fully build and put on a loading device. It is currently a <type="oem"..... installiso="true"> so it creates the ISO output files which is perfect for shipping the image to external service people that load it on our product. It is a read-only image and easy for them to put on a DVD or USB stick and full-proof once it is there.

However, in development and for changes needed in the device configuration, small changes take a long time to turn around to test. (i.e. a 25 minute build for one file change.)

I'm looking for a better option for the "change and test" cycle. I would like to include additional files on the USB stick (i.e read-write USB) that can be loaded in once the initial .raw image file is loaded onto the hard disk and "pivoted" to. This would allow a single file to be updated on the usb and a new reboot to allow it to be tested.

 I have yet to experiment with the <type="oem" ....... installpxe="true"> output files. Could the kernel, initrd, and system image be put on a (by hand made) bootable USB flash? I'm guessing that this type is tightly tied to PXE booting and loading.

How about <type="kis" ...> type? I have used the <type="pxe" ...> in older versions of Kiwi which this is meant to replace. The document states that this type is very flexible, but leaves it to the reader to figure out how to adapt it.  Could some offer a few steps to help me along?

Example:

• Add kiw.dracut.... module to the root/ tree in your configuration directory
  • more detail
  • additional suggestion
• <some_name>.raw file will be looked for in the ??? directory of the USB stick

I would be happy to contribute a real world examples of a "kis" configuration to the Kiwi document once I've been through the process.

Regards,

J.

[Marcus Schäfer]

Hi,

> However, in development and for changes needed in the device
> configuration, small changes take a long time to turn around to test.
> (i.e. a 25 minute build for one file change.)

Before I think a bit closer on possible solutions I'd like to
understand more detailed what type of changes you would like to
manage easier in the "change and test" cycles.

So your target image is an install ISO. This means the production
image comes with a kernel the initrd and the .raw disk image on
a bootable ISO. So far so good:

- where in this data set do you expect changes when developing ?

* kernel ?
* initrd contents (your custom initrd code extensions) ?
* .raw disk image contents ?

> I have yet to experiment with the <type="oem"
> ....... installpxe="true">

I would also go in that direction. Such that you have a remote
deployment method in your network which can be handled very
flexible. However you should have in mind that the basic boot
method would be different so it does not replace an end-to-end
test of the production ISO

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer
-------------------------------------------------------
Marcus Schäfer Brunnenweg 18
Tel: +49 7562 905437 D-88260 Argenbühl
Germany
-------------------------------------------------------

[J. Mixer]

Marcus,

Thanks for the reply as always.

- where in this data set do you expect changes when developing ?

* kernel ?
* initrd contents (your custom initrd code extensions) ?
* .raw disk image contents ?

The changes are almost always in the .raw disk image. Mainly the changes are in scripts that configure the hardware in a rack of up to 14 computers. 

(I apologize for the length of this description. It is necessary for understanding of our environment.) All of these machines are connected to a network internal to the physical rack. I actually use two different Kiwi disk images:  One for our "head node" which is the "master" for all of the rest of the "remote nodes". The head node gets a very big set of packages and code and the remote nodes a much smaller image. The head node sets itself up to be a PXE server to serve all of the remote nodes their disk image. So we load 14 computers with one piece of install media. Manufacturing and service work with DVDs and USB sticks respectively. We in development have a great PXE setup to re-install disk images to all of our test machines remotely.

So, the head node disk image contains a large package list PLUS the disk image and initrd of the remote nodes. This is why the head node image is significantly large.

In the past, I have used Kiwi to create the (two different) disk image trees and initrd(s). I would pack up (tar) the image trees myself and "hack the living daylights out of" the Kiwi generated initrd and install the tar'd up disk images myself using some of your script tools that found and initialized many of the hardware devices that dracut/systemd does now. My initrd hacking broke in openSuse 15.3 and the latest version of Kiwi tools. 

I would now like to use all of the power of Kiwi and dracut in the re-architecting of our system. I would also "like" to have extra files to be added to the USB so everything doesn't need to be built into the head node .raw system image every time I make a change to the head node configuration script. The OEM ISO image initrd requires an ISO image to be read from DVD/USB. This disables being able to add files outside of the created ISO.  The PXE version requires the system image to be brought over the network. This is good for most of my development but in initial development anything that breaks the network requires local intervention at the test machine.

Where would a <type="kis" ...> build look for the system image file? Is it up to the developer to write all of the dracut hooks to go find it and download it?

 

> I have yet to experiment with the <type="oem"
> ....... installpxe="true">

I would also go in that direction. Such that you have a remote
deployment method in your network which can be handled very
flexible. However you should have in mind that the basic boot
method would be different so it does not replace an end-to-end
test of the production ISO 

 

Yes. I actually would like to use both. The PXE version is (definitely) necessary for loading my end nodes from the head node. Also, for development with almost all developers working from home PXE is necessary. Manufacturing and service cannot use the PXE method and require a good DVD (manufacturing) and USB (service) method. A USB method for initial development since networking on new hardware can be a problem. Therefore a "mixed" system image installation method would be best for me.

Best regards,

J.

[Marcus Schäfer]

Hi J,

> The changes are almost always in the .raw disk image. Mainly the
> changes are in scripts that configure the hardware in a rack of up to
> 14 computers.

ok got it and thanks for the background information.

Going forward I suggest to use installpxe="true" and a setup
using dnsmasq and QEMU for proof of concept testing without
harming your existing PXE boot infrastructure.

So here is how I would proceed:

1. In your development image <type> set

<type image="oem" ... installpxe="true"/>

And build this image. As part of the results you get

Some-Name.install.tar

2. Unpack that .install.tar into an extra directory

mkdir result
tar -C result -xf Some-Name.install.tar

3. Next take a look at the following documentation

https://osinside.github.io/kiwi/building_images/build_expandable_disk.html#network-deployment

It describes which files from your result/* data to copy
where in /srv/tftpboot

You can do this on some individual host (not your real PXE server !)

4. Once you have the data from the image build setup in /srv/tftpboot
you need to add the PXE booting loader and config. I'm pretty sure
you know more about all this than I do, but just in case you need
some docs, this one should be helpful:

https://osinside.github.io/kiwi/working_with_images/setup_network_bootserver.html

It uses dnsmasq to provide the protocols TFTP, DHCP.
You can run dnsmasq on your testing host. I recommend to
disable it's DNS capabilities with:

port=0

To avoid conflicts

5. Once you have that you need a boot menu for which I usually
use syslinux menu.c32 together with the pxelinux.0 loader.
(Note: the above docs uses grub and a self compiled grub module)

If you decide to go old school with syslinux your config file
is in: /srv/tftpboot/pxelinux.cfg/default

with contents like:

default menu.c32
prompt 0
timeout 120

menu title PXE Menu

label Install
menu label ^Install Leap
kernel /boot/linux
append initrd=/boot/initrd rd.kiwi.install.pxe rd.kiwi.install.image=tftp://192.168.178.34/image/Leap-15.3_appliance.x86_64-1.15.3.xz console=ttyS0 rd.kiwi.install.pxe.curl_options=--retry,3,--retry-delay,3,--speed-limit,2048

6. The essence of the boot menu in 5. are the options to pass along
on the kernel cmdline. These allow you to control the deployment.
You asked for "where does it fetch the actual image from..." This
you can see in rd.kiwi.install.image=...

kiwi uses curl to fetch files from the network. Thus all options
here are passed along to curl. Every feature of curl is at your
hand now. So you could also setup an ftp server put the contents
from /srv/tftpboot/image/* there and use ftp:// instead of tftp://

This would also be my suggestion how you can handle changes to
the image raw file. It is possible that you change the contents
of that file offline and re-deploy a new variant which should be
a quick process given all the setup above works.

7. Regarding initrd and kernel usage you should know that for
deployment the PXE loader loads "/boot/linux" and "/boot/initrd"
and after deployment the initrd calls kexec to start up. The
kernel and initrd for kexec are stored /srv/tftpboot/image

So you have in theory also the opportunity to have different
kernel/initrd combinations for deployment and runtime. Please
note you will not have this option when you build an ISO and
it should also only be used for development cycles if at all.

In the kiwi produced .install.tar file all files needed are
bundled as a working and consistent bundle. You can now do
whatever you want with the collection of files which gives you
some power but you could also create very strange and inconsistent
setups ;)

Hope this gives you some ideas to move forward.

Cheers,

[J. Mixer]

Marcus,

Many thanks for all of the detail. I'll need to get more familiar with qemu and dnsmasq for local debugging. Our own PXE installation will make the over the network deployment very simple.

Now I have some work to do.

Best regards,

J.

[J. Mixer]

Marcus,

I have the pxeboot....kernel and pxeboot...initrd booting on my test machine. (i.e. BIOS is finding DHCP and TFTP server for loading the kernel and initrd)

Dracut is too quickly timing out trying to get a DHCP address on eth0. Our corporate switches have about a 35 second delay from hardware network insert to actually going on the network because of a big spanning tree. (i.e. once the OS brings up the network hardware port, you have to wait 35 seconds before being able to see anything on the network.) I end up in an emergency prompt.

I'm looking for method to tell dracut to extend it's network wait time or number of retries (or even a fixed ipv4 address and gateway.)

I can get you a copy of  /run/initramfs/rdsosreport.txt tomorrow if it will help in finding a solution.

[Marcus Schäfer]

Hi,

> Dracut is too quickly timing out trying to get a DHCP address on eth0.
> Our corporate switches have about a 35 second delay from hardware
> network insert to actually going on the network because of a big
> spanning tree. (i.e. once the OS brings up the network hardware port,
> you have to wait 35 seconds before being able to see anything on the
> network.) I end up in an emergency prompt.
>
> I'm looking for method to tell dracut to extend it's network wait time
> or number of retries (or even a fixed ipv4 address and gateway.)

You have some options with the dracut network code. See here:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configuring_ip_networking_from_the_kernel_command_line

Regards,

[J. Mixer]

Marcus,

You have some options with the dracut network code. See here:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configuring_ip_networking_from_the_kernel_command_line

Thank you. This would give us a work around for a few machines that could be given "static" IPs via the kernel options. I have already worked around this machine be placing a small, dumb (i.e. non-managed) switch between the machine and the corporate switch. Dracut gets its DHCP address almost immediately and dracut continues.

Next problem:  (By the way, I would be open to creating a new discussion when a new problem is encountered. Let me know.)

Dracut (and Kiwi code hook) attempts to download the <image_name>.md5 file with sftp:// protocol and fails. I receive a nice (curses-like) graphic error asking me to view details in a log file. Unfortunately, I cannot find a way to even get an emergency prompt after the error occurs.  The machine will only reboot on enter or a short timeout. I have included the rd.kiwi.debug in the kernel options (and in the config.xml in the build) and see no difference. (I have seen this give me good information in the OEM ISO case.)  I have carefully verified that the properly named files do exist in the proper directory. 

Here is the grub2 stanza from the config file being booted:

label 221
 
  menu label Install CF/cc8 TEST BOS Do not use 153.01.100 HeadNode Only and Happy
  kernel img2a.cc8test
  append initrd=img3a.cc8test rd.kiwi.debug rd.kiwi.install.pxe rd.kiwi.install.image=sftp://172.27.18.54/install/cc8test/Ricoh-DFE-BOS.x86_64-1.15.3.xz rd.kiwi.oem.maxdisk=1100G rd.kiwi.install.pxe.curl_options=--user,downloader:R1c0h,--retry,3,--retry-delay,3,--speed-limit,2048 rd.kiwi.install.pass.bootparam="srcip=172.27.18.54 gateway=None NONODES=1 HAPPY=1 enablessh=yes enablenetwork=yes"

I'll also attach the current config.xml from the description that built the PXE image, A picture of testing curl from the same machine imaged from a USB (with the same kiwi config.xml), and a picture for the display of the PXE install failure.

Question 1:  Is there a way to break out of the found error to view the log file?

Question 2:  Do you have a suggestion to debug the problem otherwise?

Regards,

J.

[Marcus Schäfer]

Hi,

> Next problem: (By the way, I would be open to creating a new
> discussion when a new problem is encountered. Let me know.)

No problem I think it all belongs to the same story, so let's keep
the thread here

> Dracut (and Kiwi code hook) attempts to download the <image_name>.md5
> file with sftp:// protocol and fails. I receive a nice (curses-like)

ok, sftp probably requires some certs to be in the initrd

> graphic error asking me to view details in a log file. Unfortunately, I
> cannot find a way to even get an emergency prompt after the error
> occurs.

That you can fix with:

rd.debug rd.shell

> Question 1: Is there a way to break out of the found error to view the
> log file?

see above, should drop you into shell inside of the initrd

> Question 2: Do you have a suggestion to debug the problem otherwise?
> Regards,

I usually test the sftp command on the host first. If it succeeds there
the chance that it works from within the initrd is high. If it does not
work it's not because of the way sftp was called but because of either
no nic interface or missing data like certs

Hope this helps

[J. Mixer]

Marcus,

> Dracut (and Kiwi code hook) attempts to download the <image_name>.md5
> file with sftp:// protocol and fails. I receive a nice (curses-like)

ok, sftp probably requires some certs to be in the initrd

Yes, it was certs

> graphic error asking me to view details in a log file. Unfortunately, I
> cannot find a way to even get an emergency prompt after the error
> occurs.

That you can fix with:

rd.debug rd.shell

These options worked like a charm. rd.debug makes scrambles the screen with the pv graphics tries to come up, but it is acceptable. 

 

I usually test the sftp command on the host first. If it succeeds there
the chance that it works from within the initrd is high. If it does not
work it's not because of the way sftp was called but because of either
no nic interface or missing data like certs

Thanks for the suggestion. All I needed was to add a curl option: --insecure and the transfer of the  image started and completed.

I have a debug environment!!

Best regards,

J.

[Marcus Schäfer]

Hi J,

> Thanks for the suggestion. All I needed was to add a curl option:
> --insecure and the transfer of the image started and completed.
> I have a debug environment!!

Hooray :) That brings a smile on my face

Best regards,