PXE boot with nvidia drivers

[jacobg...@gmail.com]

Hi, 

I'm attempting to PXE boot coreos with nvidia drivers. I'm finding it to be quite a challenge. I've found a tool

https://github.com/Clarifai/coreos-nvidia

that builds them, but I'm not too sure how to get the output into the vmlinuz / initrd format needed by pxe boot. I do have the modules outputted from that script though, and I notice that in the squashfs there's a folder of drivers at /lib/modules/4.13.90-coreos/kernel/drivers.  

Do I just put the modules in there and re-squash? 

Sadly, I think there's something more that I'll need to do to ensure that the drivers get loaded. 

Anyway, I greatly appreciate any advice that people are able to throw my way. 

Best,

-Jake

[David Michael]

The initrd setup supports copying /usr/share/oem over the squashfs
file system, so you don't have to repack it. You could do something
like this:

mkdir -p usr/share/oem
cp -a modules usr/share/oem
find usr | cpio -co | gzip | cat coreos_production_pxe_image.cpio.gz -
> new.cpio.gz

So booting with new.cpio.gz as your PXE initrd will get the modules
into the file system, but if you want them to be useful through
modprobe etc., see this page about creating an overlay and running
depmod: https://coreos.com/os/docs/latest/kernel-modules.html . This
part can be configured with Ignition.

Thanks.

David

[Jacob Gadikian]

We're not installing to disk, so I think that ignition is out of the question. 

Is there any way to get the drivers depmod'd in the pxe boot image?  I guess I'm also curious how the PXE images are built, since we might be able to sneak our drivers (and hopefully their activation) in during the build of the vmlinuz/cpio.gz files? 

Thanks a lot for your help :). 

[David Michael]

On Sun, Nov 5, 2017 at 12:33 PM, Jacob Gadikian <jacobg...@gmail.com> wrote:
> We're not installing to disk, so I think that ignition is out of the
> question.

Why? You can give a URL to the JSON file with coreos.config.url= as a
kernel argument (along with coreos.first_boot=1 to make Ignition run).

It supports oem:// URLs that are paths relative to /usr/share/oem, so
you can bundle your Ignition config in the same directory as your
modules when appending to the initrd. For example, write the config
at usr/share/oem/ignition.json and boot with
"coreos.config.url=oem:///ignition.json coreos.first_boot=1".
Alternatively, all current OS releases support TFTP URLs, so you
should be able to serve them from your existing PXE server.

> Is there any way to get the drivers depmod'd in the pxe boot image? I guess
> I'm also curious how the PXE images are built, since we might be able to
> sneak our drivers (and hopefully their activation) in during the build of
> the vmlinuz/cpio.gz files?

No, you can't run depmod due to the modules directory being read-only.
The initrds are built by this function:
https://github.com/coreos/scripts/blob/master/build_library/vm_image_util.sh#L592
.

You could either recreate the squashfs image as you mentioned, or
maybe build your own customized OS image:
https://coreos.com/os/docs/latest/sdk-modifying-coreos.html .
However, I think it will be less work in the long run if you can use
the stock PXE initrd, append OEM files, and configure them into place
with Ignition.

Thanks.

David