Did you ever figure this out? I imagine you did, but if not, I had this same problem. Took me a few days to figure out but its a mixture of the kernel arguments and what values you use to make the ISO.
The Highlighed areas need to be updated as follows:
- rhcos-...... this needs to be the name you are using for your ISO volume "RHCOS-x86_64" it can't point to the OOTB value just something different
- coreos.inst.install_dev=sda - this needs to be /dev/sda
- coreos.inst.secure - this in my case I put =true and had it before coreos.inst.image_url in the order
Final values should be below - this is just what I had to do to get mine to start working :)
menu label ^RHEL CoreOS (Live)
ISO (should be same as you had)
genisoimage -U -A "RHCOS-x86_64" -V "RHCOS-x86_64" -volset "RHCOS-x86_64" -J -joliet-long -r -v -T -x ./lost+found -o /tmp/rhcos-4.5.6-x86_64-installer-custom.x86_64.iso -b isolinux/isolinux.bin -c isolinux/boot.cat
-no-emul-boot -boot-load-size 4 -boot-info-table -eltorito-alt-boot -e images/efiboot.img -no-emul-boot .
Hopefully it helps someone else out as I'm sure you got this already. For me, it was a pain to get to this point.