Bare metal not getting IP after iPXE boot

[Chris Cowley]

I am trying to run a small Container Linux cluster using iPXE (chainloaded from PXE).

I have the nodes booting Linux nicely, but once Container Linux is running, it does not get an IP.

- On boot, the nodes make a request a DHCP request (which succeeds) via PXE.
- It downloads the iPXE env via TFTP, loads that env and makes another DHCP request (also succeeds).
- In the iPXE environment, it downloads the coreos kernel, initrd and an oem.cpio.gz with a basic ign config.
- It boots that kernel, loads the initrd and OEM.
- Once booted into Coreos, it makes another DHCP requested. My DHCP server responds with the address, but it is never applied.

At this point, it cannot apply its config as it has no networking.

When I tell it to boot straight to a console, I see the interface (using `ip link`). Running `networkctl link` shows the wired interface (eno1) stuck in "degraded, configuring".

TL;DR: after PXE booting CoreOS, NIC is present, but is not applying the config provided by the DHCP server.

[Seán C. McCord]

This sounds like a firmware or driver problem.  Are you using a really new or esoteric NIC?

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Seán C McCord

CyCore Systems, Inc

+1 888 240 0308

PGP/GPG: http://cycoresys.com/scm.asc

[Chris Cowley]

Nope. 1st gen Intel NUC which uses e1000

[Seán C. McCord]

I'm presuming by your statements that you are validating these claims with a packet capture or server-side log, but if not, we should check lower level configs first.

Once the system is booted to a console, are you then able to manually set up the network and have it work fine?

Another curiosity:  you say that the oem.cpio.gz and ignition config are downloaded, but that doesn't happen in iPXE on any system I know.  That happens within the first boot initrd.  The console you achieve (unless it is the emergency console) is achieved as a result of processing that ignition config and booting the "real" OS.  In this case, checking the Ignition config (and then its application) is the next thing to do.

[Chris Cowley]

On the DHCP server (ISC) I see the request from the NUC, and the address assigned. However, Coreos does not apply that address to the interface. If it comes to it, I'll stick a switch port in mirror mode and run a packet capture.

Yes, I from the console I can use `ip address ...` to define an IP address and successfully use the network.

For the oem.cpio.gz, I have this:

```
kernel ${base-url}/stable/coreos_production_pxe.vmlinuz initrd=coreos_production_pxe_image.cpio.gz,oem.cpio.gz coreos.first_boot=1 console=tty0 console=ttyS0 coreos.autologin=tty1 coreos.autologin=ttyS0
initrd ${base-url}/stable/coreos_production_pxe_image.cpio.gz
initrd ${base-url}/stable/oem.cpio.gz
```

In my ipxe bootscript, and the oem.cpio.gz contains a super simple ignition file. Sure enough I find that ignition in /usr/share/oem however, it does not seem to actually being applied. Although I haven't looked at that in any detail yet though. I suspect that it will be applied once the network comes up successfully.

I would also add, that I booted Fedora 27 and Centos 7 on the NUC this morning and DHCP works perfectly, but RancherOS gives the same result as Coreos however.

[Alex Crawford]

On 12/31, Chris Cowley wrote:
> On the DHCP server (ISC) I see the request from the NUC, and the address
> assigned. However, Coreos does not apply that address to the interface. If
> it comes to it, I'll stick a switch port in mirror mode and run a packet
> capture.
>
> Yes, I from the console I can use `ip address ...` to define an IP address
> and successfully use the network.
>
> For the oem.cpio.gz, I have this:
>
> ```
> kernel ${base-url}/stable/coreos_production_pxe.vmlinuz
> initrd=coreos_production_pxe_image.cpio.gz,oem.cpio.gz coreos.first_boot=1
> console=tty0 console=ttyS0 coreos.autologin=tty1 coreos.autologin=ttyS0
> initrd ${base-url}/stable/coreos_production_pxe_image.cpio.gz
> initrd ${base-url}/stable/oem.cpio.gz
> ```
>
> In my ipxe bootscript, and the oem.cpio.gz contains a super simple ignition
> file. Sure enough I find that ignition in /usr/share/oem however, it does
> not seem to actually being applied. Although I haven't looked at that in
> any detail yet though. I suspect that it will be applied once the network
> comes up successfully.
>
> I would also add, that I booted Fedora 27 and Centos 7 on the NUC this
> morning and DHCP works perfectly, but RancherOS gives the same result as
> Coreos however.

This sounds like a networkd issue. If you re-run networkd with
SYSTEMD_LOG_LEVEL=debug, do you see any additional logs?

-Alex

[Chris Cowley]

Maybe I am being dim, but how? For the life of me I cannot find how to actually run networkd (which seems to be the worst documented project in the universe) from the command line

[Alex Crawford]

On 01/09, Chris Cowley wrote:
> Maybe I am being dim, but how? For the life of me I cannot find how to
> actually run networkd (which seems to be the worst documented project in
> the universe) from the command line

It's easiest to write a drop-in for systemd-networkd.service and add the
following:

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

Then you can `systemctl daemon-reload` and `systemctl restart
systemd-networkd`.

-Alex