PXE booting Flatcar with ignition.config.url fails at systemd-resolved config

[Joris Scheppers]

I'm just starting my Flatcar Container Linux journey and I've hit a wall. I'm using an Intel NUC with integrated Intel Gbit NIC as the machine to do my first deployment on. I'm using PXE, configured in my local DHCP server and pointing to my local Synology NAS as a tftp server.

From there, pxelinux gets loaded on the PXE client which targets the Flatcar Linux latest production image (version 2983.2.0). The pxelinux.cfg-file which corresponds to the MAC-address of the NUC contains for following (as copied from https://www.flatcar-linux.org/docs/latest/installing/bare-metal/booting-with-pxe/): 

default flatcar

prompt 1

timeout 15

display boot.msg

label flatcar

  menu default

  kernel flatcar_production/flatcar_production_pxe.vmlinuz

  initrd flatcar_production/flatcar_production_pxe_image.cpio.gz

  append flatcar.first_boot=1 ignition.config.url=tftp://10.0.2.23/xx-yy-zz-64-fd-54/ignition.conf

fyi: MAC digits obfuscated, MAC address matching for NUCs works as expected.

Booting without the ignition.config-url kernel option gets me into the login prompt without problems, but booting with the option results in an error when configuring resolved, specifically:

systemd-resolved[123]: Failed to listen on UDP socket 127.0.0.53:53: Cannot assign requested address systemd-resolved[123]: Failed to start manager: Cannot assign requested address

[FAILED] Failed to start Network Name Resolution

This error, along with a lot of other output, loops every few seconds. During this time, the link light of my network switch stays off which seems to indicate the NIC is not properly initiated during the boot process.

Since I do not know (due to my Linux knowledge level of -infinity) how to capture the tty output from my NUC in a sharable format, I am not able to post the entire log output of the boot process. I hope this error will at least create some follow-up questions which I will hopefully be able to answer with specific output snippets. If it helps, I can put a video of the boot process online.

Does anyone know what I'm doing wrong here?

[k...@kinvolk.io]

Hi,

there seems to be a problem with Ignition either being unable to fetch the config (network configuration in initramfs, or the URL not working) or with a directive inside the config.

In releases >= 3005.0.0 there is a new mechanism which halts the boot process and makes it easier to spot the error message. If you want to debug, I recommend using the latest Beta or Alpha release.

By the way, I see that you specified a tftp:// URL - this is supposed to work but honestly I never it (only http) and can't tell more about it, but maybe first get the Ignition error message using the latest Beta/Alpha release (a screenshot is fine).

Regards,
Kai

[Joris Scheppers]

Hi Kai, thanks for your input.

I did try an HTTP-based Ignition location, I put it on pastebin and linked the Ignition URL to the raw pastebin link (https://pastebin.com/raw/BRgPAHnQ). This produced a different (but imho related) error:

ignition[358] GET https://pastebin.com/raw/BRgPAHnQ: attempt #1

ignition[358] GET error: Get "https://pastebin.com/raw/BRgPAHnQ" dial tcp: lookup pastebin.com on [::1]:53: read udp [::1]:50172->[::1]:53: read connection refused

I'll try the latest beta release next, brb.

[Joris Scheppers]

Using the pastebin config link results in the same error as described in my previous post, but after 7 attempts it shows: GET result: OK. Apparently the config is not valid (I probably failed to transpile the YAML into JSON), but it was able to fetch it in the end. 

I'll try the tftp option again, just to be sure.

[Joris Scheppers]

The tftp option gives the following output (thanks for the emergency shell!)

[    4.719285] localhost ignition[353]: failed to fetch config: write udp [::]:51601->10.0.2.23:69: sendto: network is unreachable

[    4.720046] localhost ignition[353]: failed to acquire config: write udp [::]:51601->10.0.2.23:69: sendto: network is unreachable

[    4.720310] localhost ignition[353]: Ignition failed: write udp [::]:51601->10.0.2.23:69: sendto: network is unreachable

[    4.720817] localhost systemd[1]: ignition-disks.service: Main process exited, code=exited, status=1/FAILURE

[    4.721469] localhost systemd[1]: ignition-disks.service: Failed with result 'exit-code'.

[    4.721770] localhost systemd[1]: Failed to start Ignition (disks).

The output of 'ip link show' shows both the 'lo' and 'eno1' interfaces as 'state DOWN', even after booting in the emergency shell and the network cable plugged in. Could this be a timing issue?

I did find this link (https://flaviutamas.com/2019/fixing-name-resolution-after-sleep) which seems related but I'm not sure how it can be of help.

[k...@kinvolk.io]

Yes, you need a JSON, not YAML input. You can transpile it: https://www.flatcar-linux.org/docs/latest/provisioning/config-transpiler/getting-started/

There is also a new web application for a quick try: https://jepio.github.io/hackathon2021/

I would recommend to try again using the JSON with the HTTP URL. Ignition retires until the network is configured, at least for TCP. I don't know if a retry is missing for UDP-based tftp access.

If it works with HTTP but not with tftp, please file a GitHub issue to track this.