Hey Kai,
Following up on this. I believe the root cause of the issue I'm seeing is that we are using the v2 of the metadata service (IMDSv2). I resorted to baking an SSH key into an image and then finally was able to SSH into an instance booted from that image. Once in, I could see that the metadata service could not be reached, and thus explains why my SSH key was not being inserted through cloud-init (since it couldn't be read). This problem also seems to happen with your example using Container Linux Config and poseidon/ct provider.
When I SSH into the instance, I see the failure of the coreos-cloudinit system:
There's also the failure of the AWS ssm agent, which looks to be related to metadata service failure:
ip-10-6-12-49 /var/log # cat amazon/ssm/amazon-ssm-agent.log
2022-03-07 20:27:34 INFO Entering SSM Agent hibernate - EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
status code: 404, request id:
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>404 - Not Found</title>
</head>
<body>
<h1>404 - Not Found</h1>
</body>
</html>
2022-03-07 20:28:54 INFO Got signal:terminated value:0x5650657d4f90
2022-03-07 20:28:54 INFO Stopping agent
In TerraForm, we are using IMDSv2 with this block:
metadata_options {
http_endpoint = "enabled"
http_put_response_hop_limit = 1
http_tokens = "required"
}
I am forced to use the v2 of the metadata service by security policy of my employer.
So my question is: how do I get Flatcar working with IMDSv2? Once working, I think I can accomplish what I need from our config injection via cloud-init or ignition, but neither seem to work at the moment with IMDSv2
- Justin