Flatcar image on AWS

[Justin Hopper]

Hello,

I've recently started using the Flatcar image on AWS, and I'm having some issues SSHing to the instance after boot.  I'm using TerraForm's "aws_instance" to create the instance, using Flatcar image 3033.2.1

In the aws_instance config block, I'm specifying an AWS keypair, but regardless of how I try, I cannot SSH to the instance with the private key.  It continues to prompt for a password.

The documentation here makes it sound like this is the right procedure: https://www.flatcar.org/docs/latest/installing/cloud/aws-ec2/#ssh-to-your-instances

Do I need to instead set up a user and public SSH key via Container Linux Config?

Thanks for your help.

- Justin

[Justin Hopper]

An update on the below issue: it seems to be some issue between TerraForm and the Flatcar AMIs, as I can set a key-pair when using the aws console and the Flatcar images, and I can use TerraForm to set a key-pair with Ubuntu images, but cannot get a key-pair to work for SSH authorized-keys for Flatcar images.

Would be great to hear if anybody else has had this issue, or any known workarounds.

- Justin

[k...@kinvolk.io]

Hi,

I've tried it again now and using the "key_name" attribute for AWS SSH keys works:

https://github.com/flatcar-linux/flatcar-terraform/pull/10/files#diff-b098c847879d62a703b85c0a0a94c5391344ccefe569810b793695ae5e188040R126

You should see the keys from the VM attribute under /home/core/.ssh/authorized_keys.d/coreos-metadata while the ones from Ignition are under /home/core/.ssh/authorized_keys.d/flatcar-ignition.

Regards,
Kai

[Justin Hopper]

Hey Kai,

Thanks for responding to this.

So I hadn't yet tried using a Container Linux Config block yet.  I had been using standard cloud-init for other VM setup, and in the AWS config, it specified either shell script or cloud-init were the only two options.

I'll try the setup you are using, with a Container Linux Config set in the user_data, as well as sending the public key in the key_name.

This setup seems like duplication though, because you should be able to just set the key_name and have AWS inject the SSH key into the authorized_keys, right?  That's the way it works in Azure for us, and it even works on Ubuntu AMIs in AWS.  But maybe both is required for Flatcar, both a Container Linux Config to inject the SSH key, _and_ sending the key in the ssh_keyname input for the instance?

- Justin

--
You received this message because you are subscribed to the Google Groups "Flatcar Container Linux User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatcar-linux-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/flatcar-linux-user/f19cbd5c-2f77-469c-a91c-5ed7dfdd3d3an%40googlegroups.com.

[Justin Hopper]

Hey Kai,

Following up on this.  I believe the root cause of the issue I'm seeing is that we are using the v2 of the metadata service (IMDSv2).  I resorted to baking an SSH key into an image and then finally was able to SSH into an instance booted from that image.  Once in, I could see that the metadata service could not be reached, and thus explains why my SSH key was not being inserted through cloud-init (since it couldn't be read).  This problem also seems to happen with your example using Container Linux Config and poseidon/ct provider.

When I SSH into the instance, I see the failure of the coreos-cloudinit system:

Mar 08 08:48:58 ip-10-6-12-49.corporate.t-mobile.com coreos-cloudinit[757]: 2022/03/08 08:48:58 Checking availability of "cloud-drive"
Mar 08 08:48:58 ip-10-6-12-49.corporate.t-mobile.com coreos-cloudinit[757]: 2022/03/08 08:48:58 Checking availability of "ec2-metadata-service"
Mar 08 08:49:06 ip-10-6-12-49.corporate.t-mobile.com coreos-cloudinit[757]: 2022/03/08 08:49:06 No datasources available in time
Mar 08 08:49:06 ip-10-6-12-49.corporate.t-mobile.com systemd[1]: oem-cloudinit.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 08:49:06 ip-10-6-12-49.corporate.t-mobile.com systemd[1]: oem-cloudinit.service: Failed with result 'exit-code'.
Mar 08 08:49:06 ip-10-6-12-49.corporate.t-mobile.com systemd[1]: Failed to start Cloudinit from platform metadata.

There's also the failure of the AWS ssm agent, which looks to be related to metadata service failure:

ip-10-6-12-49 /var/log # cat amazon/ssm/amazon-ssm-agent.log 
2022-03-07 20:27:34 INFO Entering SSM Agent hibernate - EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
	status code: 404, request id: 
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>

2022-03-07 20:28:54 INFO Got signal:terminated value:0x5650657d4f90
2022-03-07 20:28:54 INFO Stopping agent

In TerraForm, we are using IMDSv2 with this block:

 metadata_options {
    http_endpoint               = "enabled"
    http_put_response_hop_limit = 1
    http_tokens                 = "required"
  }

I am forced to use the v2 of the metadata service by security policy of my employer.

So my question is: how do I get Flatcar working with IMDSv2?  Once working, I think I can accomplish what I need from our config injection via cloud-init or ignition, but neither seem to work at the moment with IMDSv2

- Justin

On Mon, Mar 7, 2022 at 11:32 AM k...@kinvolk.io <k...@kinvolk.io> wrote:

--

[Kai Lüke]

Hi,
yes, for AWS the Terraform example uses both ways of specifying and
SSH key (the CLC example is the same for all platforms).

IMDSv2 is coming soon. The next Alpha will have a new Ignition version
which supports it and the next thing we need to update is Afterburn,
then it should work.

Regards,
Kai

> To view this discussion on the web visit https://groups.google.com/d/msgid/flatcar-linux-user/CA%2B1gPOZxGWFjBpP_hYKH%2BKB1XQ49uQRVKuPY7QhiEeyXD2jrqg%40mail.gmail.com.



--
Kinvolk GmbH | Adalbertstr.6a, 10999 Berlin | tel: +491755589364

Geschäftsführer/Directors: Benjamin Owen Orndorff

Registergericht/Court of registration: Amtsgericht Charlottenburg

Registernummer/Registration number: HRB 171414 B

Ust-ID-Nummer/VAT ID number: DE302207000

[Justin Hopper]

Kai,

So are you saying that Flatcar does NOT support IMDSv2 right now?  If so, what's the ETA for this?  Any experimental branch or work I can use in the meantime?

- Justin

[Kai Lüke]

The next Alpha release in two weeks
(https://github.com/orgs/flatcar-linux/projects/5) should have basic
support from the Ignition side (already merged).
However, at the current state you would have to supply the SSH key
inside Ignition because the SSH key VM attribute fetching code is not
yet updated for IMSDv2.
Not sure if something else isn't working, we didn't test this at all yet.

Kai

> You received this message because you are subscribed to a topic in the Google Groups "Flatcar Container Linux User" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/flatcar-linux-user/iCdzSUBvyT0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to flatcar-linux-u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/flatcar-linux-user/CA%2B1gPOYYcweAWcV6ZyKBQ9n8%2B6e8Zab%2BQzK2ByA8azCzRBC%3DAw%40mail.gmail.com.