Ignition: download from s3

[Cemo Koc]

I am trying to bypass limit of aws userdata and s3 is only viable solution right now. There are some issues to support IAM and s3 bucket natively so I need to find a way as I did in cloud init. Is there a way of running some scripts and let us to download necessary file as this:

```

---

coreos:

  units:

    - name: bypass-user-data-limit.service

      command: start

      content: |

        [Unit]

        Description=Update the machine using our own cloud config as AWS user-data sucks

        [Service]

        EnvironmentFile=/etc/environment

        ExecStartPre=/opt/bin/fetch-from-s3 ${s3-cloud-init-file}

        ExecStart=/usr/bin/coreos-cloudinit --from-file /etc/systemd/system/${s3-cloud-init-file}

        RemainAfterExit=yes

        Type=oneshot

write-files:

  - path: /opt/bin/fetch-from-s3

    permissions: 0755

    owner: root:root

    content: |

      #!/bin/bash -e

      until /usr/bin/rkt run \

        --net=host \

        --trust-keys-from-https \

        --volume=dns,kind=host,source=/etc/resolv.conf,readOnly=true --mount volume=dns,target=/etc/resolv.conf \

        --volume=ssl,kind=host,source=/etc/systemd/system,readOnly=false --mount=volume=ssl,target=/etc/systemd/system \

        quay.io/coreos/awscli -- aws s3 cp s3://${ s3-bucket }/$1 /etc/systemd/system

      do

        echo "retrying"

        sleep 5.2

      done

      echo "✓"

```

Any help? 

[Alex Crawford]

On 05/29, Cemo Koc wrote:
> I am trying to bypass limit of aws userdata and s3 is only viable solution
> right now. There are some issues to support IAM and s3 bucket natively so I
> need to find a way as I did in cloud init.

As you noticed [1], we are working on getting S3 URLs working natively
in Ignition. In the meantime, we recommend using a pre-signed S3 URL.
This will allow you to access the object but prevent others (I believe
S3 imposes a maximum of seven days for the pre-signed URL).

-Alex

[1]: https://github.com/coreos/bugs/issues/1631#issuecomment-304665186

[Charles Allen]

We have a simple signing service that will do a one-time sign on an S3 item and return a 302 directing the client to the signed URL. It works pretty well for general items, but we don't use it in ignition configs.

We have various acute issues with S3 including DNS problems or temporarily service failures.

Is the retry behavior of ignition fetches documented anywhere?

I would be very concerned if a simple 5XX response from an S3 endpoint could prevent an instance from booting without warning.

[Alex Crawford]

On 06/07, Charles Allen wrote:
> Is the retry behavior of ignition fetches documented anywhere?

It is not, but it really should be. Ignition will retry fetches until
they succeed (there are other mechanisms in the OS which will cut the
entire boot process short though). Fetches are considered complete if
they return HTTP codes less than 500. So, for example, if you fetch a
config which results in a 404, Ignition will immediately fail. If it
returns a 500, Ignition will keep trying.

> I would be very concerned if a simple 5XX response from an S3 endpoint
> could prevent an instance from booting without warning.

Agreed. On top of the above behavior, if Ignition fails to complete and
it gets stuck in the initramfs, the machine will begin a five-minute
countdown. If there is no user input within that time, the machine will
reboot and start the whole process again. The effect is that your
machine will continually attempt to boot and run Ignition until it is
sucessful.

-Alex

[Charles Allen]

Thank you, Alex!

It is worth noting that AWS's S3 has 99.9% monthly uptime SLAs as of the time of this writing (as per https://aws.amazon.com/s3/sla/ ).

For 5 minutes to be enough for ignition to expect recovery, that means the underlying service should have 99.99% monthly uptime SLA. 

So it is not immediately clear that pulling from S3 is a suitable target for ignition as it is currently configured.

Thoughts?

[Charles Allen]

since ignition can constantly retry though (after reboots), I guess it depends on how long the apps running are it are allowed to not have the node available.