user-data file size is too large

[Krish]

I am trying to launch an instance in AWS EC2 and it complains that the user-data file is too large.

My user-data file is 18490 bytes; and I have not even put all my stuff in there!

Has anyone faced this issue and solved it successfully?

ubuntu@host:~

> aws ec2 run-instances \

>  --image-id ami-05783d60 \

>  --region us-east-1 \

>  --instance-type m3.medium \

>  --security-group-ids sg-002a379 \

>  --key-name workspace/krish-virginia.pem \

>  --user-data file://workspace/cloud-infra/1.yaml \

>  --disable-api-termination \

>  --instance-initiated-shutdown-behavior stop

A client error (InvalidParameterValue) occurred when calling the RunInstances operation: User data is limited to 16384 bytes

ubuntu@host:~

> echo $?

255

--

κρισhναν

[anton....@coreos.com]

Dear Krish,

> User data is limited to 16384 bytes

You have to decrease your cloud-config file size. It is partly possible by defining "write_files:" with compression. You can find more info in this doc: https://coreos.com/os/docs/latest/cloud-config.html#write_files

Regards,

Anton

[Krish]

Thanks Anton. Your suggestion is useful, but then I will have to modify the binary/base64 encoded string every time the source is changed, automate it, etc etc. I would like the source modification to the user-data to be traceable. :)

A little digging up shows that CoreOS will support gzipped user data file from version 921.0.0. It is currently in alpha.

For now, I am going to reduce the size of my user-data file & remove some services and run a single service on smaller instances in AWS EC2, instead of running a larger instance with multiple containers/services.

--

κρισhναν

[Henrik Jonsson]

I also hit this limitation at my last gig, specifically when wanting to pack up iptables rules for CoreOS nodes running on EC2 via the cloud-config mechanism.

Shipping iptables rules to nodes via cloud-config worked like a charm for our baremetal clusters, but the 16K limit imposed by Amazon made it infeasible there. 

The path I took at the time involved having cloud-config write a systemd timer + service which would periodically pull down the rules from an external source and apply them, but it was both rube goldbergesque and fragile, due to the external dependency.

I realize that the 16K limit is imposed by Amazon, but it would be nice to hear if there's a robust pattern that CoreOS team can recommend in these cases, ideally without needing to rely on anything outside the CoreOS node itself.

The cloud-config mechanism feels like a very powerful tool in order to package up the "infrastructure setup" stuff that a new CoreOS node needs in order to join a cluster and perform its tasks, while completely avoiding needing any kind of orchestration to put stuff on top of the node once it's up.

Even if gzipping the user-data might have gotten a few more bytes in, having to optimize a config to fit in such a tiny total footprint in 2016 feels a bit silly.

Best,

Henrik

[Alex Crawford]

On 02/27, Henrik Jonsson wrote:
> I realize that the 16K limit is imposed by Amazon, but it would be nice to
> hear if there's a robust pattern that CoreOS team can recommend in these
> cases, ideally without needing to rely on anything outside the CoreOS node
> itself.

This just isn't possible. The utility will always have to fetch more data from
another source external to the node. You might be interested in the proposed
solution [1] for Ignition. The idea here is to reference remote configs and
either append them to the existing config or replace it outright. Since the data
is being fetched from an external source, there are a number of provisions for
verifying that the data hasn't been tampered with while on disk or while in
transit. I'd love any feedback you have.

-Alex

[1]: https://github.com/coreos/ignition/pull/166/files

[Henrik Jonsson]

Thanks for the reply and link Alex.

Right, makes sense that data that doesn't fit in user-data needs to be somewhere.

(I was hoping that there was some other EC2-specific metadata where extra cloud-config data could be hacked in, or something along those lines.)

The patterns you use for Ignition sound interesting and conceptually similar to what I'd want to happen for "extra cloud-config data" that doesn't fit. As long as I could have a stable setup which didn't require messing around with node state after they're done starting up to get all the infra dependencies and services running I'd be happy.

So if I have to bite the bullet and have a dependency on some shared resource which CoreOS nodes coming up need to access to fetch those extra configs, I'd maybe do something like:

• Write a bootstrap oneshot systemd service with cloud-config that runs on node startup
• That systemd service should fetch "extra cloud-config data" over TLS, ideally verifying cert of config server as well as checksum or signature of config data itself
• Config format for the "extra data" should ideally just be the regular cloud-config format and be applied with the same mechanism, same way as you want to append Ignition configs

Thoughts?

Cheers,

Henrik

 

[Alex Crawford]

On 03/04, Henrik Jonsson wrote:
> So if I have to bite the bullet and have a dependency on some shared
> resource which CoreOS nodes coming up need to access to fetch those extra
> configs, I'd maybe do something like:
>

> - Write a bootstrap oneshot systemd service with cloud-config that runs
> on node startup
> - That systemd service should fetch "extra cloud-config data" over TLS,

> ideally verifying cert of config server as well as checksum or signature of
> config data itself

> - Config format for the "extra data" should ideally just be the regular

> cloud-config format and be applied with the same mechanism, same way as you
> want to append Ignition configs
>
> Thoughts?

If you want to use coreos-cloudinit, you could do something like this (I haven't
tested this):

#cloud-config
coreos:
units:
- name: extra-cloudinit.service
command: start
content: |
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/coreos-cloudinit --from-url=https://example.com

I think this is just about the most simple implementation. As long as you keep
example.com running, this should be fine.

-Alex

[Henrik Jonsson]

Cool, not sure why I didn't think to just check if coreos-cloudinit had such a flag!

Definitely a lot simpler than what I expected, cheers. :)

//Henrik