Configuration of AMI-Based Instances

[Ross McFarlane]

Hi,

Sorry if this isn't the right place for this question. If there's somewhere better suited, please let me know.

I'm curious about how Netflix configure AMI-based instances with environment-specific configuration (e.g. dev/test/prod). I understand that Archaius provides context-specific configuration, and that Eureka will allow instances to find external services. What I'd like to know is, assuming that the Eureka/Archaius instances are different between environments and AWS regions, how do instances find them?

We're looking at a similar setup, possibly using Archaius, for our own purposes and I see a few options:

1. Have a global configuration server and bake its details into the AMI.
2. Bake the hostname of the configuration server into the AMI and use split-horizon DNS to resolve it to the environment-specific one. I'm not sure if/how this would work in AWS. If you're doing it, I'd love to know how.
3. Add all environment-specific configuration server addresses into the AMI, and switch using startup data when the instance is created.
4. Explicitly pass in the configuration server address when the instance starts.

I'd be interested to know which, if any, of these methods you're using and why.

Thanks in advance,

Ross

[Brian Moyles]

Is there any reason you couldn't use some sort of DNS hierarchy composed of environmental pieces to help services locate Eureka/Archaius? If you are consistent in your deployments and how you name your hosts, you should be able to bake in a basic top-level name like "eureka" and then compose the domain name from availability zone and/or region, account name, etc. eureka.us-west-2.somedomain.net or eureka-us-west-2a.test.somedomain.net, etc. 

Also, Archaius doesn't necessarily have an "instance" depending on what you're using to store the configuration details. We are using DynamoDB, for example, so there isn't much that needs to change at all between regions (and those details tend to be handled by the AWS SDKs).

Lastly, consider that you have the ability to pass in run-time userdata to your instances that can help inform your instances' decision making (and tools like Asgard help ensure you're delivering said userdata to your instances in a consistent fashion.

Netflix uses a combination of those three to help with service location in the cloud. I prefer to avoid the userdata route where possible as I like to do as little twiddling at runtime based on those values as possible, but that might be the way to go depending on your setup.

Hope that helps a little.

Brian

--
You received this message because you are subscribed to the Google Groups "Aminator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aminator+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Ross McFarlane]

Hi Brian,

Thanks for getting back to me.

That's a good plan. I think we'll go with something along those lines, e.g. userdata to indicate the environment (dev/test/prod), a DNS hierarchy to identify the configuration server (test-config.blah.com), then have the rest come in from the config server.

Thanks,

Ross

[Demetri Mouratis]

This is a great question and one I have been wrestling with as well.

I have a current goal of being able to reuse identical AMIs as code progresses from dev->test->staging->prod.

Here's an idea I came up with but have yet to get to:

Use AWS tags.

In particular, if you have one (or more) ASG(s) for each environment, you can add tags to the ASG as KV pairs. You can then use the AWS API to retrieve those KV pairs and pass them to your application(s).

I'm thinking of things like db hostname, db username/password, config elements, etc.

http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/ASTagging.html

Thoughts?

[Brian Moyles]

You'd probably want to use some combination of tags and user data. One problem with tags is that they are not exposed to the instance without the instance making API calls. It'd be *wonderful* if an instance could read the metadata service and find out how it was tagged.

User data, though, is available through the metadata service, and if you're using something like Netflix's Asgard, you can provide a new user data implementation that provides arbitrary data to instances at launch time that helps them make decisions at run time. Asgard also provides a number of convenience K/V pairs out of the box that help immensely. It provides an application concept and injects a K/V pair with the application name, provides the name of the autoscaling group an instance belongs to, and so on. Those dimensions may be enough for you to devise a scheme that allows for slightly different runtime configurations with a common foundation AMI.

Brian

--