Boot System On AWS

[inganta]

Our company is using AWS for development and testing purpose. As system engineer i have responsibility to handle stuffs related to aws. One thing i found annoying is dinamic ip addressing. Why? Let me explain

Suppose we set up a system which comprise of severals ec2, rds or others aws stuffs for dev or testing. Some of component refer to other instance IP address in their configuration file. Since ip address of an instance will be changed every time we start the instance, we have to make adjustment to the configuration file. Suppose we have made a script to adjust options on the configuration file, we still need to execute that script and pass needed argument to the script. I found that this process is such a pain and It can be automated. So it will be good if there is some kind of monkey that have ability to start that system, executing scripts and passing needed argument dynamically just by clicking a button or time triggered. In other word, we need monkey to start and configure system.

Benefits that i can see are the following:

1. Less work since starting and stopping a system can be automated
2. Cut cost, since we can easily shutdown the system and starting it without headache.

I have not found that kind of monkey yet. I wonder if there is anybody out there doing that thing already. If no, i will be glad to collaborate building that monkey.

Any insight is welcome.

Thanks.

[Peter Sankauskas]

I would suggest studying the finer points of Service Oriented Architecture (SOA), and then using Eureka for discovery of services rather than hard-coding IP addresses on one or both sides.

For an example, have a look at Netflix RSS recipe

https://github.com/Netflix/recipes-rss/wiki/Architecture

or http://fluxcapacitor.com/

--

Kind regards,

Peter Sankauskas

Answers for AWS

[inganta]

I think it is too complex and can not be used in some cases.

For example, i have one mogilefs (distributed file system) cluster on aws. It is needed for testing purpose, not production. SOA can not be implemented on this mogilefs cluster since it will read others component ip address statically from configuration file when this cluster start. 

[Sean Kavanagh]

Have you tried Elastic IP Addressing?

Here is a good article..

http://alestic.com/2009/06/ec2-elastic-ip-internal

So your config files would contain the elastic ip (and i guess you would have separate elastic ips for dev, test, etc..).  That way when you want your app to point to a new EC2 instance, you just have to reassign where the elastic ip points to.

Sean

[inganta]

Using elastic IP is impractical for this purpose. Why? because the system comprises of a lot of server so it need much EIP. I know we can request more EIP but it will take time. Since the system is running at working hour (around 8 hours a day), it takes more effort to release all EIP and reassign them again later.

[Adrian Cockcroft]

This is the same kind of problem that Cassandra has, we built Priam to manage the allocation of IP addresses, setup config files and automatically create and repair large clusters without using EIPs. We use Eureka to register the members and for clients to locate the cluster by name. 

Another way to handle this is with DNS, you can use the denominator command on an instance to assign its own IP to a DNS name, then have your config files use the Route53 DNS names of the nodes. 

Adrian

[Scott Emmons]

On Wed, Aug 14, 2013 at 8:02 AM, Adrian Cockcroft <acock...@netflix.com> wrote:

Another way to handle this is with DNS, you can use the denominator command on an instance to assign its own IP to a DNS name, then have your config files use the Route53 DNS names of the nodes. 

This is the path I was going to suggest as well. Simply create an init.d/upstart script that registers and deregisters DNS records with the lifecycle of your service. You can CNAME to an "ec2-a.b.c.d.compute-1.amazonaws.com" name, or use an A record to the 'public' IP. There are several different ways to do it, such as denominator or the Python boto library. It's pretty simple and reliable enough, especially for non-production use.

--

Thanks,

-Scott